OpenRouter (and potentially other proxy providers) expose OpenAI's
text-embedding-3 models with a provider prefix in the model ID, e.g.
`openai/text-embedding-3-large` rather than bare `text-embedding-3-large`.
`dimsProviderOptions()` checks `modelId.startsWith('text-embedding-3')`
which fails for the prefixed form, so the `dimensions` parameter is never
sent. The upstream provider returns its native dimensionality (3072 for
-large) instead of the configured value (e.g. 1536), causing an immediate
"dim mismatch" error on first embed.
The default OpenRouter embedding (`text-embedding-3-small` at 1536d)
masked this because its native size happens to match the default config.
The bug surfaces when using `-large`, or `-small` with a non-1536 dim
(512, 768, 1024 — all listed in the recipe's `dims_options`).
Fix: strip the provider prefix before the `startsWith` check. The full
prefixed ID is preserved in the error message for user clarity.
* feat(subagent): claude-cli MessagesClient adapter (baseline, no tool use)
Closes#334 (partially — text-only baseline; tool use lands in the next
commit on this branch).
Adds a MessagesClient adapter that shells out to `claude --print
--output-format json --model <model>` instead of the Anthropic SDK. When
`GBRAIN_USE_CLAUDE_CLI=1` is set, the subagent worker registers the adapter
in place of the SDK client; the default path (Anthropic SDK with
ANTHROPIC_API_KEY) is unchanged when the env var is unset or set to
anything else.
The benefit is that Claude Max subscribers can run Minions subagents
against their existing OAuth subscription, no ANTHROPIC_API_KEY needed.
New: src/core/minions/handlers/claude-cli-adapter.ts
- Implements the MessagesClient interface exported from subagent.ts.
- Strips provider prefixes (`anthropic:`, `litellm:`) from the model id
because `claude --print` only accepts CLI-native aliases (`sonnet`,
`opus`, `haiku`, or the bare `claude-*-N-M` form).
- Flattens the Anthropic messages array into a single text prompt for
claude-cli stdin. Tool blocks (tool_use / tool_result) are stringified
as placeholders so multi-turn conversations stay coherent in this
baseline; native tool_use round-tripping is the follow-up commit.
- Spawns claude with stdio piped, captures stdout, parses the
`{type:"result", subtype:"success", result, usage, ...}` JSON envelope,
and returns it as a properly shaped Anthropic.Message with
`stop_reason: 'end_turn'`.
- Token totals propagate from the claude usage block so the subagent
handler's `ctx.updateTokens()` reports usable numbers.
- AbortSignal is wired through to SIGTERM the child so the subagent loop's
cancellation path stays correct.
Modified: src/commands/jobs.ts (worker registration)
- Conditionally constructs a MessagesClient via the new adapter when
GBRAIN_USE_CLAUDE_CLI=1.
- Passes it into makeSubagentHandler({ engine, client: subagentClient }).
- Logs `[minion worker] subagent routing via claude-cli (GBRAIN_USE_CLAUDE_CLI=1)`
on startup so the env var status is operator-visible.
Limitations of this commit (addressed in the follow-up):
- Tool use is not yet supported. Tools in params.tools are ignored; the
adapter returns a single text block with stop_reason='end_turn'.
- Token counts come from claude-cli's reporting and may not match the
Anthropic API's accounting precisely (especially for cache tiers).
Original design from #334; this commit preserves that author's attribution.
The follow-up commits on this branch carry the tool-use implementation.
* feat(subagent): tool use + context isolation + convention rename on top of jarvisdoes baseline
Builds on the previous commit (jarvisdoes's #334 baseline) by adding three
things the upstream issue called out as gaps or that surfaced during review:
1. Tool use support via system-prompt-instructed JSON emission.
2. Context isolation flags so claude-cli does not load operator-level
CLAUDE.md, skills, and local project context into every subagent call.
3. Env var rename from GBRAIN_USE_CLAUDE_CLI=1 to
GBRAIN_SUBAGENT_PROVIDER=claude-cli to match the existing
GBRAIN_<noun>_<role>=<value> convention used by GBRAIN_CHAT_MODEL,
GBRAIN_EMBEDDING_MODEL, GBRAIN_EXPANSION_MODEL.
## Tool use
The MessagesClient interface returns Anthropic.Message objects whose
content array may include tool_use blocks. The subagent handler filters
those blocks and dispatches each tool, so any backend that produces
correctly shaped tool_use blocks gets the same loop behavior as the
Anthropic SDK.
The adapter injects a system-prompt addendum describing the tool registry
plus an emission protocol:
<use_tools>
[{"id": "...", "name": "...", "input": {...}}, ...]
</use_tools>
After the response comes back, extractToolCalls() scans for the block,
parses the JSON (tolerant of optional ```json fencing), and converts each
entry into a tool_use content block. Multiple parallel tool calls in one
turn are supported via the array shape; this is the exact case that
breaks today on the codex-proxy / litellm GPT-5.x bridge where parallel
tool-call response IDs get dropped.
Defensive fallbacks:
- Malformed JSON inside the block: drop to text-only, stop_reason='end_turn'.
- Unterminated <use_tools> (no close tag): drop to text-only.
- Model omits id field: adapter synthesizes a toolu_claude_cli_<rand> id.
- Empty response: still hand the subagent loop a well-formed content
array so the .filter chain does not crash.
## Context isolation
claude-cli auto-discovers CLAUDE.md from cwd upward and injects the
operator's skills + plugins + auto-memory into the default system prompt.
On a real install that is ~42-65k tokens of contamination per subagent
call, with both cost and behavioral consequences (the subagent picks up
the operator's coding conventions, opinions, and preferences).
The maximum suppression that still preserves OAuth / Claude Max
subscription auth is:
- Spawn from a dedicated clean cwd (tmpdir-based) so LOCAL CLAUDE.md
auto-discovery has nothing to find. -13k tokens on a real gbrain
install where CLAUDE.md is substantial.
- --disable-slash-commands so skill resolution does not pull in
/skill-name handlers.
- --system-prompt <gbrain prompt> so the default system prompt is
replaced rather than appended to.
The --bare flag would also strip user-level ~/.claude/CLAUDE.md but it
forces ANTHROPIC_API_KEY auth, defeating the whole point of this adapter.
The remaining ~42k cached tokens from user-level instructions are
accepted as a cost-trivial trade-off because the Max subscription absorbs
the per-call cost. Behavioral contamination is mitigated by gbrain's
strong per-call system prompt overriding any operator-level drift.
## Env var rename
Surveyed all ~140 GBRAIN_* env vars in src/. The codebase uses three
patterns: GBRAIN_NO_<feature> (negative toggles), GBRAIN_<noun>_<role>
=<value> (routing keys), GBRAIN_ALLOW_<feature> (permissive toggles).
GBRAIN_USE_* does not appear anywhere except jarvisdoes's original
commit; it would introduce a fourth pattern.
GBRAIN_SUBAGENT_PROVIDER=claude-cli aligns with the routing-keys family
and is value-extensible — adding codex-cli / meridian-proxy / etc. later
means a new value, not a new env var. The scope ('SUBAGENT_*') is also
unambiguous about which calls the toggle covers; GBRAIN_USE_CLAUDE_CLI
was silent on whether it applied to all gbrain LLM calls or only the
subagent path.
Unknown values are rejected with a fail-fast error message naming the
two valid values rather than silently falling through to the default.
## Tests
New file: test/claude-cli-adapter.test.ts — 12 tests, 33 assertions:
- Text-only round trip (single text block, usage propagation, end_turn).
- Provider prefix stripping ('anthropic:claude-sonnet-4-6' -> 'claude-sonnet-4-6').
- Single tool_use parsing.
- Multiple parallel tool calls in one block (the case that triggered
the codex-proxy regression).
- Fenced JSON inside <use_tools> block.
- Model-omitted id gets synthesized to toolu_claude_cli_<rand>.
- Malformed JSON falls back to text.
- Unterminated block falls back to text.
- AbortSignal SIGTERMs the child.
- Error envelope rejected with informative message.
- Non-JSON output rejected with raw-output excerpt in the error.
- argv + cwd assertion: --disable-slash-commands + --system-prompt are
present and cwd is the dedicated tmpdir.
Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN that emits a
scripted --output-format json envelope, so the suite runs without
claude-cli installed and without API credits.
* feat(ai): claude-cli recipe with native gateway integration (supersedes #334 baseline)
Replaces the MessagesClient adapter + GBRAIN_USE_CLAUDE_CLI=1 env-var
gate from the previous commit on this branch with a proper gateway recipe.
The recipe path gives per-call routing as a native capability: a model
string like `claude-cli:claude-sonnet-4-6` lands here while a sibling
`litellm:gpt-5.4` continues through the litellm-proxy / codex-proxy path
in the same worker. No global env-var switch, no agent.use_gateway_loop
bypass, no MessagesClient injection at jobs.ts worker startup.
The previous commit on this branch (jarvisdoes baseline) is preserved
in the history for #334 authorship attribution. Its functional changes
are backed out here because the recipe pattern is gbrain's established
integration seam; introducing a parallel MessagesClient + env-var path
would have created two routing mechanisms competing for the same job.
New: src/core/ai/recipes/claude-cli.ts
- Recipe declaration: id 'claude-cli', tier 'native', implementation
'claude-cli', chat-only (no embedding or expansion touchpoints).
- Models: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5-20251001.
- supports_tools and supports_subagent_loop both true.
- supports_prompt_cache false because the CLI handles caching internally
and does not surface cache_control via the standard control plane.
- auth_env.required is the empty array because the CLI owns auth (OAuth
session managed by `claude login`).
- Friendly aliases mirror the `anthropic` recipe: `sonnet`, `haiku`,
`opus` and the same legacy-id rewrites for back-compat with stale
config strings.
New: src/core/ai/providers/claude-cli-language-model.ts
- ClaudeCliLanguageModel class implementing the ai-sdk LanguageModelV2
interface.
- doGenerate: renders the ai-sdk prompt array into a system text + user
text, injects the use_tools protocol instructions when tools are
present, spawns `claude --print --output-format json --model <X>
--disable-slash-commands --system-prompt <gbrain prompt>` from a
dedicated tmpdir (contamination suppression: no local CLAUDE.md
auto-discovery), parses the JSON envelope, extracts <use_tools>
blocks, and returns ai-sdk-shaped LanguageModelV2Content (text +
tool-call parts with stringified-JSON input matching the V2 contract).
- Tolerates fenced JSON inside use_tools blocks, malformed JSON
(falls back to text), missing close tag (falls back to text),
model-omitted ids (synthesizes toolu_claude_cli_<rand>).
- Parallel tool calls in one block round-trip cleanly: this is the
case that drops IDs on the litellm + codex-proxy bridge today.
- AbortSignal SIGTERMs the child for proper cancellation.
- doStream throws not-supported (gateway.toolLoop is non-streaming).
Modified: src/core/ai/gateway.ts
- Adds case 'claude-cli' to instantiateChat (returns ClaudeCliLanguageModel).
- Adds case 'claude-cli' to instantiateExpansion (same wrapper, reserved
for a future expansion touchpoint declaration).
- Adds case 'claude-cli' to instantiateEmbedding (throws, no embedding
model, mirrors the native-anthropic path).
- Lazy require() at the call site keeps the gateway module load cheap
for users who never use the claude-cli path.
Modified: src/core/ai/recipes/index.ts
- Registers `claudeCli` in the ALL[] array next to `anthropic`.
Modified: src/core/ai/types.ts
- Adds 'claude-cli' to the Implementation union so the gateway switch
is exhaustive at compile time.
Reverted: src/commands/jobs.ts
- Drops the GBRAIN_USE_CLAUDE_CLI=1 env-var gate the prior commit
added. Routing now happens at the gateway based on the model string.
Deleted: src/core/minions/handlers/claude-cli-adapter.ts
- The MessagesClient adapter is superseded by the recipe + LanguageModelV2
path. Two routing mechanisms competing for the same job would have
forced users to reason about which one wins; the recipe is the single
source of truth.
New file: test/claude-cli-recipe.test.ts (16 tests, 46 assertions):
- Recipe registration: getRecipe returns chat-only Recipe; aliases map
short names (sonnet/haiku/opus) to canonical model ids.
- Text round trip: single text content block, usage propagation, stop
finish reason.
- Provider prefix stripping.
- Single tool-call parsing.
- Multiple parallel tool calls in one block.
- Fenced JSON inside the block.
- Model-omitted id synthesizes toolu_claude_cli_<rand>.
- Malformed JSON falls back to text + stop reason.
- Unterminated block falls back to text + stop reason.
- Tools offered but model declines: returns text-only with stop reason
so the gateway-loop treats it as a final answer rather than wedging
for tool calls that never come.
- AbortSignal SIGTERMs the child.
- is_error envelope rejected.
- Non-JSON output rejected.
- doStream throws.
- argv + cwd assertion: --print, --disable-slash-commands,
--system-prompt are present and cwd is the dedicated tmpdir.
Tests use a POSIX shell stub at GBRAIN_CLAUDE_CLI_BIN so the suite runs
without claude-cli installed and without API credits.
End-to-end smoke verified against a real `claude --print --model haiku`
invocation: model emitted `<use_tools>` block with toolu_add_001 +
{"a":12,"b":30}, adapter parsed back into a `tool-call` content block,
finishReason 'tool-calls'.
* feat(ai/claude-cli): harden subagent isolation, env scrub, verbose + stdin robustness
Four defensive fixes to the claude-cli provider so a subagent call behaves
identically regardless of the host's ambient Claude Code config:
- Agent isolation: pass `--tools ''` and `--strict-mcp-config` so the subprocess
runs as a raw LLM with no built-in tools and no inherited user MCP servers.
Without `--strict-mcp-config`, each call boots the user's MCP servers (including
gbrain's own), causing recursion plus PGLite single-writer lock contention.
- Env scrub: drop ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL
from the child env so the CLI authenticates via its own OAuth subscription
session. An inherited API key silently flips billing to per-token API usage,
the exact setup this recipe exists to replace.
- Verbose-mode compat: with `"verbose": true` in ~/.claude/settings.json,
`--print --output-format json` emits an event array instead of a bare result
object. Tolerate both shapes and select the result event.
- stdin robustness: handle the child stdin 'error' event and wrap write/end so a
missing binary (ENOENT) or early child death (EPIPE) rejects cleanly instead of
crashing the worker with an unhandled error.
Adds unit coverage for the env scrub, the isolation argv, and the verbose event
array. Verified against claude CLI 2.1.x.
* test(ai/claude-cli): cover verbose-array no-result + missing-binary reject paths
Two error branches in the hardened claude-cli provider had no coverage: the
verbose event-array path when no result event is present, and a missing binary
surfacing as a clean spawn-failed rejection. The missing-binary case is the
deterministic form of the stdin/EPIPE robustness; a synchronous stdin-write
throw is not reliably triggerable in a unit test, so the real ENOENT path the
handlers defend is exercised instead. Both reuse the existing shell-stub harness.
---------
Co-authored-by: jarvisdoes <258486803+jarvisdoes@users.noreply.github.com>
Co-authored-by: Marco Maldonado <34176133+loweaxerium@users.noreply.github.com>
queue.add() with an idempotency_key returns any existing row regardless
of status. This means dead jobs (exhausted retries from a transient
provider outage) permanently block re-submission of the same work —
even after the underlying issue is fixed.
Fix: when the existing row is dead or cancelled, NULL its
idempotency_key (preserving the row for audit) and fall through to the
INSERT path so a fresh job can be created.
Affects dream synthesize children that died during provider migrations
(429 rate-limit on old Anthropic proxy, tool-results-missing on old
OpenRouter). 45 dead children were blocking re-synthesis of transcripts
in production.
Includes 4 new tests covering dead, cancelled, completed, and active
status interactions with idempotency dedup.
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
OpenRouter's POST /api/v1/rerank is wire-compatible with gateway.rerank()
({query, documents, model} → {results: [{index, relevance_score}]}). This
adds a recipe-only reranker touchpoint declaring four models:
- cohere/rerank-v3.5 (default; $0.001/search)
- cohere/rerank-4-fast ($0.002/search, 32K context)
- cohere/rerank-4-pro ($0.0025/search, SOTA quality)
- nvidia/llama-nemotron-rerank-vl-1b-v2:free (multimodal)
Unlike embedding/chat, the reranker path strictly enforces the models
allowlist — the openai-compat extended-model bypass does not apply. New
rerank models must be added to this recipe before they can be called.
The cost_per_1m_tokens_usd value is a pseudo-rate for the budget tracker's
chars/4 heuristic — Cohere bills per-search, not per-token. At ~4K chars
the estimated cost is in the right ballpark.
Recipe-only change; no gateway or search-layer modifications. gateway
auto-concatenates path → .../api/v1/rerank.
Adds hermetic unit test (test/openrouter-reranker-recipe.test.ts) covering
shape, models, default_model, path, max_payload_bytes, default_timeout_ms,
and cost field. No DB, no env mutation — survives the parallel 8-shard
fan-out.
Verified: bun run verify (30/30 green); 285 targeted recipe+rerank+budget
tests pass.
Co-authored-by: Hippityy <Hippityy@users.noreply.github.com>
Idempotency was keyed on atom rows alone — a page the LLM judges
un-atomizable leaves no row, so it re-entered the discovery window every
run. Two production consequences: --drain false-stopped with
no_progress once the window head was mostly zero-yield pages (remaining
frozen while batches report +0), and every nightly re-spent extraction
budget on the same pages.
Fix:
- After a SUCCESSFUL chat call that parses to zero atoms, stamp the
source page with frontmatter.atoms_scan_hash = contentHash16. LLM
failures take the catch path and stay retryable.
- discoverExtractablePages + countExtractAtomsBacklog (both variants)
exclude pages whose stamp matches the CURRENT content hash prefix —
content edits re-eligibilize, mirroring atom-row staleness semantics.
- Drain no_progress now recounts the backlog on a zero-atom batch and
only stops when it genuinely didn't shrink — tombstoning IS progress.
Tests: +2 pure-loop drain cases (shrinking backlog continues / flat
backlog stops) and +3 PGLite integration cases (stamp + exclusion /
content-change re-eligibility / failed chat does not stamp).
29 pass / 0 fail across the two files; tsc clean.
Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
synthesize-concepts.ts's design comment says extract_atoms stamps a
`concepts:` frontmatter field on each atom and :92 consumes ONLY that
field — but the extractor never wrote it, so the atoms → concepts
pipeline was dead end-to-end: every cycle reported "synthesize_concepts:
skipped — no atoms with concept refs" no matter how many atoms
accumulated (696 page-derived atoms / 0 with concepts on our production
brain before an external backfill).
Fix, all on the extractor side (no synthesize change needed):
- EXTRACT_PROMPT asks for `concepts` (1-3 kebab-case TOPIC labels) with
an explicit reuse-over-coinage instruction — labels must cluster,
since synthesize_concepts only materializes groups of >=2.
- parseAtomsResponse validates labels (kebab regex, max 3, drop
invalid; empty -> undefined).
- The putPage frontmatter write stamps `concepts` alongside lesson /
source_quote.
Tests: 4 parse cases + an end-to-end regression that goes extractor ->
real frontmatter -> synthesize_concepts' OWN DB query path -> concept
page. The existing tests fed synthesize via the `_atoms` seam, which is
exactly how this gap survived.
Validated in production ahead of this PR by stamping the same shape
externally: the next synthesize_concepts run wrote 33 concept pages
(T2=7/T3=26) from 60 stamped atoms, zero failures.
Co-authored-by: 陈源泉 <chenyuanquan@chenyuanquandeMac-mini.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(drift): skip node_modules/dist/build in multi-source drift walk
The drift walker recursed into node_modules (50k+ files in RN/Astro repos),
exhausting the time budget before completing, so multi_source_drift always
reported 'walk hit limit/timeout' on real projects. Skip heavy non-content
dirs + add a deadline check on directory descent.
* fix(integrity): skip inline-code spans + [Source:] citations in bare-tweet detection
Recipe/doc pages that show the CORRECT citation format inline (e.g.
`Tweeted about {topic} [Source: X, @handle, date]`) were false-flagged.
The fenced-code skip didn't cover inline backticks; add inline-code
stripping + an explicit-citation exemption.
---------
Co-authored-by: Son Le <tuanson1200@gmail.com>
splitLargeNode can only break up a node that exposes a `body` with >= 2
named children. A node without one -- a giant object/array literal, a single
huge assignment, a massive template literal -- is emitted whole. On real
source that yields a chunk far larger than the embedder's context window; the
embedder then rejects it ("input exceeds context length") and it is never
embedded. Example: a 372 KB service file produced 113 chunks, one a single
281 KB (~70k-token) node -> permanently unembedded.
Add a final safety-net pass (capOversizedChunks) that recursively re-splits
any chunk over a token budget (default 2000, configurable via maxChunkTokens),
with a hard character split as a last resort for no-whitespace content
(minified one-liners). Normal files are untouched.
Verified: that 372 KB file now yields 188 chunks, max ~1.5k tokens, zero
oversized; a small file is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gbrain think printed "## Gaps" twice: the synthesis prompt asked the model for a Gaps section inside the answer body AND a separate structured gaps array, then both render paths printed both — the CLI human output (src/commands/think.ts) and the --save page (persistSynthesis in src/core/think/index.ts).
Make the structured gaps array the single source. The prompt now routes gaps into the array, not an answer-body section. New exported stripGapsSection(answer) defensively removes any "## Gaps" section a model still emits (any heading level, case-insensitive, bounded by the next same/higher heading); both render sites call it, so the dedup is structural rather than dependent on the model obeying the prompt.
Adds test/think-gaps.test.ts (hermetic): strip helper across heading levels / case / no-section / mid-document / false-match, the one-render-only repro, and the prompt contract.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
PGLite can return zero rows from INSERT ... ON CONFLICT DO UPDATE ...
RETURNING in no-op/trigger edge cases. The previous code called
rowToPage(rows[0]) unconditionally, so rows[0] was undefined and
rowToPage threw "undefined is not an object (evaluating 'row.deleted_at')",
which aborted the import and silently skipped the file during sync.
getPage() already has the empty-rows guard; putPage() was missing the
parallel one. The row was in fact written by the upsert, so re-read it
via getPage() instead of crashing. On a real monorepo index this
recovered ~19% of files (985/5148) that were failing to embed.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
`findByTitleFuzzy` on both `postgres-engine.ts` and `pglite-engine.ts`
has no `source_id` filter and no `deleted_at IS NULL` filter. `tryFuzzyMatch`
in `src/core/entities/resolve.ts` got both of those filters via #1436
(v0.41.13.0) for exactly the reasons that apply to its sibling here:
fuzzy resolution can suggest cross-source slug candidates that the
caller then silently drops at the FK filter (or worse, picks a
soft-deleted page).
This is the missing twin of #1436. In multi-source brains, the live-mode
auto-link resolver invoked from `put_page` (`operations.ts:937`) calls
`engine.findByTitleFuzzy` with no scope. When two sources contain pages
with similar titles (`people/alice-example` on `source-a`,
`people/alice-other` on `source-b`), the fuzzy lookup can return the
wrong-source slug, which then fails the downstream `allSlugs` /
`addLink` FK filter — the link silently doesn't get created, and from
the caller's view the resolver "failed" even though the page existed
under the right source.
Reproducible with a 2-source PGLite setup + identical-title pages on
both sides; the fuzzy call returns a slug whose `source_id` doesn't
match the put_page caller's source.
- 2-source brains: auto-links between same-title-different-source pages
now resolve under the caller's source instead of the wrong neighbor.
- Soft-deleted pages can no longer be returned as fuzzy candidates
(mirroring the resolve.ts fix from #1436).
- 1-source brains: no behavior change. `sourceId` is optional; when
omitted the SQL takes the pre-existing unscoped path.
- `engine.ts`: add optional 4th `sourceId` param to the
`findByTitleFuzzy` interface + JSDoc explaining the scope semantics.
- `postgres-engine.ts` / `pglite-engine.ts`: implement the param via a
conditional SQL branch that adds `AND source_id = $N AND
deleted_at IS NULL` when `sourceId` is set; existing query path
unchanged when omitted.
- `link-extraction.ts`: add optional `sourceId` to `makeResolver` opts,
forward to `findByTitleFuzzy` in step 3 of the resolve chain.
- `operations.ts`: pass `opts?.sourceId` to `makeResolver` from the
live-mode put_page resolver (the place that already knows the
caller's source).
- New unit tests in `test/link-extraction.test.ts` (2 cases):
- `opts.sourceId` is forwarded to `findByTitleFuzzy` when set.
- `opts.sourceId` omitted → `findByTitleFuzzy` receives `undefined`
(back-compat).
- `bun run typecheck` clean.
- `bun test test/link-extraction.test.ts test/entity-resolve.test.ts
test/operations.test.ts test/extract.test.ts` — 145/145 pass.
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
llama.cpp's llama-server rejects /v1/embeddings requests with more inputs
than its launch --batch-size (default 32): "batch size 100 > maximum allowed
batch size 32". gbrain sends batches of 100, so any page with >32 chunks fails
to embed, and embed --stale then trips the Postgres statement_timeout retrying
the doomed batches. The existing token-based protection (max_batch_tokens)
can't bound item count — N tiny chunks fit under any token budget.
Add an optional max_batch_items count cap to EmbeddingTouchpoint, enforced as a
hard re-split after the token split in embed(), and set it to 32 on the
llama-server recipe (replacing no_batch_cap: true, which wrongly assumed
llama.cpp has no per-request item cap). A declared item cap also suppresses the
missing-max_batch_tokens startup warning.
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
The cron daily briefing writes 90_Briefings/<date>.md, which gets
re-ingested on the next sync and then dominates tomorrow's
getRecentSalience output as pure self-reference (observed: top
result score 0.9956, everyone else clustered at 0.587).
Filter `p.slug LIKE 'briefings/%'` out of getRecentSalience in both
the PG and PGLite engines. Suppressed by default; callers can still
opt in by passing `slugPrefix: 'briefings/'` (or `--kind briefings/`
from the CLI). search and list_pages are unaffected.
Co-authored-by: CTO <cto@timelycare.local>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
On Windows, `core.autocrlf=true` is the default and SKILL.md files are
checked out with CRLF line endings. `extractTriggers` used regexes
anchored to `\n` (`/^---\n.../` and `/^triggers:\s*\n.../`), which
never matched `\r\n`, so the parser returned `[]` for every skill.
Result: `gbrain doctor --fast --json` on Windows reported every skill
not in `OVERLAP_WHITELIST` (39 of 42) as a false `mece_gap` warning —
even though `skill_conformance` in the same run reported "42/42 skills
pass". CI runs Ubuntu-only so the divergence never surfaced.
Fix: normalize CRLF → LF at the top of `extractTriggers`. Single-line
change preserves existing LF behavior. Function is now exported so the
test can target it directly.
Tests: added `describe("extractTriggers")` block covering LF input,
CRLF input (regression case), missing frontmatter, missing triggers
field, and quote-stripping. All 30 tests in `check-resolvable.test.ts`
pass.
Verified locally on Windows: `gbrain doctor --fast --json` now reports
`resolver_health: ok, 42 skills, all reachable` (health_score 90 → 95).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Qwen3-Embedding family on Ollama supports Matryoshka truncation via the
'dimensions' field on /v1/embeddings. Without this passthrough, gbrain
ignores user-selected reduced dims and the provider returns its native
size, causing dim-mismatch errors against brains configured for narrower
widths (e.g. existing 1536-dim brains).
Matches by bare name 'qwen3-embedding' or any tag variant
'qwen3-embedding:0.6b' / ':4b' / ':8b'.
Native dims: 0.6B=1024, 4B=2560, 8B=4096. All MRL-truncatable.
5 new tests; full AI suite 137/137 green.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Takeover of #2719 (fork head; rebased onto origin/master).
- writeProposed now writes both best.md (current-best pointer) and
proposed.md (stable human-review artifact); returns the proposal path.
- Orchestrator reports the real proposed.md path for accepted --no-mutate runs.
- Tutorial updated; llms bundles regenerated (no content drift — tutorial
is not inlined in the bundle).
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Ziyang Guo <121015044+RerankerGuo@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Ports #3015 by @jdewoski-cmd onto current master:
- src/core/orphan-policy.ts centralizes the orphan-reporting exclusion
convention so `gbrain orphans`, doctor's orphan_ratio, and both engines'
getHealth orphan_pages can no longer drift.
- getHealth stale_pages now uses the link-extractor stale watermark
(countStalePagesForExtraction) so health agrees with what `gbrain extract
--stale` will actually process.
- New `gbrain maintain` command: dry-run by default, `--safe` applies only
the conservative runbook actions (DB-backed stale extraction + source-scoped
dream cycles for doctor cycle_freshness findings), `--json` for structured
before/action/after reports. Frontmatter mutations, schema-pack upgrades,
and semantic hub links stay review-only by design.
Changed from the original PR: the shared defaults carried slugs specific to
the contributor's own brain ('josa-secrets/', '*-ga4-property-id.md',
'*-josa-test', literal 'welcome'/'untitled' fixtures). Global defaults now
carry only GBrain-wide conventions; brain-specific exclusions move to a new
per-brain config plane the policy reads through loadOrphanPolicyOverrides:
gbrain config set orphans.exclude_prefixes "my-private-folder/,archive/"
gbrain config set orphans.exclude_slugs "some-one-off-page"
Both engines' getHealth and the orphans command thread the overrides;
tests cover the neutral defaults, the override plane, and health parity.
Also registered `maintain` in CLI_ONLY_SELF_HELP so `gbrain maintain --help`
reaches the command's own usage block.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: jdewoski-cmd <jdewoski@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The generic wikilink pass (issue #972) forwarded the raw literal to
resolveBasenameMatches, whose index is keyed by final path segments —
so [[notes/struktura]] (any dir outside DIR_PATTERN) silently produced
zero edges from `extract links --source db` and put_page auto-link,
while the FS extractor resolves the identical content (resolveSlugAll
strips the dirname before its basename lookup).
Query by the literal's final segment, then keep only matches whose slug
ends with the written path — [[notes/struktura]] can resolve to
vault/notes/struktura but never attach to wiki/struktura. Bare literals
are untouched. Flag-gated by link_resolution.global_basename as before.
Co-authored-by: YMYD <53603073+OJ-OnJourney@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Closes#769. Every re-embed pass clobbered code-chunk metadata
(language, symbol_name, symbol_type, start_line, end_line,
parent_symbol_path, doc_comment, symbol_name_qualified) to NULL,
disabling code-def queries across thousands of indexed chunks.
Two complementary fixes:
embed.ts — three re-upsert call sites (embedPage, embedAll
non-stale, embedAllStale autopilot path) build ChunkInputs from
loaded chunks; they were stripping the 8 metadata fields. New
preserveCodeMetadata helper threads those fields through
consistently. Integrated cleanly with v0.34.4.0's cursor-paginated
--stale hardening — the wrap sits inside the worker function
between embedBatchWithBackoff and engine.upsertChunks.
postgres-engine.ts + pglite-engine.ts — upsertChunks ON CONFLICT
clause OVERWROTE metadata columns from EXCLUDED. Asymmetric vs the
embedding/embedded_at columns which already used a chunk_text-gated
CASE pattern (re-chunk → trust EXCLUDED, re-embed → COALESCE
preserve). Applied the same pattern to all 8 metadata columns.
Three regression tests in test/embed.serial.test.ts cover --stale
(autopilot), --all, and --slugs paths. Each loads a chunk with
full metadata, runs runEmbed, and asserts engine.upsertChunks
receives the metadata round-tripped. Coexists with master's D5
embedBatchWithBackoff test block.
Backfill required after deploy: \`gbrain sync --strategy code
--force --source <id>\` per code source to re-populate metadata via
the chunker. Without backfill, existing NULL columns stay NULL —
re-embed alone never produces metadata, only the chunker does.
Originally landed as part of PR #768 (the wave that bundled #767 +
fix; this PR carries the #769 fix alone with no scope overlap.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
* feat(synopsis): tail-truncate documentText for small-model chat handlers
Small local chat models (Gemma 4 E2B, Qwen3 4B) get dramatically
slower on long contexts even at 131K declared windows. A 73K-char
page synopsis on Gemma 4 E2B takes 60-120s, exceeding the worker's
default 30s `lockDuration` and tripping `lock-lost` errors.
Add `SYNOPSIS_DOC_MAX_CHARS` env-overridable cap (default 32768
chars, ~8K tokens) applied in `buildUserPrompt`. Truncate the TAIL
so the head (title, frontmatter, intro) preserves the document-level
anchor the synopsis needs.
Anthropic Haiku is unaffected at this cap; bump via
`GBRAIN_SYNOPSIS_DOC_MAX_CHARS` for frontier models that want
richer document anchoring.
Belt-and-suspenders companion to commit 0aaff691 (--lock-duration
flag on the worker). Combined: bumping lock TTL gives the handler
more time, AND truncating doc cap makes the handler complete faster.
Either alone helps; both together get the synopsis backfill running
reliably on small local LLMs.
Verified: real 1383-chunk personal brain backfill at
GBRAIN_SYNOPSIS_MODEL=lmstudio:google/gemma-4-e2b +
GBRAIN_SYNOPSIS_DOC_MAX_CHARS=16384 +
`gbrain jobs work --concurrency 4 --lock-duration 300000`
transitions from "lock-lost on every transcript page" to "no
deaths, no stalls, steady throughput."
RECOVERY REBUILD 2026-05-26 of original ac213aa6.
* fix: fold SYNOPSIS_DOC_MAX_CHARS into corpus_generation hash
Codex review of #1427 flagged that changing GBRAIN_SYNOPSIS_DOC_MAX_CHARS
shifts the synopsis prompt + downstream embeddings for long documents
but was NOT folded into the computeCorpusGeneration hash. Pages
re-embedded with a different cap would retain the same
corpus_generation, defeating the v0.40.3.0 D27 P1-5 cache invalidation
contract.
Three changes:
1. Export SYNOPSIS_DOC_MAX_CHARS from src/core/page-summary.ts
2. computeCorpusGeneration accepts optional synopsisDocMaxChars param.
When set, folded into hash via '|doc_cap=<N>'. Omitted for
non-synopsis modes (title / none don't consult the cap) so existing
pre-PR caches stay valid for those.
3. Service-layer call sites (2 in contextual-retrieval-service.ts)
pass SYNOPSIS_DOC_MAX_CHARS when attemptMode/resolution.mode is
per_chunk_synopsis, undefined otherwise.
4. import-file.ts inline path passes undefined (per_chunk_synopsis
refused upstream there).
One-time effect: per_chunk_synopsis pages re-embedded post-PR get a
NEW corpus_generation including the cap. v0.40.3.0 query_cache.page_generations
contract auto-invalidates cached query results on first re-embed.
Future cap changes track correctly.
Addresses codex review P2 on PR #1427.
---------
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
Synth pages written by dream synthesize currently open straight into
detail (quotes, cross-references) with no framing, so a reader who
lands on the page later — without the source transcript in front of
them — has no way to tell what it's about without reading the whole
thing.
Add OUTPUT POLICY item 5: every new page's body must open with a 2-3
sentence self-contained summary a reader unfamiliar with the source
conversation could understand on its own, before any quotes or detail.
Long-lived minion workers can outlive DB-backed model config changes. Refresh the AI gateway before gateway-backed handlers run so queued cycle/propose_takes work does not fall back to a stale Anthropic default when the operator configured another provider.
Also record the active gateway chat model in propose_takes budget/proposal metadata instead of hardcoding claude-sonnet-4-6, and keep provider:model IDs intact for budget pricing.
Regression coverage verifies queued worker refresh, propose_takes model metadata, nested provider IDs, skipFence threading, and the updated autopilot signal source guard.
Co-authored-by: maxpetrusenkoagent <[REDACTED EMAIL]>
Setting `models.tier.deep anthropic:claude-opus-4-8` silently disabled
think and auto_think: the Anthropic recipe's chat allowlist stopped at
Opus 4.7, the tier-resolved model never joined the extended set that
assertTouchpoint's contract promises for config-chosen models, and the
resulting probe failure was stamped NO_ANTHROPIC_API_KEY — sending the
operator to debug env/keychain when the fix was the model id. Three
fixes, one per layer:
- recipes/anthropic.ts: add claude-fable-5, claude-opus-4-8, and
claude-sonnet-5 to chat models; claude-sonnet-5 to expansion models.
- gateway.ts reconfigureGatewayWithEngine: resolve all four tiers and
register the results as extended models, honoring the documented
contract for models.default / models.tier.* (model-resolver.ts
docstring). A tier-only model now validates like a chat/expansion one.
- think/index.ts: when the gateway client can't be built, re-probe and
label honestly — MODEL_NOT_USABLE:<reason> for unknown_model /
unknown_provider, NO_ANTHROPIC_API_KEY only for the actual missing-key
case; the stub answer carries the probe detail and fix hint.
Tests: recipe-list presence pins; a new gateway-tier-extended-models
suite proving a fictional tier model validates post-reconfigure (and an
unconfigured one still doesn't); think-pipeline coverage for the honest
label (unknown_model beats missing-key even keyless); the existing
non-explicit bogus-provider test updated from the old catch-all label to
the honest one (no-throw contract unchanged).
Verified live: a brain with tier.deep=claude-opus-4-8 had think degrade
to gather-only with the misleading key warning; with this change the
probe passes and synthesis runs.
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(dream): --once for one-shot phase runs without toggling config gates
Fixes the "toggle enabled true, run, toggle back to false" workaround
that gbrain doctor's extract_atoms_backlog message implicitly
recommends and that #2860's reporter had to script around: with an
external orchestrator running `gbrain dream --phase patterns` on a
cadence outside the autopilot, the only way to run patterns once was
`config set dream.patterns.enabled true` -> run -> `config set ...
false`. A crash between steps left the flag stuck true, and the
autopilot (which polls the same flag) re-enqueued patterns every
cycle -- 119 LLM jobs / ~$400 over 24h before it was caught.
Root cause: `--phase X` only controls which phase FUNCTION cycle.ts
calls; it does not bypass that phase's own `dream.<phase>.enabled` /
`cycle.<phase>.enabled` config read. Each gated phase (patterns,
synthesize, conversation_facts_backfill, enrich_thin, skillopt) reads
its enabled flag internally and skips regardless of how the phase was
selected -- confirmed by reading each phase module, not assumed.
extract_atoms/synthesize_concepts are a DIFFERENT mechanism entirely
(pack-declaration via packDeclaresPhase, not a config .enabled read)
and already have a working one-shot escape hatch: `--drain`. The
existing doctor message for extract_atoms already says `--phase
extract_atoms --drain --window 120`, so no doctor text needed
updating there -- verified by reading src/commands/doctor.ts directly
rather than assuming the paraphrase in the issue was literal.
Design: `gbrain dream --phase <name> --once`. Requires an explicit
--phase (bare --once is a usage error, exit 2) so it can never
force-enable every disabled phase at once in a full/default cycle --
that would recreate the same unbounded-spend risk the flag exists to
prevent. Threaded through CycleOpts as `onceForPhase?: CyclePhase`
(the literal phase name, not a boolean) so the bypass can never leak
to a phase other than the one named, even if a future programmatic
caller passes a wider `phases` array than the CLI does. Never reads
or writes config -- the phase still evaluates its .enabled gate every
call; --once only overrides the boolean OUTCOME for that one
invocation, mirroring the existing --unsafe-bypass-dream-guard /
--input precedents (stderr warning at the bypass point, no new
config-touching code path).
Rejected alternatives (documented per task instructions):
- Making explicit --phase X always bypass .enabled: breaking change
for existing crons that rely on the disabled flag as a cheap no-op;
an upgrade would silently start running LLM/write phases.
- A new subcommand: adds a whole dispatch/help/arg surface that
internally routes through the same override anyway.
- Extending --once to also bypass packDeclaresPhase for
extract_atoms/synthesize_concepts: conflates two different gating
mechanisms (config toggle vs. pack membership) under one flag;
extract_atoms already has --drain, which is purpose-built for its
batched/windowed execution model.
Design was cross-validated by an independent second-model review
(external design consultation) before implementation; its
recommendation to also update the extract_atoms doctor message to
`--once` was NOT adopted because that phase has no .enabled gate to
bypass -- doing so would be a documented no-op, contradicted by
reading src/commands/doctor.ts:3264 directly.
Tests: 9 new (structural CLI-flag wiring in dream-cli-flags.test.ts;
a real PGLite E2E test in dream-patterns-pglite.test.ts proving the
bypass fires AND that dream.patterns.enabled is never written; 4
runCycle-level tests in cycle.serial.test.ts proving onceForPhase
does not leak across phases). Verified 6 of 9 fail against the
pre-fix source (via git stash of source-only changes) to confirm
they're meaningful regressions, not tautologies.
Closes#2860
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(dream): --help short-circuits before --once usage validation
Codex review finding (P2): `gbrain dream --help --once` (no --phase)
called process.exit(2) from the new --once usage-error check inside
parseArgs before runDream's documented IRON RULE ("--help
short-circuits BEFORE any engine-bearing work") ever got a chance to
run -- parseArgs computes ALL its validations unconditionally before
runDream checks opts.help. Repo precedent for this ordering already
exists as a pinned regression test (test/dream.test.ts's "--help
--source whatever prints help and exits 0").
Fix: compute wantsHelp once in parseArgs and exempt the --once
validation when it's set, mirroring that precedent. Added the same
class of pinned tests here: bare `--once` still exits 2 with the
usage hint, `--help --once` prints help and exits 0, and a real
--phase patterns --once run against a PGLite engine proves the
bypass actually fires (falls through to insufficient_evidence
instead of disabled) without writing dream.patterns.enabled. Also
fixed the structural test in dream-cli-flags.test.ts that asserted
the exact pre-fix guard-condition source text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(dream): --once must require an EXPLICIT --phase, not a derived one
Codex review finding (P3): the --once validation checked the derived
`phase` value, but `phase` gets defaulted implicitly by --input
(implies --phase synthesize) and --drain (implies --phase
extract_atoms) BEFORE that check ran. So `gbrain dream --input <f>
--once` and `gbrain dream --drain --once` both slipped past the
"explicit --phase required" contract silently -- and --once became a
true no-op in both cases: --drain returns from runDream before
onceForPhase is ever read (the drain path doesn't call runCycle at
all), and --input already bypasses the synthesize enabled-gate on its
own via the existing opts.inputFile check, so onceForPhase would
never even be consulted.
Fix: capture `phaseWasExplicit = phaseIdx !== -1` at the very top of
parseArgs, before the --input/--drain defaulting blocks run, and
validate --once against that instead of the derived `phase`. Updated
the usage-error message and --help text to say "an explicit --phase"
so a user hitting this understands why `--input ... --once` doesn't
count.
Tests: 2 new pins in test/dream.test.ts exercising runDream directly
(--input <file> --once exits 2; --drain --once exits 2), plus a
structural test in dream-cli-flags.test.ts pinning that
phaseWasExplicit is captured before both implicit-defaulting blocks.
Updated the two existing structural/behavioral tests whose literal
guard-condition / error-message assertions changed shape.
Verified: dream-cli-flags.test.ts 27/27, dream.test.ts 31/31,
typecheck clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: masashiono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Adds NVIDIA NIM / API Catalog as a first-class OpenAI-compatible AI recipe:
- chat via nvidia/nemotron-3-super-120b-a12b (conservative capability
claims: no tools, no subagent loop until proven)
- hosted embedding models incl. nvidia/llama-nemotron-embed-1b-v2 with
Matryoshka-style dimension overrides (1024/1280/1536/2048) and fixed
natural dims for the other catalog models
- asymmetric input_type mapping (document -> passage, query -> query) via a
gateway compat fetch shim, since the generic openai-compatible recipe
cannot infer that provider-specific requirement
- base URL https://integrate.api.nvidia.com/v1 verified live (OpenAI-shaped
/v1/models, all five recipe model ids present in the catalog)
Changed from the original PR: dropped the recipe's custom resolveAuth — it
duplicated defaultResolveAuth's Authorization-Bearer behavior exactly and
violated the IRON RULE that only Azure overrides resolveAuth
(test/ai/recipes-existing-regression.test.ts). NVIDIA_API_KEY now flows
through defaultResolveAuth via auth_env.required, and the recipe test pins
resolveAuth === undefined + the default Bearer resolution + the missing-key
AIConfigError. Also scrubbed a private downstream-agent name from ported
comments per the repo privacy rule.
Takeover of #2965 by @ravehorn.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: SAGE Codex <codex@sage.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
getLastSeen had no upper date bound, so a future-dated chronicle event (a
scheduled calendar-event, a planned milestone) became the entity's "last
seen" date; finalizeLastSeen's Math.max(0, ...) then clamped the negative
delta, reporting days_ago: 0 -- the entity reads as seen-today. Recording
future events is intended (eligibility ELIGIBLE_TYPES includes calendar-event);
the reader just needs to stop counting them as "seen".
Bound the query to te.date <= COALESCE(asof, current_date) in both engines,
mirroring getOnThisDay's existing te.date < target bound. asof now reaches
the WHERE clause (previously it only reached finalizeLastSeen), so as-of
time-travel is honored for the date filter too.
Regression test added: an entity with past events plus a future event -> last
seen returns the most recent PAST event, not the future one; and as-of after
the future date lets it through. Fails before, passes after.
putPage's INSERT used COALESCE(<chunkerVersion>, 1), so callers that don't
supply chunker_version (no MCP/subagent caller does — it's internal metadata)
landed new pages at version 1. Dream subagents write through putPage directly,
so their pages got v1 and doctor's contextual_retrieval_coverage check flagged
them as "older chunker_version" forever, even though they were chunked and
embedded with the current chunker.
Default the INSERT to MARKDOWN_CHUNKER_VERSION on both engines. The ON CONFLICT
UPDATE still COALESCE-preserves an explicitly supplied version. Add an
engine-level regression test.
embedStaleForSource rebuilds each page's chunks as a merged ChunkInput[]
carrying only five fields (chunk_index, chunk_text, chunk_source,
embedding, token_count), while upsertChunks writes the metadata columns
as EXCLUDED.<col>. Any page containing at least one stale chunk
therefore has ALL its chunks' metadata reset on the next embed-stale
pass:
- image chunks flip modality 'image' -> 'text' and disappear from the
cross-modal image search arm permanently (it filters
modality='image'), while keeping their embedding_image vector — the
data looks intact but is unreachable;
- code chunks lose language, symbol_name, symbol_type,
symbol_name_qualified.
The read side compounds this: rowToChunk never returned modality, so a
correct merge was impossible without also extending the Chunk shape.
Fix: expose modality on Chunk/rowToChunk and carry modality, language,
and the symbol fields through the merge. embedding_image is deliberately
not carried — upsertChunks already COALESCEs it server-side.
Repair for affected brains: UPDATE content_chunks SET modality='image'
WHERE chunk_source='image_asset' AND embedding_image IS NOT NULL.
The new test seeds a mixed page (settled image chunk + stale text chunk
with symbol metadata) and asserts both survive an embedStaleForSource
pass; it fails on master.
Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
The submission-time backpressure cap counts all waiting (name, queue)
rows regardless of which source a job targets. On a multi-source brain
this makes per-source submissions with maxWaiting: 1 mutually exclusive:
while one source's sync sits waiting, every other source's freshness
sync coalesces into that row and never runs. The dispatch log shows the
starved source 'dispatched' each interval (queue.add returns the other
source's waiting row), so the starvation is invisible unless you notice
sources.last_sync_at falling behind — we found a secondary source 29
hours stale on a 5-minute freshness interval.
Fix: when the submitted data carries a string sourceId, key the advisory
lock, the waiting count, and the coalesce target on it. Submissions
without sourceId keep the existing single-scope behavior, so
single-source brains and non-sync jobs are unchanged.
The new test asserts same-source submissions still coalesce while a
different source gets its own row and its own cap; it fails on master.
Co-authored-by: Forge (Ron) <forge@zsimovan.dev>
* fix(sources): validate --path is a git repo at registration time (#2707)
`sources add --path <dir>` accepted any existing non-git directory with
zero validation, deferring the failure to the first `gbrain sync` ("Not
inside a git repository: ..."). By the time that surfaces, the source
has been silently stale for however long nobody read the sync logs.
Add a registration-time check (git-remote.ts:isInsideGitRepo, mirroring
sync.ts's discoverGitRoot walk-up so subdir-of-git-repo sources still
pass) that rejects an existing-but-non-git --path directory with an
actionable error pointing at `git init && git add -A && git commit`.
Non-existent paths are unaffected (out of scope — different, pre-existing
failure mode) and `--force` opts out for callers who want to register
before git-init exists.
This is registration-time validation ONLY — it never auto-`git init`s
the directory, preserving the consent boundary #2967 established for
sync-time self-heal (a --path source is the user's own external
directory; gbrain must not mutate it without explicit ask).
Also documents the git requirement (docs/guides/multi-source-brains.md),
including the "files must be committed, not just present" gotcha and
that a stale/unreachable sync anchor already self-heals on plain
`gbrain sync` (verified manually against HEAD — no reset-anchor command
needed).
* fix(sources): require a committed HEAD + shell-quote remediation cmd (codex round 1)
Codex review round 1 on #2707 found two real gaps:
1. isInsideGitRepo alone accepts a `git init`ed-but-never-committed
directory (rev-parse --show-toplevel succeeds with no HEAD), so
registration would still pass a source that fails sync's own
"No commits in repo ... Make at least one commit before syncing."
Add hasGitCommits (git rev-parse HEAD) as a second required check.
2. The remediation command in the error message interpolated the raw
path unquoted — spaces, $(), backticks, etc. would break or, worse,
execute unintended shell syntax if pasted. POSIX single-quote it
(mirrors src/commands/connect.ts:shellQuote; duplicated locally
rather than imported, since commands/ depends on core/ not the
reverse).
* fix(sources): require tracked content in HEAD, not just a resolvable HEAD (codex round 2)
Codex review round 2 P1: hasGitCommits (rev-parse HEAD) accepted a repo
with an empty commit (git commit --allow-empty) followed by untracked
files — HEAD resolves fine (to git's well-known empty-tree object), so
registration passed, but the first sync would "succeed" importing
nothing and then silently never notice the untracked files change. The
exact same gap applied to an untracked subdirectory of an otherwise-
real git repo (monorepo case).
Replace hasGitCommits with hasTrackedContent (`git ls-tree HEAD -- .`,
non-recursive — one entry is enough, no need to walk the whole
subtree). `-C path` + pathspec `.` scopes correctly to both a repo
toplevel and a subdirectory-of-a-repo source, and an empty tree lists
zero entries where a bare `rev-parse HEAD` would still succeed. Also
subsumes the "no commits at all" case hasGitCommits covered (ls-tree
on an unborn repo fails the same way), so this is one check instead
of two.
Updated the error copy and docs/guides/multi-source-brains.md to match
what's actually verified now.
* fix(sources): O(1)-output tree-emptiness probe, avoid maxBuffer overflow (codex round 3)
Codex review round 3 found the round-2 `git ls-tree HEAD -- .` listing
buffers the whole (non-recursive) tree — a real repo with ~17-20K
directly-tracked entries exceeds execFileSync's default 1 MiB
maxBuffer, throws ENOBUFS, and the catch-all incorrectly rejects a
perfectly valid registration.
Replace the listing with `git rev-parse --verify HEAD:./` (resolves
the tree object for `path` specifically, correct for both toplevel and
subdirectory sources same as before) compared against git's canonical
empty-tree SHA-1 (4b825dc6...) — a fixed ~40-byte read regardless of
how many entries the tree has, structurally immune to this class of
bug rather than just raising the threshold. Added a 300-file
regression test locking this in.
Declined a second round-3 finding (P1: reject a tree if ANY untracked
file exists anywhere under the path, not just when the tree is
entirely empty) — untracked files never being synced is standard,
existing git-source behavior throughout this codebase (identical for
--url managed clones), not a bug specific to this validation. Enforcing
zero-untracked-files at registration would reject ordinary repos with
gitignored build output, .DS_Store, editor swapfiles, etc. Out of
scope relative to what #2707 actually asks for (a directory with real,
committed content that will sync) and how every other git source in
this system already behaves.
* fix(sources): derive empty-tree OID per repo instead of hardcoding SHA-1 (codex round 4)
Codex review round 4 P2, confirmed by directly testing against a
`git init --object-format=sha256` repo: the hardcoded SHA-1 empty-tree
constant only matches SHA-1 repositories. An empty SHA-256 repo's real
empty-tree OID is a different (64-char) hash, so the SHA-1 comparison
silently mismatched and let an empty/untracked SHA-256 source through
— exactly the case this validation exists to catch.
Replace the constant with `emptyTreeOid()`: `git hash-object -t tree
--stdin < /dev/null` computed in the target repo's own context, so it
returns the correct empty-tree OID for whichever object format that
repo actually uses, without gbrain needing to know or care which one.
Added gated regression tests (git 2.29+ / --object-format=sha256,
test.skipIf on older git) for both the empty-repo-rejected and
real-content-registers-fine cases.
Converging here (4 review rounds; this is the last outstanding
finding from round 4, and round 4 raised only this one issue).
* fix(test): --force the incidental non-git second source in #1434 routing test (#2707)
CI caught a real regression from this PR's registration-time git
validation: test/sync-sole-non-default-routing.test.ts's "2+ non-default
sources" case registers a bare mkdtempSync temp dir (no git init) as a
second source purely to have 2 sources present — the directory's content
was never meant to be exercised, only its existence as a distinct
local_path. #2707's new validation correctly rejects that dir at
registration time, since nothing else in the test suite told it
otherwise.
--force is the right fix, not adding unnecessary git-init/commit
boilerplate to secondRepo: it documents that this specific registration
intentionally doesn't care about git-validity, matching what a real
caller opting into the legacy lenient behavior would do.
Verified: the specific test (3/3 pass), plus every other test file in
the repo using `sources add --path` (sources.test.ts, sources-ops.test.ts
already covered by the PR's own commits; repos-alias.test.ts,
sync-cost-gate.serial.test.ts — 11/11 pass, no similar fixture gap).
typecheck clean, verify 31/31.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Same class as #3019's v123 fix; the guard test added there flagged this
within one push. Route the notice through process.stderr.write.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A single markdown page whose compiled_truth exceeds Postgres's hard
1,048,575-byte tsvector cap made update_page_search_vector() throw
"string is too long for tsvector" INSIDE the pages UPSERT transaction.
Not a per-file ledger entry — a transaction abort. The whole source's
sync checkpoint stayed pinned (Sync BLOCKED) until the oversized file was
fixed or manually skipped, even though every other file in the run
imported fine. --retry-failed re-failed the same files every run; the
3-consecutive-failure auto-skip eventually moved past them, but for a
scheduled collector that meant hours of blocked cycles per oversized
file, per source.
Root cause: pages.search_vector indexed compiled_truth — the unbounded
whole-page body — even though it's write-only dead weight for actual
search. searchKeyword() (postgres-engine.ts / pglite-engine.ts) ranks and
queries content_chunks.search_vector exclusively (Cathedral II Layer 3,
chunk-grain, already populated separately from compiled_truth via
chunking at import time, and well under the tsvector cap since
chunkText() targets embedding-sized pieces). Verified directly: no
pages.search_vector / bare search_vector read appears anywhere outside
this trigger's own definition and the reindex/backfill machinery that
maintains it.
Fix: v124 migration recreates update_page_search_vector() without
compiled_truth — title + timeline stay (both naturally small), so the
column keeps carrying some signal rather than going fully inert. Updated
in lockstep (documented contract, see reindex-search-vector.ts's own
comment): migrate.ts's new v124, reindex-search-vector.ts's
recreatePagesFn, and the fresh-install baselines in pglite-schema.ts +
src/schema.sql (regenerates schema-embedded.ts via `bun run
build:schema`). No backfill: existing rows keep whatever search_vector
they already computed until their next UPDATE — harmless, since nothing
reads this column, and the brains that actually hit this bug never
successfully wrote a value for the oversized page in the first place.
Considered (from the issue) and rejected: truncating compiled_truth to
fit under the cap. Silent, position-dependent recall loss, and the byte
cap doesn't line up cleanly with any natural character/token boundary
for UTF-8 content. content_chunks.search_vector already gives full,
untruncated chunk-grain coverage for large pages — truncating a
now-redundant whole-page vector would trade a real bug for a subtler one.
## Test plan
- New test/page-search-vector-overflow.test.ts: a >1MB page (genuinely
diverse tokens — a repetitive lorem-ipsum-style fixture does NOT
reproduce this bug, since to_tsvector's cap is on its DEDUPLICATED
output size, not raw input length) now imports successfully instead of
throwing; remains keyword-searchable via the chunk-grain path; a normal
page's search_vector still carries title signal (not fully inert).
Verified the test is meaningful both directions: fails with the exact
reported error on the pre-fix trigger (git-stashed the fix, reran,
confirmed byte-for-byte match: "string is too long for tsvector
(2684620 bytes, max 1048575 bytes)"), passes with the fix restored.
- Updated fts-language-migration.serial.test.ts: removed an assertion
that configurable_fts_language (v123) is LATEST_VERSION — that was only
ever true until the next migration landed; the codebase's own pattern
elsewhere for this (migrate.test.ts) uses toBeGreaterThanOrEqual, not
exact-match, for exactly this reason.
- bun run typecheck clean, bun run verify 31/31 green.
- test/page-search-vector-overflow.test.ts (3/3),
test/reindex-search-vector.serial.test.ts,
test/fts-language-migration.serial.test.ts, test/migration-v120.test.ts,
test/sync.test.ts, test/bootstrap.test.ts, test/migrate.test.ts — 254
total, 0 fail, no regressions.
## Design consultation
Investigated jointly with masa-codex (async design review) before
implementing — their read of the codebase (content_chunks.search_vector
already covers keyword search; pages.search_vector's compiled_truth feed
is the only overflow-prone, effectively-dead write) matched independent
verification and shaped the "remove from trigger" fix over the issue's
alternative options (chunk-grain rebuild — largely already exists; input
truncation — rejected above; ledger-entry-only — insufficient alone,
since the page upsert failing in the same transaction also loses the
chunk write, so checkpoint advancing without this fix would make the
content permanently unsearchable, not just delayed).
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
resolveGbrainCliPath() (both copies — src/commands/autopilot.ts and the
inlined duplicate in src/core/brain-repo-durability.ts) called `execSync`
without an explicit `env`, relying on default inheritance. Under Bun,
execSync/execFileSync snapshot process.env at BUN'S OWN STARTUP, not at
call time — a runtime PATH mutation (dotenv/config loading, wrapper-script
env sourcing, etc.) happening after Bun boots but before this call is
invisible to `which gbrain` unless the current env is forwarded
explicitly.
This is a known, already-precedented Bun quirk in this exact codebase:
spawn-helpers.ts's detectTini() was already fixed for the identical
symptom with the identical one-line fix (`env: process.env`), with a
comment explaining the mechanism — this call site was simply missed.
Matches the reported symptom precisely: "which gbrain" resolves fine when
run standalone (a fresh Bun process, no prior env mutation to hide), but
throws specifically from inside autopilot's managed-worker spawn path
(src/commands/autopilot.ts:416, guarded by `spawnManagedWorker` — Postgres
engine + minion_mode enabled), which fires after config/dotenv loading has
already run in that process. Impact per the report: this silently
degrades to no worker ever picking up queued jobs (including embed jobs),
with `gbrain doctor` only showing a growing "N stale chunks" warning that
reads like an ordinary backlog rather than a broken worker.
Also improved the throw-path error message to include the actual
PATH/execPath/argv[1] values observed at failure time, so a future report
doesn't require guessing at what the process actually saw.
Not fixed here (documented as a separate, smaller finding): a third call
site with the identical missing-env pattern exists in
src/core/claw-test/runners/openclaw.ts ('which openclaw'). Left out of
scope for this PR, which is specifically about #2747's reported symptom;
worth a small follow-up.
Verification: bun run typecheck clean, bun run verify 31/31 green,
test/autopilot-resolve-cli.test.ts 4/4 pass (existing coverage, no
regressions — a genuinely-simulated Bun-env-snapshot race isn't
reproducible in a same-process unit test, and this codebase's own
convention explicitly avoids mock.module for child_process per
doctor-orphan-ratio.test.ts's stated test-isolation rule, so this PR
relies on the fix's precedent-match + existing coverage rather than a new
mock-based regression test).
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(jobs): retry resets started_at/attempts_made/attempts_started (#2783)
`gbrain jobs retry` re-queued a dead job by resetting status/error_text/
locks/delay/finished_at, but left started_at, attempts_made, and
attempts_started untouched. On re-claim, claim()'s
`started_at = COALESCE(started_at, now())` preserved the ORIGINAL
first-claim timestamp instead of re-stamping it. handleWallClockTimeouts()
anchors on `now() - started_at`: a retry issued more than timeout_ms * 2
after the original claim was immediately dead-lettered again in under a
second, with attempts_made already past max_attempts — making retry
useless for exactly the case it exists for (recovering work after an
outage that outlasted the job's timeout).
An explicit `jobs retry` is an operator asserting "run this fresh", so
retryJob now also clears started_at (NULL, re-stamped on next claim) and
resets attempts_made/attempts_started to 0.
Two new tests: direct assertion that retry resets all three columns, and
a full repro of the reported bug (wall-clock-killed job retried long
after the original claim now survives re-claim instead of being
immediately re-killed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS
* fix(jobs): also reset stalled_counter on retry (#2783)
Codex review round 1 found the fix was incomplete: a job dead-lettered by
stall exhaustion (handleStalled() at stalled_counter + 1 >= max_stalled)
retained its exhausted stalled_counter across retry. The retried job's
very first lock expiry after re-claim would immediately re-satisfy the
dead-letter threshold, contradicting the same "run this fresh" intent the
started_at/attempts reset already established.
New test mirrors the existing wall-clock repro: exhaust the stall budget
via two real handleStalled() calls, retry, confirm one more stall now
requeues instead of dead-lettering again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NCVEvUVm15bFuKqskWgfNS
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
extractTakesFromPages hardcoded anthropic:claude-haiku-4-5 as the classifier
model. On an OAuth/local-only install (no ANTHROPIC_API_KEY; chat routed
through a gateway model) every takes extraction died with llm_unavailable —
the takes layer silently never populated and the takes_count health check
stayed red despite a working configured chat_model.
Resolution is now `opts.model || getChatModel()` — the same file-plane
gateway-config idiom enrich.ts uses — NOT engine.getConfig('chat_model')
(the DB config plane), keeping model routing on the single config plane the
rest of the codebase reads. Explicit opts.model still wins; unconfigured
installs fall through to the gateway's DEFAULT_CHAT_MODEL.
Adds a regression test that pins the file-plane read: a conflicting DB-plane
config.chat_model row is ignored, the gateway-configured chat_model is used
when opts.model is unset, and explicit opts.model wins. Verified the
file-plane test fails against the pre-fix code.
Takeover of #2997 by @Nazim22 with the model read moved from the DB config
plane to the file-plane gateway idiom.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Nazz <nazim.mj@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Garry Tan