Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r
(#580 env-var language for query+write side, #581 migration backfill,
#582 gbrain reindex-search-vector), rebased onto current master with the
security review's required fixes applied:
- Migration renumbered v116 -> v123 (master's v116 was already claimed by
code_edges_source_backfill; master is at v122).
- Restored the v120/#1647 search_path hardening: all four CREATE OR
REPLACE trigger-function bodies (migration handler + reindex command)
now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE
resets proconfig and would otherwise strip the hardening on upgrade.
- reindex-search-vector: shared progress reporter (stderr phases
reindex_search_vector.pages/.chunks), id-keyset batched backfill
(5000 rows/UPDATE) instead of single whole-table statements, and
--json no longer bypasses the --yes/TTY confirmation gate.
- Allowlist validation regex + injection tests kept exactly as authored.
- Stale v33/v116 comments swept; docs/guides/multi-language-fts.md
written (README referenced it but no PR added it); llms bundles
regenerated via bun run build:llms.
- Trilogy tests quarantined as *.serial.test.ts (env mutation, per
check-test-isolation).
Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese
produces portuguese-stemmed vectors with search_path pinned; the reindex
command retokenizes an english brain to portuguese in place.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Takeover of community PR #2387 with the security review's required fixes
applied. Original work by @harrisali0101.
With GBRAIN_RLS_SCOPE_BINDING=1, source-scoped Postgres read methods wrap
their queries in a transaction that binds set_config('app.scopes', $1, true)
(federated sourceIds CSV > scalar sourceId > '*') so operator-managed RLS
policies can filter rows at the SQL layer — defense-in-depth layer 2 under
the mandatory app-layer source filters.
Review fixes on top of the original PR:
- Flag-off is now a TRUE pass-through: no new per-read transaction wrap
(the #1794 PgBouncer pool-exhaustion class). Only the three search
methods keep a transaction when off — exactly the sql.begin() +
SET LOCAL statement_timeout wrap they already had on master — via the
helper's alwaysTransaction option.
- Preserved the PR's latent setseed fix: listCorpusSample pins
setseed() + SELECT to one connection when seeded (alwaysTransaction
gated on opts.seed), so the deterministic path can't split across
pooled connections.
- Updated the two postgres-engine shape tests to pin the new invariant
(search methods route through withScopedReadTransaction with
alwaysTransaction; helper owns the sql.begin(); flag-off path is
callback(this.sql)).
- New behavioral tests (test/postgres-engine-rls-scope.test.ts): flag-off
pass-through, flag-off alwaysTransaction, flag-on set_config emission,
federated > scalar > '*' precedence, and the CSV as a bound parameter
(never interpolated).
- Fixed the helper header comment to match the actual branching behavior.
- Operator docs in docs/ENGINES.md: env var, policy SQL, the
ALTER ROLE ... SET app.scopes='*' default requirement, FORCE ROW LEVEL
SECURITY for owner roles, and the honest caveat about unwrapped paths.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Harris <79081645+harrisali0101@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Five verified-open fixes to the dream/synthesize output family:
- #1586: thread the cycle's resolved sourceId (cycleSourceId) through
runPhaseSynthesize -> SubagentHandlerData.source_id -> subagent tool
OperationContext, and stamp collected refs + summary page with the same
source, so synthesized pages stop landing in 'default'.
- #2415: new config knob dream.synthesize.output_root (default 'wiki',
zero behavior change unless set) drives the synthesize prompt slug
templates, the patterns reflection lookup + prompt, and remaps the
filing-rule allow-list globs. Registered in KNOWN_CONFIG_KEYS.
- #2163: synthesize_concepts writes concept pages through
importFromContent (put_page's parse->chunk->embed pipeline) instead of
bare engine.putPage, so concepts/ pages are chunked + embedded and
reachable by retrieval.
- #2569: stampDreamProvenance persists dream_generated + dream_cycle_date
into pages.frontmatter (JSONB merge via executeRawJsonb) at write time,
so generated pages are DB-queryable and put_page write-through can't
erase the marker.
- #2606: chronicle judge detects stopReason 'length' truncation and
no-JSON-array parse failures as distinct skipped reasons
(judge_truncated / judge_parse_failed) instead of a false terminal
no_events; output cap raised to 4000 and configurable via
chronicle.judge_max_tokens.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Rebased port of PR #774 onto current master. A single git repo can hold N
logical sources at subdirectories: git operations run at the discovered
repo root (git rev-parse --show-toplevel — worktrees and submodules
resolve natively) while file walking, imports, deletes and renames are
scoped to the subpath. Passing the subdirectory directly as the repo path
(auto-discovery) works through the same code path.
Path-containment guards (the point of the feature):
- NAV-1/NAV-2: the realpath-resolved scope must live inside the
realpath-resolved git root — ../-traversal and symlinked subdirs
pointing outside the repo are rejected before any git op.
- NAV-1 TOCTOU: per-file realpath re-validation during the incremental
import drain and rename reimport; symlink-escape files are recorded as
failures (fail-closed — the bookmark cannot advance past an escape).
- NAV-4: an --exclude set that filters out every candidate warns loudly.
Scoped syncs use git-root-relative slugs + source_path in BOTH the full
and incremental paths (runImport gains slugRoot), fixing the original
PR's full/incremental slug divergence in the auto-discovery flow.
--exclude matches scope-relative paths in both paths; exclusion never
deletes previously-imported pages. The full-sync delete reconcile is
scope-restricted and relativizes against the slug base so a healthy
scoped source can't trip the #2828 mass-delete valve. .gitignore
management resolves to the git root at every call site.
Preserves all master-side sync work since the original branch: the #2828
mass-delete safety valve, #1794 resumable checkpoints + pinned targets,
#1950 stall watchdog, #2335 heartbeat bump, #1970 bookmark reachability,
and the git-ls-files walker (#2315/#2462/#2678, whose symlink/cycle
hardening is untouched).
Co-authored-by: Jeremy Knows <jeremy@veefriends.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three verified-open defects, one family: sync silently destroying or
diverging on content it should preserve.
#2404 (P0) — 'ops' was hardcoded in PRUNE_DIR_NAMES (a v0.2.0-era
carve-out), so any path with an ops segment was 'pruned-dir': committed
ops/*.md never imported, and modified ops/* files hit the
unsyncableModified delete loop (whose #1433 guard only spared
'metafile'), silently deleting put-created pages like the bundled
daily-task-manager's canonical ops/tasks on every sync. Fix: remove
'ops' from the prune list (ordinary user content; the vendor/generated
entries stay), and harden the delete loop to also skip 'pruned-dir' —
a page under a pruned dir can only exist via a deliberate put_page.
#2426 (P0) — write-through content stayed DB-only and was deleted by
sync --full. All three compounding bugs fixed:
1. writePageThrough now best-effort commits the artifact (path-limited
git commit) on durability-hardened repos, so the post-commit hook
can push it; result carries committed?: boolean.
2. scripts/brain-commit-push.sh stages+commits BEFORE any pull — the
old fetch+pull-rebase-first order aborted on any dirty tree, so the
helper could never commit a MODIFIED page; brain_push's
rebase-on-reject already handles an advanced remote.
3. The full-sync delete-reconcile partitions stale pages by git
history (listEverCommittedPaths): never-committed source_paths are
DB-only write-through — pages are KEPT and re-exported to the
working tree instead of soft-deleted. Builds on the #2828
mass-delete valve (covers the below-valve cases).
#2607 — the sync --full git ls-files fast path bypassed pruneDir, so a
full pass imported (and resurrected soft-deleted) pages under dot-dirs
and vendored trees that incremental sync excludes. Fix:
isCollectibleForWalker applies the same segment-level pruneDir gate as
classifySync, so full and incremental enumeration agree.
One regression test per defect (all verified failing against master
src): test/sync-ops-pages.serial.test.ts,
test/write-through-commit.serial.test.ts, the #2426 helper-order test
in test/brain-durability-hook.serial.test.ts,
test/sync-reconcile-db-only.serial.test.ts,
test/import-git-fastpath-prune.test.ts.
Fixes#2404Fixes#2426Fixes#2607
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Port of #1837 (mvanhorn) onto current master. The extract_facts cycle
phase unconditionally wipe-and-reinserted every page's fence-owned DB
rows, so re-running a cycle on unchanged content churned rows and — on
the Postgres engine reported in #1781 — accumulated duplicates each run.
The phase now de-dupes extracted facts by the canonical (claim, source)
content key and reconciles the page-scoped DB index: no-op when already
in sync, insert-only for new keys, wipe/reinsert only when stale rows
need cleanup.
Adjustments over the original PR to fit current master:
- preserve #1972's abortSignal threading into the batch embed call
- preserve #1928's excludeSourcePrefixes: ['cli:'] on every wipe, and
exclude cli:-origin conversation facts from the existing-row set so
they neither count as stale (which would force a wipe every cycle)
nor get compared against the fence
Fixes#1781
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four verified-open issues in the subagent-orchestration family, one PR:
- #1594: dream synthesize subagent job/wait timeouts promoted from
hardcoded 30/35-min constants to config keys
dream.synthesize.subagent_timeout_ms / subagent_wait_timeout_ms.
Approach ported from stale PR #1596 (credit @ai920wisco).
- #2778: add_timeline_entry joins the subagent brain-tool allowlist,
fenced fail-closed by the shared enforceSubagentSlugFence (extracted
from put_page's inline check — same trusted-workspace allow-list /
wiki/agents/<id>/ namespace policy). The per-turn output cap is now
resolveMaxOutputTokens (data.max_tokens → agent.max_output_tokens →
8192, was hardcoded 4096); a max_tokens stop surfaces as
stop_reason 'max_tokens' instead of a silent end_turn, and a
mid-tool-round cap hit injects a truncation note so the model
re-issues the dropped call.
- #2782: patterns phase status now reflects the child outcome —
non-complete outcome with zero writes → fail (PATTERNS_CHILD_<OUTCOME>),
partial writes → warn. Patterns timeouts get the same config-key pair
(dream.patterns.subagent_timeout_ms / subagent_wait_timeout_ms).
- #2113: facts extraction cap is config facts.extraction_max_tokens
(default 4000, was hardcoded 1500); stopReason 'length' is checked,
retried once at 2x the cap, and surfaced on stderr instead of
silently extracting zero facts.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: ai920wisco <ai920wisco@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Absorbs PR #2279's intent (drop the hardcoded ANTHROPIC_API_KEY gate) with
the end-state the gateway world actually wants: the patterns phase now
probes the RESOLVED patterns model through probeChatModel(normalizeModelId)
— the same semantics as think/index.ts and synthesize's makeJudgeClient.
Fixes two misclassifications of the old env gate:
- Non-Anthropic stacks (litellm, deepseek, openrouter, ...) were skipped as
"no upstream" even though the subagent routes them through the gateway
(agent.use_gateway_loop). They now pass the gate; their auth is checked
lazily at dispatch and surfaces in the job outcome.
- Anthropic keys set via `gbrain config set anthropic_api_key` (stdio MCP
launches without shell env) were treated as missing. hasAnthropicKey
inside probeChatModel reads both sources.
Skip reason renames no_api_key → no_provider (carrying the probe's detail).
Both pinning tests updated: the structural test asserts the probe wiring;
the PGLite E2E swaps its env-only helper for the shared hermetic
withoutAnthropicKey (env + config file) so it can't flip to a live LLM call
on a dev machine with a config-file key.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Ports the still-unmerged remnant of PR #1891 (#1593 follow-up). Master's
getConfig gained retry-with-reconnect in #1603, but its siblings —
setConfig, unsetConfig, listConfigKeys — still touched `this.sql` bare, so
the first config write/list after a mid-cycle instance-pool teardown threw
the retryable "No database connection" (issue #1678) unhandled instead of
rebuilding the pool.
Adds the connRetry helper from #1891 (same retry+reconnect posture as
batchRetry, but no batch audit JSONL — a config accessor is not a sized
batch), refactors getConfig onto it, and wraps the other three. Writes are
safe to retry: withRetry only retries connection-class failures and both
writes are idempotent (upsert / delete).
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: jalagrange <jalagrange@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Ports the still-unmerged half of PR #2442. OpenAI caches prompt prefixes
automatically, but a stable prompt_cache_key keeps requests that share a
prefix on the same inference engine, lifting the automatic-cache hit rate.
chat() now derives a stable key from the system prompt + sorted tool names
for native-openai models and passes it via providerOptions.openai.
promptCacheKey. Config provider_chat_options still overrides the derived
key; anthropic/google/openai-compatible providers are untouched.
The Anthropic half of #2442 (cache_control "silent no-op") is superseded:
@ai-sdk/anthropic 3.0.74 forwards call-level providerOptions.anthropic.
cacheControl as the Messages API's request-level cache_control (automatic
prefix caching), so master's existing marker is live on current deps.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: CoachRyanNguyen <CoachRyanNguyen@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Port of #1731 (diazMelgarejo) onto current master. gbrain import wrote
~/.gbrain/import-checkpoint.json with the caller's raw dir argument, so
a checkpoint left behind by an interrupted run (e.g. SIGTERM) could carry
"." or a symlinked spelling — an identity that resolves to whatever CWD
the next consumer happens to run from. Downstream tooling that treated
the checkpoint dir as an owned staging boundary could then act on the
wrong directory.
- runImport captures the import target ONCE via resolveImportTargetDir
(resolve + realpathSync) and threads that canonical value through
collection, checkpoint load/save, and resume filtering
- checkpoints are self-describing (schema_version: 1, owner: "gbrain",
kind: "import"); loadCheckpoint tolerates absent metadata (legacy
path-based files) but rejects present-and-wrong metadata and any
relative dir
- checkpoint contract documented in docs/guides/live-sync.md (llms
bundle regenerated)
- test/import-resume.test.ts fixture now realpaths its tmpdir so planted
checkpoints match the canonicalized dir (macOS /var -> /private/var)
Fixes#1728
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Lawrence Melgarejo <Lawrence@cyre.me>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`parseMarkdown` previously walked the lines after the opening `---` and
recorded the first `^#{1,6}\s`-shaped line as a `headingBeforeClose`,
then flagged MISSING_CLOSE when that index came before the actual closing
fence. YAML allows `#` comment lines anywhere inside the document, so a
template that leads with annotation comments inside the fence (e.g. a
`# Research Template` header before the keys) hit a false-positive
MISSING_CLOSE even though the closing `---` was present.
Fix: only walk for the closing `---`. When it is found, content between
the fences is YAML; `#` lines are comments, not headings. When the close
is genuinely missing, surface the first heading-shaped line as a
where-it-went-off-the-rails hint (this path was already correct; we keep
it for the genuine missing-close case).
Two regression tests added to `test/markdown-validation.test.ts`:
- `#` comment lines at the top of a closed frontmatter
- `#` comment lines interleaved with keys
All 68 tests across the four markdown/frontmatter test files stay green.
* fix(extract,ingest,cycle): source-provenance wave — thread source identity through ingest_capture, fs-walk links, and cycle extract (#1522#1747#1503)
Three fixes in the same invariant class (source identity silently dropped
on the write path, collapsing to the 'default' source):
- #1522: the ingest_capture Minion handler validated IngestionEvent
provenance (source_id/source_kind/source_uri) then dropped it on the
importFromContent call. Now threads source_kind/source_uri +
ingested_via='ingest_capture' into the page write, and routes the write
under event.source_id when it names a registered source AND the event
is trusted (fail-closed: untrusted webhook payloads carry a
caller-controlled x-gbrain-source-id header and must not choose their
write source; unregistered emitter ids keep default routing so the
webhook path can't FK-fail).
- #1747: the fs-walk extractors (extractLinksFromDir /
extractTimelineFromDir / extractForSlugs) built batch rows with no
source_id, so addLinksBatch/addTimelineEntriesBatch mapped missing →
'default' and the pages JOIN dropped every row on a non-default source
("Links: created 0 from N pages", no error). ExtractOpts gains
sourceId; the CLI fs path resolves it via resolveSourceId
(--source-id > env > dotfile > registered path > sole-non-default)
and rows are stamped from/to/origin_source_id + timeline source_id.
- #1503: the cycle's extract phase (runPhaseExtract) never passed a
sourceId, so federated-brain dream/autopilot cycles persisted nothing
every night. It now threads cycleSourceId (explicit --source or
resolveSourceForDir(brainDir) — the same seam runPhaseSync uses) into
runExtractCore for both the incremental and full-walk paths.
Regression tests: handler provenance write-through (registered/
unregistered/untrusted routing), fs-walk + CLI resolution + incremental
cycle route landing edges/timeline in the right source (all red on
unfixed master), and a negative control pinning the pre-fix JOIN-drop
shape.
Reuses the ExtractOpts.sourceId threading approach from PR #1719,
rebased onto the current signal-aware signatures and extended to the
cycle + timeline paths.
Fixes#1522Fixes#1747Fixes#1503
Co-authored-by: seungsu <kss530c@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: update cycle source pin for cycleSourceId threading
The #1972 source-pin test asserts the literal runPhaseExtract call site in
cycle.ts. The #1503 fix appended cycleSourceId after opts.signal; signal
threading is unchanged (still the 5th arg, forwarded to runExtractCore).
Pin updated to the new literal — invariant intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: seungsu <kss530c@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Recover the worker-owned Postgres pool when promoteDelayed escapes a retryable connection error, preventing the repeated Promotion error: No database connection loop from issue #1491.\n\nAdds a regression test proving reconnect happens before the worker continues to claim work.
Closes#345.
The bulk-import walker isCollectibleForWalker filtered admitted files by
extension only, while incremental sync excludes README/index/log/schema via
isSyncable -> SYNC_SKIP_FILES. A directory import therefore ingested every
directory README as a folder-titled ghost page that trigram-corrupts fuzzy
entity resolution and inflates orphan count. Apply SYNC_SKIP_FILES (basename
guard) at the top of isCollectibleForWalker so both the FS-walk and the
git-fast-path collection routes agree with sync.
Also add RESOLVER.md to SYNC_SKIP_FILES: a structural routing metafile
(docs-aligned with schema.md/index.md/log.md/README.md), not indexable content.
Co-authored-by: ElliotDrel <ElliotDrel@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(minions): restore rolling conversation prompt-cache on the direct SDK path
Regression since v0.42.51 (@ai-sdk bump): the direct/native subagent
path (agent.use_gateway_loop=false) only marks cache_control on the
static system-prompt and last-tool-def blocks (~5.2K tokens). The
`anthroMessages` array — the part of the request that actually grows
every turn — carries no cache marker at all, so Anthropic re-bills the
full conversation as fresh input on every turn instead of reading it
from cache.
Before this regression, cache_read grew with the conversation
(4.6K -> 125K across a session). Since the regression, cache_read is
pinned at the ~5.2K static prefix regardless of conversation length.
Fix: mark the last content block of the last message with
`cache_control: { type: 'ephemeral' }` on every turn, after first
stripping any stale marker left on an earlier message. Anthropic
caches everything up to the last cache_control breakpoint, so this
turns the trailing marker into a rolling window over the growing
conversation while staying within the 4-breakpoint limit (system +
last-tool + 1 rolling = 3 used).
Measured on a real dream synthesize run: cache_read/input ratio
0.000 -> 32-61 across 23 calls, ~78-80% cost reduction for that run
($8.5 no-cache-equivalent -> $1.71), zero dead-letter jobs.
Neither the gateway path (cache_control placed at the top level,
which @ai-sdk 3.x silently ignores — see #2490) nor #2442 (system +
last-tool only) restores this; both leave the conversation body
unrecovered.
* fix(minions): normalize seed message content before caching it
Codex review caught a real gap: a fresh job's seed user message is
initialized as `content: data.prompt` (a plain string), not a content-
block array. The rolling-cache logic added in the previous commit only
attaches cache_control when `Array.isArray(lastMsg.content)`, so it
silently skipped the very first API call — meaning the common
single-tool round-trip (seed prompt -> tool_use -> tool_result -> done)
got no conversation-cache benefit at all, only jobs with 3+ turns did.
Normalize string content to a one-block text array before checking, so
the first call gets the same rolling breakpoint as every later one.
* fix(minions): retain the prior rolling cache breakpoint, not just the newest
Second Codex review pass: with a single rolling marker, deleting every
prior message's cache_control before placing the new one means a turn
that adds more than 20 content blocks since the last marker (e.g. a
large parallel tool_use/tool_result round) can miss Anthropic's cache
lookup entirely -- the API's automatic prefix search only looks back
up to 20 blocks from a breakpoint to find a prior cached prefix.
Keep the immediately-preceding rolling marker in place instead of
stripping down to one; only evict markers older than that. This still
fits the 4-breakpoint budget (system + last-tool + 2 rolling = 4) and
guarantees the previous marker's prefix remains a valid, already-cached
read even when the new marker's own lookback misses.
The instance-pool reconnect() did disconnect() (nulling _sql) BEFORE connect(),
so a connect() failure during a transient Postgres blip left _sql null for the
rest of the process — every subsequent non-retry-wrapped call then fell through
to the never-connected module singleton and threw 'No database connection',
crashing the autopilot worker into a respawn loop. Build-then-swap: snapshot the
live pool, build a fresh one, end the old only once the new validates, restore
on failure. Keeps upstream's reap-detection + pool-recovery audit. Confirmed by
jalagrange on closed PR #1593; this is the Layer-1 fix he left to us.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The litellm recipe shipped only an `embedding` touchpoint. `getProviderCapabilities()` in `src/core/ai/capabilities.ts` throws when `recipe.touchpoints.chat` is missing, `classifyCapabilities()` returns `'unknown'`, and `enforceSubagentCapable()` in `src/core/model-config.ts` silently falls back to `TIER_DEFAULTS.subagent` (anthropic). Any brain that routes paid traffic through a litellm-style proxy AND has no `ANTHROPIC_API_KEY` then sees every subagent loop dispatch throw `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`. The user's explicit `models.tier.subagent = litellm:*` choice is overridden without their knowledge.
Declare `chat` and `expansion` touchpoints mirroring the openai recipe's shape: `models: []` (litellm proxies arbitrary backends; allowlist is intentionally empty, since `assertTouchpoint` already skips allowlist checks for `tier: 'openai-compat'`), `supports_tools: true`, `supports_subagent_loop: true`, `supports_prompt_cache: false` (OpenAI-compat backends don't honor Anthropic-style `cache_control`), `max_context_tokens: 200_000` (conservative GPT-5-family default; per-deployment override needed for smaller-context backends), costs `undefined` (varies by proxied provider).
Reproduction (deterministic):
1. Fresh brain with no `ANTHROPIC_API_KEY` in env.
2. `gbrain config set models.tier.subagent litellm:gpt-5.4` (or any `litellm:*` string).
3. `gbrain models` warns and falls back to `anthropic:claude-sonnet-4-6`.
4. Submit any subagent job; throws `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`.
After the patch, `classifyCapabilities('litellm:gpt-5.4', recipe)` returns `degraded:no_caching` (chat-capable, no Anthropic-style prompt cache). `enforceSubagentCapable` no longer steals the model choice. The subagent loop emits a one-time `degraded:no_caching` warn about prompt-cache absence; cost scales linearly with conversation length, accepted trade for the proxy path.
Rebase-port of #2452 (spinsirr:fix/calibration-profile-scope-and-cli) onto
current master after tonight's merges made the fork branch conflict.
- cli: add 'calibration' to CLI_ONLY so dispatch reaches its existing
handler instead of falling through to "Unknown command" (#2035); honor
--source / GBRAIN_SOURCE in the calibration CLI.
- takes: route takes_list / takes_search / takes_scorecard /
takes_calibration through sourceScopeOpts(ctx) (federated array > scalar
> nothing) and scope engine reads via the take's page.source_id — JOIN
filter for list/search, EXISTS for scorecard/curve — on both engines
(#2200-class).
- bigint: shared takeHitRowToHit coercion in searchTakes /
searchTakesVector (both engines) + bigintToStringReplacer on the cli.ts
output normalizer and the `gbrain call` exit, so int8/BIGSERIAL columns
no longer crash JSON.stringify (#2450); calibration profile id
BIGSERIAL → string.
- calibration: default model ids route through TIER_DEFAULTS
(provider-prefixed) instead of bare model strings; admin calibration
chart endpoints fixed (takes has no page_slug column; month-precision
since_date; Date generated_at; bigint id in drill-down).
The think/gather source-scope slice of the original PR was dropped: it
already landed on master via #2739.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: spinsirr <ID+spinsirr@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
classifyPgliteInitError() routes bare Emscripten aborts to the 'unknown'
verdict, whose hint unconditionally printed "Most common cause: the macOS
26.3 WASM bug (#223)" — even on Windows and Linux (#2195, #1870).
- buildPgliteInitErrorMessage now takes a platform param (default
process.platform): darwin keeps the #223 link as a *possible* cause;
other platforms get the plausible off-macOS causes (lock contention,
damaged data dir) plus `gbrain doctor` / `gbrain reinit-pglite`.
- New stringifyPgliteInitError(): non-Error rejections (Emscripten
aborts can throw plain objects) no longer print "[object Object]".
- Regression tests for both branches + the stringifier in
test/pglite-init-classifier.test.ts.
Canonical for a 7-report class: #2674, #1870, #1195, #1502, #2195,
#939, #391.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add provider_chat_options alongside provider_base_urls and thread it through
the gateway config path into chat(). The chat request now deep-merges
provider-scoped options and model-scoped overrides into providerOptions keyed
by recipe id, preserving existing gateway-built options such as Anthropic
cacheControl and leaving absent-config behavior unchanged.
This lets operators disable thinking for small-budget hybrid-reasoning utility
calls without hardcoding that behavior for every use of those models.
* fix(facts): durable facts-absorb jobs for one-shot CLI processes
Every gbrain capture/put from a short-lived CLI enqueued the facts:absorb
chat into the in-process FactsQueue, then the exit teardown drained for
1-2s and aborted the in-flight call — logging 'pipeline_error: [chat(...)]
The operation was aborted.' on every eligible CLI page write and never
extracting facts.
cli.ts now marks one-shot processes (everything except serve/jobs/
autopilot); runFactsBackstop's queue mode submits a durable facts-absorb
minion job for the long-lived jobs worker instead, with content-hash
idempotency and fallback to the in-process queue if submission fails.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(facts): fence-write resolves source-scoped page path
writeFactsToFence joined local_path + slug directly, writing main-source
fences to the repo ROOT (the default source's tree) and polluting ~/brain
with stray root-level fence files. Route through resolvePageFilePath —
the same helper the put_page write-through and dream-cycle reverse-render
use — so non-default sources fence into .sources/<id>/<slug>.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ragnar Åström <reghar@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The parser/doctor/extractor read the polished page body (compiled_truth +
timeline) instead of the raw turn-by-turn transcript that meeting pages store
in a `raw_transcript` frontmatter sidecar, and the brain's actual raw format
(`Speaker A: ...` / `Speaker B: ...`) had no built-in pattern. Result: scan
returned no_match / 0 messages and conversation-fact extraction produced 0
segments / 0 facts.
- new readConversationBodyForParsing(): prefer the raw_transcript sidecar when
present, fall back to compiled_truth + timeline (src/core/conversation-parser/body.ts)
- wire it into conversation-parser scan, doctor coverage check, and
extract-conversation-facts (drops the old readPageBody helper)
- add a built-in `speaker-letter-no-time` pattern for plain `Speaker A:` lines
Verified: scan now returns phase=regex_match (speaker-letter-no-time), and the
Ben page extracts 61 facts / 7 segments. Tests added; suite green.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
since/until range filters were applied against updated_at, so edited-but-old
entries leaked into time-bounded queries. Filter on the effective date instead.
Closes#1520
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(extract): recognize inline [Source: ..., YYYY-MM-DD] citations as timeline entries
gbrain's own quality conventions (skills/conventions/quality.md) require a
dated [Source: ..., YYYY-MM-DD] citation on every brain write, so curated
pages are full of dated evidence — but extractTimelineFromContent only
recognized the timeline-bullet and date-header formats. A page whose dates
all live in citations scored zero timeline coverage in brain_score, and
doctor pointed users at a formatting convention their own citations already
satisfied in spirit.
Format 3 files one entry per citation: date and source from the marker,
summary from the annotated line with citation markers stripped. Lines
already captured by Format 1 are skipped so a timeline bullet carrying its
own citation is not double-filed. Bare citations with no surrounding text
are ignored.
Idempotency is unchanged: persistence already dedupes at the DB layer.
* fix(extract): Format 3 citations also in parseTimelineEntries (db-source + ingest path)
The first commit only taught extractTimelineFromContent (fs-source) the
citation format; the db-source extract and the ingest path parse through
parseTimelineEntries in core/link-extraction.ts, which still could not see
citations. Same rules as the fs parser: bullet-captured lines skipped,
bare citations ignored, invalid calendar dates rejected; the citation
source is preserved in the entry detail.
---------
Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com>
extractTakesFromPages selected pages by updated_at DESC + LIMIT with no
exclusion of pages that already hold takes. The CLI clamps --max-pages to
1000, so on a corpus larger than one run every re-run rescanned the same
most-recent 1000: the older tail could never be bootstrapped, and each
rescan re-spent Haiku budget producing upsert-identical rows. Seen live on
a 2,311-eligible-page brain — a second run would have covered 0 new pages.
Covered pages are now skipped by default (NOT EXISTS on takes.page_id), so
repeat runs sweep a large corpus in slices; --include-covered restores the
full rescan for refresh use. Usage text documents both plus the 1000 clamp.
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
extract_atoms minted duplicate atom pages two ways:
A. Trailing-dash twins. The local slugger truncated the title at 60 chars
with no re-strip, so a cut landing on a hyphen left a trailing dash
(`…would-`). The FS-import normalizer (slugifySegment) strips it
(`…would`), so the same atom persisted under two slugs and the
dedup-by-slug check never collapsed them.
B. Cross-day re-mint. The slug used the run date (todayDate()), while the
idempotency guard keys on the whole-file source_hash. Append-only
sources (chat/transcript exports) grow daily, so the file hash changes,
the guard never matches, the source is re-extracted, and each re-mint
lands under a new date prefix → a new slug → no upsert → a duplicate.
Fix: make the atom slug a deterministic function of stable inputs —
`atoms/<source-date>/<stem>-<title-hash>`:
- source date is parsed from the source ref (transcript filename / page
slug), not the run date, so re-extraction converges on the same slug and
putPage upserts instead of duplicating;
- the 6-char title hash keeps two atoms whose titles share the first 60
chars on distinct slugs (no silent clobber of a different atom);
- the stem routes through the canonical slugifySegment and re-strips a
trailing dash, so the two write paths can no longer disagree.
The whole-file source_hash batch check is retained only as a cost fast-path
(skip re-running the model on an unchanged source); correctness no longer
depends on it.
Adds a hermetic PGLite regression test (no DATABASE_URL) asserting the
source-dated prefix, title-hash suffix, trailing-dash strip, and upsert on
re-extraction of a grown append-only transcript.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Atom-extraction page discovery hardcoded EXTRACTABLE_PAGE_TYPES and ignored the
active pack's `extractable: true` flags, so a type declared extractable in the
manifest (e.g. `note`) never actually extracted. Closes the D2 TODO the code
already flagged (extract-atoms.ts: 'future pack-aware refactor ... pull from the
active pack manifest').
Resolve the allowlist as: legacy hardcoded floor UNION the pack's extractable
types, MINUS synthesis outputs (atom, concept — extracting from these would loop,
since concepts are synthesized from atoms). Mirrors facts/eligibility.ts, which
excludes concept the same way. Back-compat: gbrain-base brains keep every legacy
target via the union; fail-soft falls back to the legacy floor if the pack can't
load.
- pure unionExtractableTypes() policy, unit-tested
- discoverExtractablePages + countExtractAtomsBacklog resolve from the pack
- page-discovery fixture updated (note now extracts; concept stays excluded)
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
claude-sonnet-5 and claude-fable-5 are GA Anthropic models, but neither
was in CANONICAL_PRICING. A brain routing a tier to them (e.g.
models.tier.reasoning = anthropic:claude-sonnet-5) ran with cost
telemetry blind on that tier: canonicalLookup missed, the budget meter
logged BUDGET_METER_NO_PRICING and disabled the gate, and cost views
under-reported spend.
- model-pricing.ts: anthropic:claude-sonnet-5 at $3/$15 and
anthropic:claude-fable-5 at $10/$50. Sonnet 5's launch intro discount
($2/$10 through 2026-08-31) is deliberately not modeled — the table
carries standard rates so estimates stay conservative and the entry
needs no time-bombed edit when the promo lapses.
- takes-quality-eval/pricing.ts: claude-sonnet-5 added to the curated
SUPPORTED_MODELS allowlist (a likely judge override). Fable 5 stays
out — priced for warn-only consumers, not a budgeted-eval panel model.
- model-pricing.test.ts: pin tests for both rows, matching the existing
Opus 4.8/4.7 pattern. Drift guards iterate the table; no changes.
All derived views (ANTHROPIC_PRICING bare view -> budget-tracker,
batch-projection, budget-meter) pick the rows up automatically.
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
resolveHardExcludes() only ran at DB-query build time (cache miss), so
query_cache rows written by a process without GBRAIN_SEARCH_EXCLUDE could
be served to a process with it (and vice versa), leaking excluded slugs.
hybridSearchCached now resolves the effective hard-exclude list exactly
as the engines' query-build path does and folds it (sorted, append-only
hx= part) into knobsHash via a new KnobsHashContext.hardExcludes field.
KNOBS_HASH_VERSION 11 -> 12: one-time global cache cold-miss on upgrade,
refills within cache.ttl_seconds.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Collector branch superseding the gateway tool-loop duplicate cluster and
adjacent provider fixes. Re-implemented from the best of each PR (deduped by
content, not file-overlap); every fix carries test coverage.
Gateway tool-loop resume (supersedes #1934#2062#2065#2112#2274#2487#2336#2257#2499#2491, test #2063):
- toolLoop now persists the tool-result user turn per round (onToolResultTurn),
so a resumed subagent job reloads a balanced transcript instead of dangling
assistant tool-calls that non-Anthropic providers reject with
AI_MissingToolResultsError.
- runSubagentViaGateway reconciles an already-corrupted transcript on resume:
it heals every dangling assistant tool-call turn (not just the tail) from the
settled subagent_tool_executions rows, re-dispatching idempotent-pending
tools and throwing on non-idempotent, mirroring the legacy Anthropic path.
Terminal early-return for a transcript that already reached end_turn.
- repairToolPairing() is a last-resort normalization at the chat() boundary
(from #2336): back-fills error stubs for any assistant tool-call still
unanswered (partial turns, provider-duplicated/dropped IDs on local models,
length-truncated batches). No-op on balanced input.
- toModelMessages is Date-safe (Postgres timestamptz -> ISO via a JSON round
trip at the SDK boundary, never a ::jsonb cast; degrades bigint/circular to a
string instead of throwing) and drops non-string text blocks reasoning models
emit that AI SDK v6 rejects (#2488).
Adjacent provider fixes:
- Model-aware default max output tokens: thinking-by-default Claude 5 models get
headroom (gateway 32000, think 16000) while everything else stays 4096/4000,
so DeepSeek/OpenAI subagents don't exceed provider caps (#2614#2806).
- DeepSeek: promote reasoning_content into content when content is empty, via a
fail-open recipe fetch shim (#2617).
- OpenRouter: map openrouter_api_key (config + env) into OPENROUTER_API_KEY
through buildGatewayConfig; register agent.use_gateway_loop, zeroentropy and
openrouter keys in KNOWN_CONFIG_KEYS so `config set` accepts them (#2572,
config key from #2112).
Preserves JSONB (no JSON.stringify into ::jsonb), engine parity, source
isolation, and trust-boundary invariants. Verified with the gbrain-pr-test-env
consumer matrix (clawlancer/Postgres, gstack + hivemindos/PGLite) baseline-FAIL
-> candidate-PASS on the same repro, plus bun run verify (31/31) and 439
targeted unit + e2e tests.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: thomaskong119 <thomaskong119@hotmail.com>
Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: ivandebot <187176982+ivandebot@users.noreply.github.com>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: fbal23 <fbal.public@gmail.com>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: David Carolan <david@joyrestart.com>
Co-authored-by: Masashi-Ono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
Co-authored-by: psam-717 <mphilannorbah@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(config): register Life Chronicle keys so the documented enable command works
The v0.42.56.0 release notes say `gbrain config set auto_chronicle true`,
but the key was never added to KNOWN_CONFIG_KEYS — the documented command
fails with 'Unknown config key' and the operator has to discover --force by
reading source. Registers 'auto_chronicle' plus the 'chronicle.' prefix
(chronicle.tz and future knobs). Same registration class as the v0.42.42.0
spend-controls fix. Regression test pins both.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk
* fix(config): register takes.bootstrap_enabled too — same unregistered-key class
Hit while enabling the takes bootstrap on a live brain: the onboard
remediation's documented enable key fails 'Unknown config key' exactly like
auto_chronicle did. Registered + pinned by the same regression test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk
---------
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Postgres returns BIGINT file sizes as native BigInt values. Returning those values directly from file_list makes JSON serialization fail, and using them in CLI arithmetic can also throw.
Convert size_bytes to Number at the operation boundary and in the CLI display path. File sizes remain exact well beyond any practical attachment size.
Add a unit regression that exercises a native BigInt row and proves the operation result is JSON-serializable, plus a real-Postgres E2E assertion for the file_list response.
* fix(import): skip marked.lexer on fence-less pages to avoid bulk-import OOM (#2437)
extractFencedChunks() ran marked.lexer() on every page body. The lexer
allocates transient memory proportional to page size even when there is no
code fence to extract — a ~2MB table/doc page spikes ~110MB of heap to
produce zero fenced chunks. During bulk import these per-page spikes stack
on accumulated chunk/embedding memory and can OOM the worker; the existing
try/catch cannot rescue an OOM (process death, not a throw).
On a representative brain ~99% of importable pages have no fence, so the
lexer pass is pure wasted work there. Add a fast-path that returns early
when the body contains no fence marker. Matches both ``` and ~~~ so tilde
fenced code still extracts.
Scope: this removes the fence-less transient-allocation surface (the
observed incident). It does not make marked.lexer safe for pages that DO
contain a fence; an input-size/nesting cap is a sensible follow-up.
Tests: add two regression cases — tilde-fenced code still extracts, and a
large fence-less table page imports with zero fenced chunks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(import): match marked's \r normalization in the fence fast-path (#2437)
Self-review follow-up. marked normalizes `\r\n|\r → \n` before lexing, but the
no-fence fast-path probed the raw body with `(^|\n)`. A CR-only (classic-Mac)
line-ended page with a real fenced block would be skipped by the guard while
marked would have extracted it — a lost fenced_code chunk. Widen the line-start
class to `(^|[\r\n])` so the probe agrees with marked. CRLF was already covered.
Add a regression test (CR-only fenced page still extracts); it fails on the old
`(^|\n)` regex and passes on the fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Carry the caller's scalar or federated source scope through every think gather stream: hybrid page retrieval, takes keyword/vector retrieval, and graph traversal. Adds source predicates to the takes retrieval methods in both engines. Part of #2200 (the think slice; #2200 stays open as the tracking issue for the remaining by-slug read ops).
parseRowCells now splits on unescaped pipes only and decodes escaped pipes while preserving ordinary backslashes and empty cells, so facts whose text contains literal | survive the fence->DB reconcile instead of being silently deleted. Fixes#2726.
Bare names resolve only when prefix expansion finds exactly one canonical candidate; ambiguous collisions and low-specificity multi-token fuzzy matches fall through to the guarded holding path instead of confident wrong attribution. Fixes#2723.
Adds the missing timeline_entries.event_page_id forward-reference bootstrap probe + bare-column repair to both engines so pre-v121 brains can replay the current schema and reach migration v121. Fixes#2724.
* fix(ai): drop empty-string env values before merge so they can't clobber config keys (#1249)
Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls; an
unconditional process.env spread let that empty string override a valid
config.json key, breaking every gateway op with NO_ANTHROPIC_API_KEY. Filter
'' / undefined before the merge; '0' and 'false' are preserved.
* fix(ai): normalize native provider base URLs + replace embedding guard with a dims-presence check (#1250, #1292)
#1250: createAnthropic/createOpenAI were called with no baseURL, so an
env-injected bare host (e.g. ANTHROPIC_BASE_URL without /v1) 404'd. Add a
shared resolveNativeBaseUrl and pass a normalized baseURL at all anthropic +
openai native sites (google deferred until its suffix is verified).
#1292/D6: the user_provided_model_unset guard was structurally unreachable as
a no-model check (parseModelId throws on a bare provider) and only ever
false-positived for litellm:<model>, silently disabling vector search. Replace
it with a real dims-presence check for user-provided/zero-default recipes and
delete the dead branch in both consumers. Also stop configureGateway from
fabricating a default embedding_dimensions, so 'no dims set' stays honest.
* fix(ai): trust user-declared embedding dims for local recipes + litellm /v1 hint (#2271, #2209)
#2271: a new trust_custom_dims flag adds a passthrough tier so ollama /
llama-server / litellm accept a user-supplied --embedding-dimensions instead of
being hard-rejected. Fail-closed for fixed-dim providers (openai/voyage/
zeroentropy) and excludes openrouter (declares dims_options). Register modern
ollama embed model names.
#2209: litellm setup_hint now states the /v1 path convention and the docs
pointer is corrected to docs/integrations/embedding-providers.md.
* docs+test(ai): KEY_FILES current-state for provider-agnostic gateway + embed-preflight dims-unset test (#1249, #1250, #1292)
* fix(ai): point user_provided_dims_unset remediation at 'gbrain init' (config set rejects it) + coverage
Pre-landing adversarial review (P1): the new dims-unset guard told users to run
'gbrain config set embedding_dimensions <N>', which config.ts hard-rejects (it's a
schema-sizing field). Both consumer messages now point at 'gbrain init
--embedding-dimensions'. Adds: pgvector-cap-still-fires regression for the
trust_custom_dims passthrough, and a configureGateway backfill-invariant test.
* chore: bump version and changelog (v0.42.57.0)
Provider-agnostic plumbing wave: #1249 empty-env clobber, #1250 native baseURL
normalization, #1292 embedding dims-presence guard, #2271 trust_custom_dims
passthrough, #2209 litellm /v1 hint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: sync embedding-providers guide for provider-agnostic gateway wave (v0.42.57.0)
Post-ship doc drift fix for the v0.42.57.0 AI-gateway wave:
- LiteLLM section now names the /v1 base-URL convention (#2209).
- Ollama section lists the newly-registered modern embedders qwen3-embed-8b
+ snowflake-arctic-embed-l-v2, and notes dims-trust for local recipes (#2271).
- llama-server section notes gbrain trusts the user-declared dimension (#2271).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: post-ship doc sweep for v0.42.57.0 provider-agnostic gateway wave
- KEY_FILES.md types.ts entry: document EmbeddingTouchpoint.trust_custom_dims
(#2271 passthrough tier, runs after dims_options + Matryoshka allowlists)
- ENGINES.md: embedding design-choice note now names the provider-agnostic
gateway delegation instead of the stale OpenAI-only parenthetical
- embedding-providers.md: drop an exact-duplicate doctor-8c paragraph
- llms-full.txt regenerated (ENGINES.md is inlined in the bundle)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: apply codex doc-review findings for v0.42.57.0 (base-URL env note, litellm multimodal)
- embedding-providers.md OpenAI section: document OPENAI_BASE_URL /
ANTHROPIC_BASE_URL bare-host /v1 normalization (#1250 user-facing surface)
- TL;DR table: litellm multimodal is backend-permitting (recipe declares
supports_multimodal: true, routed via the openai-compat multimodal path),
not "no"
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: pin engine-find-trajectory schema to 1536 + stop gateway-state leaks across shard files
CI shard 5 failed 7 findTrajectory tests with 'expected 1280 dimensions, not
1536': engine-find-trajectory hardcodes 1536-d vectors but sizes its schema
from AMBIENT gateway state in beforeAll — which runs before the
legacy-embedding-preload's per-test 1536 restore. A preceding file that ends
with a dimensionless configureGateway (facts-extract-silent-no-op) or a bare
resetGateway poisons the next fresh initSchema down to 1280-d columns. The
new test files in this PR reshuffled shard bin-packing and exposed the trap.
- engine-find-trajectory: pin OpenAI/1536 explicitly before initSchema (the
pattern bunfig's preload documents) — deterministic regardless of neighbors
- facts-extract-silent-no-op, diagnose-embedding-dims, embed-preflight:
restore the legacy 1536 pin in afterAll instead of ending reset/dimensionless
Reproduced: synthetic dimensionless-gateway file + old victim = the exact 7
CI failures; with the pin = 0. Verified in-process pair runs both orders.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pglite): never steal the data-dir lock from a live holder (#2348)
A busy `gbrain dream`/`embed` holder whose 30s heartbeat lapsed (the JS event
loop is blocked during long synchronous WASM imports/CHECKPOINTs) used to get
its lock reaped past the steal-grace window. PGLite/WASM is strictly
single-writer, so a second OS process then opened the same data dir and
corrupted the catalog + pgvector extension state (58P01 / internal_load_library
/ "type vector does not exist"), recoverable only by wipe+restore. Reap ONLY a
dead PID; a live holder is never stolen — a wedged-but-alive or PID-reused
holder makes the acquire time out with a message naming the PID. Removes the
GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS knob (no longer meaningful).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pglite): point a corrupted store at reinit-pglite recovery (#2348)
classifyPgliteInitError gains a `corrupt` verdict for the 58P01 /
internal_load_library / "vector does not exist" / "content_chunks does not
exist" signature (beating the generic wasm-runtime match), so an already-damaged
store gets actionable recovery (gbrain reinit-pglite / restore a backup) instead
of the wrong macOS-WASM hint. Updates KEY_FILES.md to current lock behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* v0.42.55.0 fix(pglite): incident — never steal a live lock + corrupted-store recovery hint (#2348)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* v0.42.56.0 chore: re-bump past #2399 version collision + refresh ownerToken comment
#2399 (security wave) claimed 0.42.55.0; take the next slot. Also updates the
LockHandle.ownerToken JSDoc to current #2348 behavior (live holders are never
reaped, so reap-then-reacquire is dead-holder-only; token guard stays as
defense-in-depth).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): register event + diary page types (#2390)
Life Chronicle Phase A.1. Adds `event` (timeline atom) and `diary`
(first-person interiority) as temporal-primitive page types under
life/events/ and life/diary/, extractable:false — registered in
ALL_PAGE_TYPES, both base schema packs, and the inferType prefix table,
with parity fixtures. Also lands the chronicle read result types
(ChronicleTimelineRow/ChronicleTimelineOpts/LastSeenResult) consumed by
Phase A.2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): event_page_id timeline projection + day/since/last-seen reads (#2390)
Life Chronicle Phase A.2. Adds a nullable event_page_id FK to
timeline_entries (migration v120; mirrored in schema.sql, pglite-schema,
schema-embedded) so a type:event page projects ONE date-index row keyed
to its depth page; a partial UNIQUE(event_page_id, date) makes
re-extraction with a changed summary an update, not a duplicate.
Dual-engine getTimelineForDate / getSince / getLastSeen filter the depth
page on deleted_at, hide soft-deleted event projections at READ time
(not just doctor), order by event effective_date for intra-day sequence,
and honor source scope (sourceIds[] > sourceId). Ops surface as
`gbrain day <date> [--week]`, `gbrain since <date> [--kind]`,
`gbrain last-seen <entity>`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): auto-emit extractor — backstop + chronicle_extract job (#2390)
Life Chronicle Phase A.3. A put_page backstop (gated on status==='imported',
the auto-link/timeline trust gate, and the default-OFF auto_chronicle flag;
diary + event pages never eligible) enqueues a chronicle_extract minion job.
The job runs the extractor off the write path: deterministic when/who, an
injectable LLM judge (default = chat gateway; no-op when no gateway), an
all-or-nothing parse barrier (a malformed proposal writes NOTHING), then
content-addressed event pages + a timeline projection via the new dual-engine
upsertEventProjection (idempotent — re-run yields one event + one projection).
New: src/core/chronicle/{eligibility,config,extract-events,backstop}.ts,
engine.upsertEventProjection (both engines), the jobs.ts handler + a 10-min
timeout. 14 unit tests (eligibility, idempotency, parse barrier, backstop
gating + enqueue) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): quick-capture for diary + manual events (#2390)
Life Chronicle Phase A.5. `gbrain capture` now routes the default slug by
type (diary → life/diary/, event → life/events/, else inbox/) and accepts
--who/--what/--where/--kind/--depth to assemble the `event:` frontmatter
block for `--type event`. User-declared event keys win per-key over the
flags. Goes through the existing put_page → write-through → embed path. 6
new unit tests; existing capture tests stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): bi-temporal per-entity ontology on the facts table (#2390)
Life Chronicle Phase B.10 — the feature's differentiator. Rather than a
parallel store, the open-world per-entity ontology EXTENDS the existing
`facts` table (eng-review G1): migration v121 adds dimension/value/value_hash
/dim_status columns + a deterministic partial-UNIQUE dedup key + an asof read
index. facts already supplies bi-temporal validity, supersession
(superseded_by), visibility (remote redaction), confidence, provenance, and
embedding — all inherited.
Dual-engine methods: mergeOntologyFact (deterministic value_hash dedup →
idempotent retry; same value corroborates; a different value forward-closes
the prior row's valid_until + superseded_by; a BACKDATED conflicting value is
recorded WITHOUT rewriting, surfaced by findOntologyConflicts), getOntology
with `--asof` valid-time travel (expired_at + status + validity-window in the
predicate so quarantined/superseded never leak), discoverOntologyDimensions,
findOntologyConflicts. Novel/LLM-proposed dimensions quarantine; a seed lexicon
canonicalizes name drift (job_role → role). 9 unit tests cover the full
lifecycle; typecheck pins both engines to the interface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): ontology ops — get/add/dimensions/contradictions (#2390)
Life Chronicle Phase B.11. Exposes the bi-temporal ontology over CLI + MCP
(contract-first, auto-generated): `gbrain ontology <entity> [--asof]`,
`gbrain ontology-add <entity> <dim> <value>`, `gbrain ontology-dimensions`
(meta-ontology rollup), `gbrain ontology-contradictions`. All reads route
through sourceScopeOpts. Privacy: ontology_get redacts diary-sourced
observations (source under life/diary/) for untrusted (remote) callers. 3 op
tests (incl. the remote-redaction path); 47 op-registry/tool-def/description
tests stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): agent-context loader — volunteer_chronicle (#2390)
Life Chronicle Phase B.12. `loadChronicleContext` hands an agent the recent
timeline (last N days) + the validity-resolved current ontology for the
entities in play, in one zero-LLM payload, so it orients before acting — the
exact gap behind fumbled chronology. Pure composition over getSince +
getOntology (no new SQL). Exposed as the `volunteer_chronicle` read op
(`gbrain orient [--days] [--entities a,b]`); diary-sourced ontology is redacted
for remote callers. 2 loader tests + op-registry green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): backfill op — sweep existing meetings into events (#2390)
Life Chronicle Phase A.8. `chronicle_backfill` (local-only admin op;
`gbrain chronicle-backfill [--since] [--limit] [--dry-run]`) lists existing
meeting/conversation/calendar pages (source-scoped via listPages), filters
through the chronicle eligibility predicate, and enqueues one chronicle_extract
job per eligible page so existing brains populate the timeline. --dry-run
counts only; per-page enqueue failures are surfaced in `errors`, never
swallowed. 2 op tests (dry-run count + enqueue).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): delight — on-this-day + narrative rendering (#2390)
Life Chronicle Phase A.6 (delight). Dual-engine getOnThisDay (events from the
same month-day in prior years; `gbrain on-this-day [--date]`) reusing the
chronicle JOIN shape (deleted-event hiding + source scope). A pure
renderTimelineNarrative turns timeline rows into prose; `gbrain day --narrative`
returns it alongside the events. 5 tests. (Coverage gap-detection ships with
the advisor collectors next.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): proactive advisor collector (#2390)
Life Chronicle Phase A.7. A brain-state advisor collector surfaces two
proactive signals in `gbrain advisor`: unresolved ontology conflicts (warn,
→ `gbrain ontology-contradictions`) and recent meetings with no timeline
coverage (info, → `gbrain chronicle-backfill`). Advisory-only (no dispatch_id);
runs over MCP too (not workspace-dependent); tolerant of pre-migration brains.
3 tests + advisor-op-gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): doctor chronicle_projection_health probe (#2390)
Life Chronicle Phase B.13. An always-run doctor check (keyed off the
event_page_id schema, NOT a migration verify-hook) counts timeline
projections whose event page is soft-deleted — hidden at read time, surfaced
here as a cleanup backlog (`gbrain integrity auto`). Tolerant of pre-migration
brains. 1 detection test. (auto_chronicle / chronicle.tz flags already work
via getConfig defaults; their docs land with document-release at ship.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): E1 temporal recall — chronicle-type boost on temporal queries (#2390)
Life Chronicle Phase A.4 (E1, ambient temporality). Rather than a separate RRF
arm (which needs chunk hydration + risks the fusion path), E1 is a bounded
post-fusion boost: applyChronicleTypeBoost lifts `event`/`diary` results on
temporal queries, wired INSIDE runPostFusionStages' `recency !== 'off'` branch
so it fires ONLY on temporal intent — non-temporal search is bit-for-bit
unchanged (proven by 110 passing search-path tests). Bounded ([1.0,1.25]) +
floor-gated like the other metadata stages; attribution via `chronicle_boost`.
3 unit tests. (Empirical precision/negative measurement lands with the eval.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(chronicle): feature eval — gbrain eval chronicle (PRIMARY proof) (#2390)
Life Chronicle Phase A.9, the North-Star proof. A deterministic, CI-safe eval
(brings its own in-memory PGLite; no LLM, no gateway) builds a synthetic month
corpus with a known gold chronology + a planted ontology supersession + a
planted conflict, then scores the chronicle layer on six gold tasks: day
reconstruction (intra-day order), last-seen exact date, ontology supersession,
--asof valid-time travel, contradiction surfacing, and source isolation.
`gbrain eval chronicle [--json]` exits 0 iff all pass — currently 6/6. The OFF
baseline (raw meeting pages) structurally can't order intra-day events or
time-travel ontology; the ON path does. (The live-LLM OFF-vs-ON agent arm +
LongMemEval temporal slice are a follow-up; this deterministic bar gates CI.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(chronicle): pre-landing review — conflict validity, parse-barrier date, remote conflict redaction (#2390)
Three bugs caught by the codex pre-landing review on the diff:
1. findOntologyConflicts ignored valid_until, so a normal forward supersession
(founder→advisor from two sources) falsely reported as a live conflict. Now
restricted to currently-open rows (valid_until IS NULL) in both engines.
2. The extractor parse barrier accepted any when-string >= 4 chars; a non-date
value slipped past, wrote the event page, then threw on the projection's
::date cast (partial write). isValidProposal now requires a real parseable date.
3. The ontology_conflicts op had no remote diary redaction (ontology_get did);
remote callers now get diary-sourced values filtered, and conflicts that lose
their disagreement after redaction are dropped.
Three regression tests added; 29 chronicle tests + eval 6/6 green; typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* v0.42.56.0 feat(chronicle): Life Chronicle — temporal timeline + diary + bi-temporal ontology (#2390)
Bump VERSION + package.json to 0.42.56.0 and add the CHANGELOG entry for the
Life Chronicle feature (#2390, closes duplicate #2388).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(chronicle): register #2390 surfaces with the five CI guard suites (#2390)
CI caught five guard tests that pin registration invariants my diff tripped:
- schema-bootstrap-coverage: the four v122 facts ontology columns join
COLUMN_EXEMPTIONS (facts is migration-created; the partial indexes live
inside v122; every reader filters dimension IS NOT NULL — same precedent
as facts.claim_metric et al).
- no-valid-until-write (R8): both engines' mergeOntologyFact forward-
supersession is a deliberate, documented valid_until write authority
(engine-layer, dimension IS NOT NULL only; the contradiction probe still
never mutates).
- doctor-categories: chronicle_projection_health registered under
BRAIN_CHECK_NAMES (same class as child_table_orphans).
- checkTypeProliferation: the test is now threshold-relative (computes
declared from the active pack, seeds declared+6) so base-pack growth
can't silently move the fixed threshold again.
- schema-cli: gbrain-base page-type count 25 → 27 (event + diary), with
assertions on both new types.
All 41 guard tests green; typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: file Life Chronicle follow-up TODOs (v0.42.56.0, #2390)
Eight deferred items from the CEO/eng review decisions (auto-emit default-flip
fast-follow, live eval arm + LongMemEval slice, passive diary consent,
interval-splitting, federation, place-as-entity, meta-ontology dashboard,
materialized daily pages), each with decision provenance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(architecture): KEY_FILES entry for the Life Chronicle module (#2390)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): confine routing dotfiles, skills dir, slugs, and transcription exec
Shared src/core/path-confine.ts consolidates the realpath-containment idiom
(moved from sources-ops.ts) and adds isTrustedDotfile + isWriteTargetContained.
- .gbrain-source (source-resolver) and .gbrain-mount (brain-resolver) walk-up
dotfiles are now lstat trust-gated: a symlink, foreign-owned, or world-writable
file is refused on multi-user hosts (#418), fail-closed on stat error.
- resolveWorkspaceSkillsDir + every skills-dir tier (env, cwd_walk_up, repo_root,
cwd_skills, install_path) route through realpath containment so a symlinked
workspace/skills can't escape the declared workspace (#419).
- resolveSourceId/resolveBrainId realpath both sides of the registered local_path
/ mount prefix match so a symlinked cwd can't misattribute source/brain.
- validateSlug rejects NUL/control, bidi/RTL overrides, backslashes, and
URL-encoded path separators at the shared putPage/updateSlug chokepoint;
write-through confirms the file path stays within the source tree.
- transcribeLargeFile uses execFileSync arg-arrays + fs.rmSync (no shell), so a
path with shell metacharacters is never parsed by a shell (#245).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): default dynamic-registration clients to authorization_code
Self-registered DCR clients (the unauthenticated network registration path)
previously defaulted to the client_credentials grant, which bypasses the
/authorize consent screen. They now default to authorization_code; an explicit
client_credentials request is rejected with invalid_client_metadata unless the
operator opts in with the new --enable-dcr-insecure flag. A loud stderr WARNING
prints at startup whenever DCR is enabled (#1353). Manual CLI/admin client
registration is unchanged (operator-trusted).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): schema-lint hardening migration (search_path + view security_invoker)
Migration v120 brings existing brains to the same posture as fresh installs:
- ALTER VIEW page_links SET (security_invoker = on) on Postgres so the view
honors the caller's RLS instead of the owner's (the view-through-RLS bypass).
- ALTER FUNCTION ... SET search_path on the gbrain-owned trigger/event functions
(both engines, IF EXISTS so engine-only functions are skipped; body untouched,
so the load-bearing auto_enable_rls event trigger is unchanged). Closes#171.
- Broaden the BYPASSRLS preflight in the historical RLS migration gates to honor
superuser and inherited-role BYPASSRLS, so a superuser-connected fresh install
no longer aborts (#1385).
Fresh-install function definitions in schema.sql / pglite-schema.ts are
born-correct (regenerated schema-embedded.ts). scripts/check-search-path.sh is a
new CI guard (wired into verify) that fails if a trigger function in the schema
base files is added without SET search_path. Postgres-only assertions live in the
bootstrap E2E; the PGLite path is covered by test/migration-v120.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* v0.42.55.0 fix(security): dotfile/skills/slug confinement, DCR consent default, schema-lint migration
Bump VERSION + package.json to 0.42.55.0 and add the CHANGELOG entry for the
security-hardening wave (#418#419#245#1353#1647#171#1385).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(security): note the DCR consent default in SECURITY.md (#1353)
The "disable client_credentials, only allow authorization_code" guidance is now
the built-in DCR default; document the new --enable-dcr-insecure escape hatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(todos): add takes_search + code_def to the federated by-slug P1 (#2200)
The v0.42.55.0 eng-review codex pass flagged takes_search (holder-allowlist only)
and code_def (brain-wide raw SQL over content_chunks) as remaining same-class
surfaces. Noted on the existing #2200 P1 follow-up, with the caveat that the
#2399 close-list deliberately keeps #1371/#2200 open until this lands.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): correct plpgsql alias collision in #1385 BYPASSRLS gate (real-PG)
The broadened BYPASSRLS preflight aliased `pg_roles r`, but several RLS DO-blocks
already declare `r record` for their backfill FOR loop, so plpgsql resolved
`r.oid`/`r.rolbypassrls` to the unassigned record variable → "record \"r\" is
not assigned yet" on real Postgres (PGLite tolerated it; the DATABASE_URL-gated
e2e jobs are the backstop). Renamed the subquery alias to `pr` at all 10
migrate.ts sites; also broadened the schema.sql base RLS gate the same way (with
the `pr` alias) for #1385 consistency on superuser fresh installs, and
regenerated schema-embedded.ts.
Also fixes a PRE-EXISTING engine-parity bug (confirmed failing on clean
origin/master): the relationalFanout shape compared `canonical_chunk_id`, a
serial id that diverges between a fresh PGLite engine and a shared Postgres DB
(setupDB TRUNCATEs without RESTART IDENTITY). Compare its presence, not the
exact value.
Validated on real Postgres (pgvector/pg16): migration v120 applies, the v35 RLS
backfill runs, and engine-parity + postgres-bootstrap + jsonb-parity are green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Garry Tan