* fix(search): per-call token-budget meta no longer masks the real cut; restore vacuous search-lite coverage
The search-lite integration tests were structurally vacuous: putPage never
creates chunks and searchKeyword joins content_chunks, so every fixture
query returned zero rows on every machine. The tight-budget cut test's
defensive skip ('keyword search may dedupe by page') silently returned
before its assertions had ever executed anywhere, and the two budget-meta
tests ran against empty result sets.
Restoring the fixture (upsertChunks per page) and hardening the
assertions immediately exposed a real meta bug: with a per-call
tokenBudget, the inner hybridSearch enforces the same resolved budget
(per-call wins in resolveSearchMode) and its meta carries the true
dropped count — but hybridSearchCached re-applies the budget to the
already-cut set and published THAT pass's meta, which always reads
dropped=0. onMeta consumers saw a budget record claiming nothing was
dropped while rows were; telemetry (recorded from the inner meta)
disagreed with the caller-visible meta.
- hybrid.ts finalMeta: prefer innerMeta.token_budget when a per-call
budget is set (outer budgetMeta stays as the fallback and remains the
enforcement for the cache-HIT path, where no inner run exists)
- test fixture: chunk each page (pattern: chunk-grain-fts.test.ts)
- cut test: defensive skip replaced with a hard >=2 precondition;
results non-empty + strictly-fewer-than-unbounded + dropped>0 now
actually execute (revert-checked red on the unfixed meta)
- budget-meta tests: assert non-empty result sets so kept=results.length
can no longer pass vacuously at 0=0
The unbounded 'builder' query returns 2 of 3 fixture pages by design —
dedup Layer 3 caps any single page type at 60% of results and the
fixture is all-person — which the >=2 precondition accommodates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT
* fix(search): cache-HIT budget meta prefers the stored cut record per review; exact-count assertions, mixed-type fixture
- hit path had the symmetric masking (codex P2): the re-application runs
on the already-trimmed stored set and read dropped=0 while the miss
that produced the same result set reported the real cut. Prefer
hit.meta.token_budget unconditionally — tokenBudget is folded into
knobsHash ('tb='), so a hit only ever serves a lookup with the
identical resolved budget as the write and the outer pass can never
cut further (verified against mode.ts; this is why the reviewer's
'outer wins when it drops' branch is unreachable). budgetMeta remains
the fallback for legacy rows stored without a budget record
- new serial test drives a real store-then-hit roundtrip (mocked
embedQuery, real PGLite cache) and pins hit token_budget == miss
token_budget; revert-checked red (dropped=0) on the unfixed path
- lite test: dropped asserted as the exact unbounded-minus-kept count
and used>0 (dropped>0 alone accepts any wrong positive; used<=250
alone accepts a bogus zero), fixture types mixed (person/company/note)
so dedup Layer-3 type-diversity policy no longer shapes the test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(search): classify cache hit/miss in telemetry — hits were invisible, misses unclassified (#2952)
search stats reported 0 hit / 0 miss forever: recordSearchTelemetry fired
only from bare hybridSearch, whose meta never carries a cache field, and a
cache HIT returned from hybridSearchCached before any record at all — so
hit searches also vanished from count/results/tokens/rank-1.
- HybridSearchOpts: internal _telemetryCacheStatus ('miss' | 'disabled')
threaded from hybridSearchCached into the inner hybridSearch (same
pattern as _queryEmbedDeadline), folded into the RECORDED meta only —
onMeta payloads unchanged, count/sum_tokens/budget_dropped/rank-1
behavior byte-identical for the miss/disabled paths
- hit path: record once from hybridSearchCached with the already-built
cachedMeta (cache.status='hit'), post-slice/budget result count, tokens
from the budget pass, and the same rank-1 rule as the inner paths
- bare hybridSearch direct callers (think/gather, brainstorm, enrich,
evals, ...) keep recording exactly as before, with no cache field
- test: serial wiring test drives a real store-then-hit roundtrip through
hybridSearchCached (mocked embedQuery, real PGLite SemanticQueryCache)
and pins the decision matrix (miss / hit / consult-skipped / bare);
revert-checked red on pre-fix source at the miss classification
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT
* fix(search): harden cache-hit telemetry per review — mode-gated tokens, embed-failure coverage
- hit-path tokens_estimate now gated on the MODE-resolved budget,
mirroring the inner paths' resolvedMode.tokenBudget > 0 meta condition
(a tokenmax budget-off brain would otherwise record real tokens on hits
but 0 on misses, inflating avg-tokens as the hit rate rises)
- test: exact token-delta parity assertion (hit contributes the same
tokens as the miss that stored the served set) — catches the class of
hit/miss accounting asymmetry the increase-only check accepted
- test: embed-provider-failure flavor of the disabled path (consult
degrades via catch, keyword fallback serves, neither counter bumps)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of #2946. The hung-COUNT deadline race derived its verdict from a
post-await wall-clock re-check, which races the timer's own drift: on
loaded CI runners setTimeout callbacks fire measurably EARLY relative to
Date.now(), so each padding/boundary adjustment (>=, +1ms) only moved
which wrong status the partial-scan test received ('scanned', then
'partial').
The race's timeout arm now resolves a module-private sentinel; the
sentinel winning IS the deadline verdict (deadlineHit), consulted by the
post-await check without re-reading the clock. A COUNT that resolves
null (failed/absent count) stays distinguishable and does not skip the
scan; a COUNT that resolves slowly without the timer winning is still
caught by the retained wall-clock re-check. The +1ms pad is gone — the
sentinel makes timer drift irrelevant for the hung path.
Verified: partial-scan suite green 8 consecutive runs incl. the new
null-vs-sentinel distinction test.
Claude-Session: https://claude.ai/code/session_01KFWV7zBZFcmD94Vek6xRDT
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The v123 configurable-FTS migration (#2941) logged its completion notice
via console.log. Migrations run lazily inside any command's first DB
connect, so on the nightly heavy run (fresh Postgres service DB) the
line landed as the first line of `gbrain doctor --json` stdout and broke
the fm_wallclock jq parse ("Invalid numeric literal at line 1, column 7",
run 29731426470). runMigrations' contract routes all migration noise to
stderr; move the v123 prints (and the pre-existing v2 slug-rename print)
there.
Also un-vacuous the fm_wallclock harness: its register-source step used
`bun run -e` (bun dumps usage with exit 0 instead of running the code)
and `connect({})` (in-memory), so the source was never registered and
doctor scanned nothing. It now resolves the engine the way the CLI does
and registers the source in the DB doctor actually reads.
Regression test: test/migrate-stdout-clean.test.ts re-runs migrations
from v122 asserting zero stdout writes, plus a source-level guard that
migrate.ts contains no console.log.
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(sync): --src-subpath + --exclude for monorepo subdir-source support (#753)
Rebased port of PR #774 onto current master. A single git repo can hold N
logical sources at subdirectories: git operations run at the discovered
repo root (git rev-parse --show-toplevel — worktrees and submodules
resolve natively) while file walking, imports, deletes and renames are
scoped to the subpath. Passing the subdirectory directly as the repo path
(auto-discovery) works through the same code path.
Path-containment guards (the point of the feature):
- NAV-1/NAV-2: the realpath-resolved scope must live inside the
realpath-resolved git root — ../-traversal and symlinked subdirs
pointing outside the repo are rejected before any git op.
- NAV-1 TOCTOU: per-file realpath re-validation during the incremental
import drain and rename reimport; symlink-escape files are recorded as
failures (fail-closed — the bookmark cannot advance past an escape).
- NAV-4: an --exclude set that filters out every candidate warns loudly.
Scoped syncs use git-root-relative slugs + source_path in BOTH the full
and incremental paths (runImport gains slugRoot), fixing the original
PR's full/incremental slug divergence in the auto-discovery flow.
--exclude matches scope-relative paths in both paths; exclusion never
deletes previously-imported pages. The full-sync delete reconcile is
scope-restricted and relativizes against the slug base so a healthy
scoped source can't trip the #2828 mass-delete valve. .gitignore
management resolves to the git root at every call site.
Preserves all master-side sync work since the original branch: the #2828
mass-delete safety valve, #1794 resumable checkpoints + pinned targets,
#1950 stall watchdog, #2335 heartbeat bump, #1970 bookmark reachability,
and the git-ls-files walker (#2315/#2462/#2678, whose symlink/cycle
hardening is untouched).
Co-authored-by: Jeremy Knows <jeremy@veefriends.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: listEverCommittedPaths uses gitContextRoot after #753 root-triple refactor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Jeremy Knows <jeremy@veefriends.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Squashed superset takeover of the FTS-language trilogy by @rafaelreis-r
(#580 env-var language for query+write side, #581 migration backfill,
#582 gbrain reindex-search-vector), rebased onto current master with the
security review's required fixes applied:
- Migration renumbered v116 -> v123 (master's v116 was already claimed by
code_edges_source_backfill; master is at v122).
- Restored the v120/#1647 search_path hardening: all four CREATE OR
REPLACE trigger-function bodies (migration handler + reindex command)
now carry SET search_path = pg_catalog, public, since CREATE OR REPLACE
resets proconfig and would otherwise strip the hardening on upgrade.
- reindex-search-vector: shared progress reporter (stderr phases
reindex_search_vector.pages/.chunks), id-keyset batched backfill
(5000 rows/UPDATE) instead of single whole-table statements, and
--json no longer bypasses the --yes/TTY confirmation gate.
- Allowlist validation regex + injection tests kept exactly as authored.
- Stale v33/v116 comments swept; docs/guides/multi-language-fts.md
written (README referenced it but no PR added it); llms bundles
regenerated via bun run build:llms.
- Trilogy tests quarantined as *.serial.test.ts (env mutation, per
check-test-isolation).
Verified live on PGLite: fresh init with GBRAIN_FTS_LANGUAGE=portuguese
produces portuguese-stemmed vectors with search_path pinned; the reindex
command retokenizes an english brain to portuguese in place.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Takeover of community PR #2387 with the security review's required fixes
applied. Original work by @harrisali0101.
With GBRAIN_RLS_SCOPE_BINDING=1, source-scoped Postgres read methods wrap
their queries in a transaction that binds set_config('app.scopes', $1, true)
(federated sourceIds CSV > scalar sourceId > '*') so operator-managed RLS
policies can filter rows at the SQL layer — defense-in-depth layer 2 under
the mandatory app-layer source filters.
Review fixes on top of the original PR:
- Flag-off is now a TRUE pass-through: no new per-read transaction wrap
(the #1794 PgBouncer pool-exhaustion class). Only the three search
methods keep a transaction when off — exactly the sql.begin() +
SET LOCAL statement_timeout wrap they already had on master — via the
helper's alwaysTransaction option.
- Preserved the PR's latent setseed fix: listCorpusSample pins
setseed() + SELECT to one connection when seeded (alwaysTransaction
gated on opts.seed), so the deterministic path can't split across
pooled connections.
- Updated the two postgres-engine shape tests to pin the new invariant
(search methods route through withScopedReadTransaction with
alwaysTransaction; helper owns the sql.begin(); flag-off path is
callback(this.sql)).
- New behavioral tests (test/postgres-engine-rls-scope.test.ts): flag-off
pass-through, flag-off alwaysTransaction, flag-on set_config emission,
federated > scalar > '*' precedence, and the CSV as a bound parameter
(never interpolated).
- Fixed the helper header comment to match the actual branching behavior.
- Operator docs in docs/ENGINES.md: env var, policy SQL, the
ALTER ROLE ... SET app.scopes='*' default requirement, FORCE ROW LEVEL
SECURITY for owner roles, and the honest caveat about unwrapped paths.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Harris <79081645+harrisali0101@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Five verified-open fixes to the dream/synthesize output family:
- #1586: thread the cycle's resolved sourceId (cycleSourceId) through
runPhaseSynthesize -> SubagentHandlerData.source_id -> subagent tool
OperationContext, and stamp collected refs + summary page with the same
source, so synthesized pages stop landing in 'default'.
- #2415: new config knob dream.synthesize.output_root (default 'wiki',
zero behavior change unless set) drives the synthesize prompt slug
templates, the patterns reflection lookup + prompt, and remaps the
filing-rule allow-list globs. Registered in KNOWN_CONFIG_KEYS.
- #2163: synthesize_concepts writes concept pages through
importFromContent (put_page's parse->chunk->embed pipeline) instead of
bare engine.putPage, so concepts/ pages are chunked + embedded and
reachable by retrieval.
- #2569: stampDreamProvenance persists dream_generated + dream_cycle_date
into pages.frontmatter (JSONB merge via executeRawJsonb) at write time,
so generated pages are DB-queryable and put_page write-through can't
erase the marker.
- #2606: chronicle judge detects stopReason 'length' truncation and
no-JSON-array parse failures as distinct skipped reasons
(judge_truncated / judge_parse_failed) instead of a false terminal
no_events; output cap raised to 4000 and configurable via
chronicle.judge_max_tokens.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Three verified-open defects, one family: sync silently destroying or
diverging on content it should preserve.
#2404 (P0) — 'ops' was hardcoded in PRUNE_DIR_NAMES (a v0.2.0-era
carve-out), so any path with an ops segment was 'pruned-dir': committed
ops/*.md never imported, and modified ops/* files hit the
unsyncableModified delete loop (whose #1433 guard only spared
'metafile'), silently deleting put-created pages like the bundled
daily-task-manager's canonical ops/tasks on every sync. Fix: remove
'ops' from the prune list (ordinary user content; the vendor/generated
entries stay), and harden the delete loop to also skip 'pruned-dir' —
a page under a pruned dir can only exist via a deliberate put_page.
#2426 (P0) — write-through content stayed DB-only and was deleted by
sync --full. All three compounding bugs fixed:
1. writePageThrough now best-effort commits the artifact (path-limited
git commit) on durability-hardened repos, so the post-commit hook
can push it; result carries committed?: boolean.
2. scripts/brain-commit-push.sh stages+commits BEFORE any pull — the
old fetch+pull-rebase-first order aborted on any dirty tree, so the
helper could never commit a MODIFIED page; brain_push's
rebase-on-reject already handles an advanced remote.
3. The full-sync delete-reconcile partitions stale pages by git
history (listEverCommittedPaths): never-committed source_paths are
DB-only write-through — pages are KEPT and re-exported to the
working tree instead of soft-deleted. Builds on the #2828
mass-delete valve (covers the below-valve cases).
#2607 — the sync --full git ls-files fast path bypassed pruneDir, so a
full pass imported (and resurrected soft-deleted) pages under dot-dirs
and vendored trees that incremental sync excludes. Fix:
isCollectibleForWalker applies the same segment-level pruneDir gate as
classifySync, so full and incremental enumeration agree.
One regression test per defect (all verified failing against master
src): test/sync-ops-pages.serial.test.ts,
test/write-through-commit.serial.test.ts, the #2426 helper-order test
in test/brain-durability-hook.serial.test.ts,
test/sync-reconcile-db-only.serial.test.ts,
test/import-git-fastpath-prune.test.ts.
Fixes#2404Fixes#2426Fixes#2607
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Port of #1837 (mvanhorn) onto current master. The extract_facts cycle
phase unconditionally wipe-and-reinserted every page's fence-owned DB
rows, so re-running a cycle on unchanged content churned rows and — on
the Postgres engine reported in #1781 — accumulated duplicates each run.
The phase now de-dupes extracted facts by the canonical (claim, source)
content key and reconciles the page-scoped DB index: no-op when already
in sync, insert-only for new keys, wipe/reinsert only when stale rows
need cleanup.
Adjustments over the original PR to fit current master:
- preserve #1972's abortSignal threading into the batch embed call
- preserve #1928's excludeSourcePrefixes: ['cli:'] on every wipe, and
exclude cli:-origin conversation facts from the existing-row set so
they neither count as stale (which would force a wipe every cycle)
nor get compared against the fence
Fixes#1781
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four verified-open issues in the subagent-orchestration family, one PR:
- #1594: dream synthesize subagent job/wait timeouts promoted from
hardcoded 30/35-min constants to config keys
dream.synthesize.subagent_timeout_ms / subagent_wait_timeout_ms.
Approach ported from stale PR #1596 (credit @ai920wisco).
- #2778: add_timeline_entry joins the subagent brain-tool allowlist,
fenced fail-closed by the shared enforceSubagentSlugFence (extracted
from put_page's inline check — same trusted-workspace allow-list /
wiki/agents/<id>/ namespace policy). The per-turn output cap is now
resolveMaxOutputTokens (data.max_tokens → agent.max_output_tokens →
8192, was hardcoded 4096); a max_tokens stop surfaces as
stop_reason 'max_tokens' instead of a silent end_turn, and a
mid-tool-round cap hit injects a truncation note so the model
re-issues the dropped call.
- #2782: patterns phase status now reflects the child outcome —
non-complete outcome with zero writes → fail (PATTERNS_CHILD_<OUTCOME>),
partial writes → warn. Patterns timeouts get the same config-key pair
(dream.patterns.subagent_timeout_ms / subagent_wait_timeout_ms).
- #2113: facts extraction cap is config facts.extraction_max_tokens
(default 4000, was hardcoded 1500); stopReason 'length' is checked,
retried once at 2x the cap, and surfaced on stderr instead of
silently extracting zero facts.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: ai920wisco <ai920wisco@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Absorbs PR #2279's intent (drop the hardcoded ANTHROPIC_API_KEY gate) with
the end-state the gateway world actually wants: the patterns phase now
probes the RESOLVED patterns model through probeChatModel(normalizeModelId)
— the same semantics as think/index.ts and synthesize's makeJudgeClient.
Fixes two misclassifications of the old env gate:
- Non-Anthropic stacks (litellm, deepseek, openrouter, ...) were skipped as
"no upstream" even though the subagent routes them through the gateway
(agent.use_gateway_loop). They now pass the gate; their auth is checked
lazily at dispatch and surfaces in the job outcome.
- Anthropic keys set via `gbrain config set anthropic_api_key` (stdio MCP
launches without shell env) were treated as missing. hasAnthropicKey
inside probeChatModel reads both sources.
Skip reason renames no_api_key → no_provider (carrying the probe's detail).
Both pinning tests updated: the structural test asserts the probe wiring;
the PGLite E2E swaps its env-only helper for the shared hermetic
withoutAnthropicKey (env + config file) so it can't flip to a live LLM call
on a dev machine with a config-file key.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Ports the still-unmerged remnant of PR #1891 (#1593 follow-up). Master's
getConfig gained retry-with-reconnect in #1603, but its siblings —
setConfig, unsetConfig, listConfigKeys — still touched `this.sql` bare, so
the first config write/list after a mid-cycle instance-pool teardown threw
the retryable "No database connection" (issue #1678) unhandled instead of
rebuilding the pool.
Adds the connRetry helper from #1891 (same retry+reconnect posture as
batchRetry, but no batch audit JSONL — a config accessor is not a sized
batch), refactors getConfig onto it, and wraps the other three. Writes are
safe to retry: withRetry only retries connection-class failures and both
writes are idempotent (upsert / delete).
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: jalagrange <jalagrange@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Ports the still-unmerged half of PR #2442. OpenAI caches prompt prefixes
automatically, but a stable prompt_cache_key keeps requests that share a
prefix on the same inference engine, lifting the automatic-cache hit rate.
chat() now derives a stable key from the system prompt + sorted tool names
for native-openai models and passes it via providerOptions.openai.
promptCacheKey. Config provider_chat_options still overrides the derived
key; anthropic/google/openai-compatible providers are untouched.
The Anthropic half of #2442 (cache_control "silent no-op") is superseded:
@ai-sdk/anthropic 3.0.74 forwards call-level providerOptions.anthropic.
cacheControl as the Messages API's request-level cache_control (automatic
prefix caching), so master's existing marker is live on current deps.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: CoachRyanNguyen <CoachRyanNguyen@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Port of #1731 (diazMelgarejo) onto current master. gbrain import wrote
~/.gbrain/import-checkpoint.json with the caller's raw dir argument, so
a checkpoint left behind by an interrupted run (e.g. SIGTERM) could carry
"." or a symlinked spelling — an identity that resolves to whatever CWD
the next consumer happens to run from. Downstream tooling that treated
the checkpoint dir as an owned staging boundary could then act on the
wrong directory.
- runImport captures the import target ONCE via resolveImportTargetDir
(resolve + realpathSync) and threads that canonical value through
collection, checkpoint load/save, and resume filtering
- checkpoints are self-describing (schema_version: 1, owner: "gbrain",
kind: "import"); loadCheckpoint tolerates absent metadata (legacy
path-based files) but rejects present-and-wrong metadata and any
relative dir
- checkpoint contract documented in docs/guides/live-sync.md (llms
bundle regenerated)
- test/import-resume.test.ts fixture now realpaths its tmpdir so planted
checkpoints match the canonicalized dir (macOS /var -> /private/var)
Fixes#1728
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Lawrence Melgarejo <Lawrence@cyre.me>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`parseMarkdown` previously walked the lines after the opening `---` and
recorded the first `^#{1,6}\s`-shaped line as a `headingBeforeClose`,
then flagged MISSING_CLOSE when that index came before the actual closing
fence. YAML allows `#` comment lines anywhere inside the document, so a
template that leads with annotation comments inside the fence (e.g. a
`# Research Template` header before the keys) hit a false-positive
MISSING_CLOSE even though the closing `---` was present.
Fix: only walk for the closing `---`. When it is found, content between
the fences is YAML; `#` lines are comments, not headings. When the close
is genuinely missing, surface the first heading-shaped line as a
where-it-went-off-the-rails hint (this path was already correct; we keep
it for the genuine missing-close case).
Two regression tests added to `test/markdown-validation.test.ts`:
- `#` comment lines at the top of a closed frontmatter
- `#` comment lines interleaved with keys
All 68 tests across the four markdown/frontmatter test files stay green.
* fix(extract,ingest,cycle): source-provenance wave — thread source identity through ingest_capture, fs-walk links, and cycle extract (#1522#1747#1503)
Three fixes in the same invariant class (source identity silently dropped
on the write path, collapsing to the 'default' source):
- #1522: the ingest_capture Minion handler validated IngestionEvent
provenance (source_id/source_kind/source_uri) then dropped it on the
importFromContent call. Now threads source_kind/source_uri +
ingested_via='ingest_capture' into the page write, and routes the write
under event.source_id when it names a registered source AND the event
is trusted (fail-closed: untrusted webhook payloads carry a
caller-controlled x-gbrain-source-id header and must not choose their
write source; unregistered emitter ids keep default routing so the
webhook path can't FK-fail).
- #1747: the fs-walk extractors (extractLinksFromDir /
extractTimelineFromDir / extractForSlugs) built batch rows with no
source_id, so addLinksBatch/addTimelineEntriesBatch mapped missing →
'default' and the pages JOIN dropped every row on a non-default source
("Links: created 0 from N pages", no error). ExtractOpts gains
sourceId; the CLI fs path resolves it via resolveSourceId
(--source-id > env > dotfile > registered path > sole-non-default)
and rows are stamped from/to/origin_source_id + timeline source_id.
- #1503: the cycle's extract phase (runPhaseExtract) never passed a
sourceId, so federated-brain dream/autopilot cycles persisted nothing
every night. It now threads cycleSourceId (explicit --source or
resolveSourceForDir(brainDir) — the same seam runPhaseSync uses) into
runExtractCore for both the incremental and full-walk paths.
Regression tests: handler provenance write-through (registered/
unregistered/untrusted routing), fs-walk + CLI resolution + incremental
cycle route landing edges/timeline in the right source (all red on
unfixed master), and a negative control pinning the pre-fix JOIN-drop
shape.
Reuses the ExtractOpts.sourceId threading approach from PR #1719,
rebased onto the current signal-aware signatures and extended to the
cycle + timeline paths.
Fixes#1522Fixes#1747Fixes#1503
Co-authored-by: seungsu <kss530c@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: update cycle source pin for cycleSourceId threading
The #1972 source-pin test asserts the literal runPhaseExtract call site in
cycle.ts. The #1503 fix appended cycleSourceId after opts.signal; signal
threading is unchanged (still the 5th arg, forwarded to runExtractCore).
Pin updated to the new literal — invariant intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: seungsu <kss530c@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Recover the worker-owned Postgres pool when promoteDelayed escapes a retryable connection error, preventing the repeated Promotion error: No database connection loop from issue #1491.\n\nAdds a regression test proving reconnect happens before the worker continues to claim work.
Closes#345.
The bulk-import walker isCollectibleForWalker filtered admitted files by
extension only, while incremental sync excludes README/index/log/schema via
isSyncable -> SYNC_SKIP_FILES. A directory import therefore ingested every
directory README as a folder-titled ghost page that trigram-corrupts fuzzy
entity resolution and inflates orphan count. Apply SYNC_SKIP_FILES (basename
guard) at the top of isCollectibleForWalker so both the FS-walk and the
git-fast-path collection routes agree with sync.
Also add RESOLVER.md to SYNC_SKIP_FILES: a structural routing metafile
(docs-aligned with schema.md/index.md/log.md/README.md), not indexable content.
Co-authored-by: ElliotDrel <ElliotDrel@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(minions): restore rolling conversation prompt-cache on the direct SDK path
Regression since v0.42.51 (@ai-sdk bump): the direct/native subagent
path (agent.use_gateway_loop=false) only marks cache_control on the
static system-prompt and last-tool-def blocks (~5.2K tokens). The
`anthroMessages` array — the part of the request that actually grows
every turn — carries no cache marker at all, so Anthropic re-bills the
full conversation as fresh input on every turn instead of reading it
from cache.
Before this regression, cache_read grew with the conversation
(4.6K -> 125K across a session). Since the regression, cache_read is
pinned at the ~5.2K static prefix regardless of conversation length.
Fix: mark the last content block of the last message with
`cache_control: { type: 'ephemeral' }` on every turn, after first
stripping any stale marker left on an earlier message. Anthropic
caches everything up to the last cache_control breakpoint, so this
turns the trailing marker into a rolling window over the growing
conversation while staying within the 4-breakpoint limit (system +
last-tool + 1 rolling = 3 used).
Measured on a real dream synthesize run: cache_read/input ratio
0.000 -> 32-61 across 23 calls, ~78-80% cost reduction for that run
($8.5 no-cache-equivalent -> $1.71), zero dead-letter jobs.
Neither the gateway path (cache_control placed at the top level,
which @ai-sdk 3.x silently ignores — see #2490) nor #2442 (system +
last-tool only) restores this; both leave the conversation body
unrecovered.
* fix(minions): normalize seed message content before caching it
Codex review caught a real gap: a fresh job's seed user message is
initialized as `content: data.prompt` (a plain string), not a content-
block array. The rolling-cache logic added in the previous commit only
attaches cache_control when `Array.isArray(lastMsg.content)`, so it
silently skipped the very first API call — meaning the common
single-tool round-trip (seed prompt -> tool_use -> tool_result -> done)
got no conversation-cache benefit at all, only jobs with 3+ turns did.
Normalize string content to a one-block text array before checking, so
the first call gets the same rolling breakpoint as every later one.
* fix(minions): retain the prior rolling cache breakpoint, not just the newest
Second Codex review pass: with a single rolling marker, deleting every
prior message's cache_control before placing the new one means a turn
that adds more than 20 content blocks since the last marker (e.g. a
large parallel tool_use/tool_result round) can miss Anthropic's cache
lookup entirely -- the API's automatic prefix search only looks back
up to 20 blocks from a breakpoint to find a prior cached prefix.
Keep the immediately-preceding rolling marker in place instead of
stripping down to one; only evict markers older than that. This still
fits the 4-breakpoint budget (system + last-tool + 2 rolling = 4) and
guarantees the previous marker's prefix remains a valid, already-cached
read even when the new marker's own lookback misses.
The instance-pool reconnect() did disconnect() (nulling _sql) BEFORE connect(),
so a connect() failure during a transient Postgres blip left _sql null for the
rest of the process — every subsequent non-retry-wrapped call then fell through
to the never-connected module singleton and threw 'No database connection',
crashing the autopilot worker into a respawn loop. Build-then-swap: snapshot the
live pool, build a fresh one, end the old only once the new validates, restore
on failure. Keeps upstream's reap-detection + pool-recovery audit. Confirmed by
jalagrange on closed PR #1593; this is the Layer-1 fix he left to us.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The litellm recipe shipped only an `embedding` touchpoint. `getProviderCapabilities()` in `src/core/ai/capabilities.ts` throws when `recipe.touchpoints.chat` is missing, `classifyCapabilities()` returns `'unknown'`, and `enforceSubagentCapable()` in `src/core/model-config.ts` silently falls back to `TIER_DEFAULTS.subagent` (anthropic). Any brain that routes paid traffic through a litellm-style proxy AND has no `ANTHROPIC_API_KEY` then sees every subagent loop dispatch throw `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`. The user's explicit `models.tier.subagent = litellm:*` choice is overridden without their knowledge.
Declare `chat` and `expansion` touchpoints mirroring the openai recipe's shape: `models: []` (litellm proxies arbitrary backends; allowlist is intentionally empty, since `assertTouchpoint` already skips allowlist checks for `tier: 'openai-compat'`), `supports_tools: true`, `supports_subagent_loop: true`, `supports_prompt_cache: false` (OpenAI-compat backends don't honor Anthropic-style `cache_control`), `max_context_tokens: 200_000` (conservative GPT-5-family default; per-deployment override needed for smaller-context backends), costs `undefined` (varies by proxied provider).
Reproduction (deterministic):
1. Fresh brain with no `ANTHROPIC_API_KEY` in env.
2. `gbrain config set models.tier.subagent litellm:gpt-5.4` (or any `litellm:*` string).
3. `gbrain models` warns and falls back to `anthropic:claude-sonnet-4-6`.
4. Submit any subagent job; throws `AIConfigError: Anthropic ... requires ANTHROPIC_API_KEY`.
After the patch, `classifyCapabilities('litellm:gpt-5.4', recipe)` returns `degraded:no_caching` (chat-capable, no Anthropic-style prompt cache). `enforceSubagentCapable` no longer steals the model choice. The subagent loop emits a one-time `degraded:no_caching` warn about prompt-cache absence; cost scales linearly with conversation length, accepted trade for the proxy path.
Rebase-port of #2452 (spinsirr:fix/calibration-profile-scope-and-cli) onto
current master after tonight's merges made the fork branch conflict.
- cli: add 'calibration' to CLI_ONLY so dispatch reaches its existing
handler instead of falling through to "Unknown command" (#2035); honor
--source / GBRAIN_SOURCE in the calibration CLI.
- takes: route takes_list / takes_search / takes_scorecard /
takes_calibration through sourceScopeOpts(ctx) (federated array > scalar
> nothing) and scope engine reads via the take's page.source_id — JOIN
filter for list/search, EXISTS for scorecard/curve — on both engines
(#2200-class).
- bigint: shared takeHitRowToHit coercion in searchTakes /
searchTakesVector (both engines) + bigintToStringReplacer on the cli.ts
output normalizer and the `gbrain call` exit, so int8/BIGSERIAL columns
no longer crash JSON.stringify (#2450); calibration profile id
BIGSERIAL → string.
- calibration: default model ids route through TIER_DEFAULTS
(provider-prefixed) instead of bare model strings; admin calibration
chart endpoints fixed (takes has no page_slug column; month-precision
since_date; Date generated_at; bigint id in drill-down).
The think/gather source-scope slice of the original PR was dropped: it
already landed on master via #2739.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: spinsirr <ID+spinsirr@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
classifyPgliteInitError() routes bare Emscripten aborts to the 'unknown'
verdict, whose hint unconditionally printed "Most common cause: the macOS
26.3 WASM bug (#223)" — even on Windows and Linux (#2195, #1870).
- buildPgliteInitErrorMessage now takes a platform param (default
process.platform): darwin keeps the #223 link as a *possible* cause;
other platforms get the plausible off-macOS causes (lock contention,
damaged data dir) plus `gbrain doctor` / `gbrain reinit-pglite`.
- New stringifyPgliteInitError(): non-Error rejections (Emscripten
aborts can throw plain objects) no longer print "[object Object]".
- Regression tests for both branches + the stringifier in
test/pglite-init-classifier.test.ts.
Canonical for a 7-report class: #2674, #1870, #1195, #1502, #2195,
#939, #391.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add provider_chat_options alongside provider_base_urls and thread it through
the gateway config path into chat(). The chat request now deep-merges
provider-scoped options and model-scoped overrides into providerOptions keyed
by recipe id, preserving existing gateway-built options such as Anthropic
cacheControl and leaving absent-config behavior unchanged.
This lets operators disable thinking for small-budget hybrid-reasoning utility
calls without hardcoding that behavior for every use of those models.
* fix(facts): durable facts-absorb jobs for one-shot CLI processes
Every gbrain capture/put from a short-lived CLI enqueued the facts:absorb
chat into the in-process FactsQueue, then the exit teardown drained for
1-2s and aborted the in-flight call — logging 'pipeline_error: [chat(...)]
The operation was aborted.' on every eligible CLI page write and never
extracting facts.
cli.ts now marks one-shot processes (everything except serve/jobs/
autopilot); runFactsBackstop's queue mode submits a durable facts-absorb
minion job for the long-lived jobs worker instead, with content-hash
idempotency and fallback to the in-process queue if submission fails.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(facts): fence-write resolves source-scoped page path
writeFactsToFence joined local_path + slug directly, writing main-source
fences to the repo ROOT (the default source's tree) and polluting ~/brain
with stray root-level fence files. Route through resolvePageFilePath —
the same helper the put_page write-through and dream-cycle reverse-render
use — so non-default sources fence into .sources/<id>/<slug>.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ragnar Åström <reghar@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The parser/doctor/extractor read the polished page body (compiled_truth +
timeline) instead of the raw turn-by-turn transcript that meeting pages store
in a `raw_transcript` frontmatter sidecar, and the brain's actual raw format
(`Speaker A: ...` / `Speaker B: ...`) had no built-in pattern. Result: scan
returned no_match / 0 messages and conversation-fact extraction produced 0
segments / 0 facts.
- new readConversationBodyForParsing(): prefer the raw_transcript sidecar when
present, fall back to compiled_truth + timeline (src/core/conversation-parser/body.ts)
- wire it into conversation-parser scan, doctor coverage check, and
extract-conversation-facts (drops the old readPageBody helper)
- add a built-in `speaker-letter-no-time` pattern for plain `Speaker A:` lines
Verified: scan now returns phase=regex_match (speaker-letter-no-time), and the
Ben page extracts 61 facts / 7 segments. Tests added; suite green.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
since/until range filters were applied against updated_at, so edited-but-old
entries leaked into time-bounded queries. Filter on the effective date instead.
Closes#1520
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* feat(extract): recognize inline [Source: ..., YYYY-MM-DD] citations as timeline entries
gbrain's own quality conventions (skills/conventions/quality.md) require a
dated [Source: ..., YYYY-MM-DD] citation on every brain write, so curated
pages are full of dated evidence — but extractTimelineFromContent only
recognized the timeline-bullet and date-header formats. A page whose dates
all live in citations scored zero timeline coverage in brain_score, and
doctor pointed users at a formatting convention their own citations already
satisfied in spirit.
Format 3 files one entry per citation: date and source from the marker,
summary from the annotated line with citation markers stripped. Lines
already captured by Format 1 are skipped so a timeline bullet carrying its
own citation is not double-filed. Bare citations with no surrounding text
are ignored.
Idempotency is unchanged: persistence already dedupes at the DB layer.
* fix(extract): Format 3 citations also in parseTimelineEntries (db-source + ingest path)
The first commit only taught extractTimelineFromContent (fs-source) the
citation format; the db-source extract and the ingest path parse through
parseTimelineEntries in core/link-extraction.ts, which still could not see
citations. Same rules as the fs parser: bullet-captured lines skipped,
bare citations ignored, invalid calendar dates rejected; the citation
source is preserved in the entry detail.
---------
Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com>
extractTakesFromPages selected pages by updated_at DESC + LIMIT with no
exclusion of pages that already hold takes. The CLI clamps --max-pages to
1000, so on a corpus larger than one run every re-run rescanned the same
most-recent 1000: the older tail could never be bootstrapped, and each
rescan re-spent Haiku budget producing upsert-identical rows. Seen live on
a 2,311-eligible-page brain — a second run would have covered 0 new pages.
Covered pages are now skipped by default (NOT EXISTS on takes.page_id), so
repeat runs sweep a large corpus in slices; --include-covered restores the
full rescan for refresh use. Usage text documents both plus the 1000 clamp.
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
extract_atoms minted duplicate atom pages two ways:
A. Trailing-dash twins. The local slugger truncated the title at 60 chars
with no re-strip, so a cut landing on a hyphen left a trailing dash
(`…would-`). The FS-import normalizer (slugifySegment) strips it
(`…would`), so the same atom persisted under two slugs and the
dedup-by-slug check never collapsed them.
B. Cross-day re-mint. The slug used the run date (todayDate()), while the
idempotency guard keys on the whole-file source_hash. Append-only
sources (chat/transcript exports) grow daily, so the file hash changes,
the guard never matches, the source is re-extracted, and each re-mint
lands under a new date prefix → a new slug → no upsert → a duplicate.
Fix: make the atom slug a deterministic function of stable inputs —
`atoms/<source-date>/<stem>-<title-hash>`:
- source date is parsed from the source ref (transcript filename / page
slug), not the run date, so re-extraction converges on the same slug and
putPage upserts instead of duplicating;
- the 6-char title hash keeps two atoms whose titles share the first 60
chars on distinct slugs (no silent clobber of a different atom);
- the stem routes through the canonical slugifySegment and re-strips a
trailing dash, so the two write paths can no longer disagree.
The whole-file source_hash batch check is retained only as a cost fast-path
(skip re-running the model on an unchanged source); correctness no longer
depends on it.
Adds a hermetic PGLite regression test (no DATABASE_URL) asserting the
source-dated prefix, title-hash suffix, trailing-dash strip, and upsert on
re-extraction of a grown append-only transcript.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Atom-extraction page discovery hardcoded EXTRACTABLE_PAGE_TYPES and ignored the
active pack's `extractable: true` flags, so a type declared extractable in the
manifest (e.g. `note`) never actually extracted. Closes the D2 TODO the code
already flagged (extract-atoms.ts: 'future pack-aware refactor ... pull from the
active pack manifest').
Resolve the allowlist as: legacy hardcoded floor UNION the pack's extractable
types, MINUS synthesis outputs (atom, concept — extracting from these would loop,
since concepts are synthesized from atoms). Mirrors facts/eligibility.ts, which
excludes concept the same way. Back-compat: gbrain-base brains keep every legacy
target via the union; fail-soft falls back to the legacy floor if the pack can't
load.
- pure unionExtractableTypes() policy, unit-tested
- discoverExtractablePages + countExtractAtomsBacklog resolve from the pack
- page-discovery fixture updated (note now extracts; concept stays excluded)
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
claude-sonnet-5 and claude-fable-5 are GA Anthropic models, but neither
was in CANONICAL_PRICING. A brain routing a tier to them (e.g.
models.tier.reasoning = anthropic:claude-sonnet-5) ran with cost
telemetry blind on that tier: canonicalLookup missed, the budget meter
logged BUDGET_METER_NO_PRICING and disabled the gate, and cost views
under-reported spend.
- model-pricing.ts: anthropic:claude-sonnet-5 at $3/$15 and
anthropic:claude-fable-5 at $10/$50. Sonnet 5's launch intro discount
($2/$10 through 2026-08-31) is deliberately not modeled — the table
carries standard rates so estimates stay conservative and the entry
needs no time-bombed edit when the promo lapses.
- takes-quality-eval/pricing.ts: claude-sonnet-5 added to the curated
SUPPORTED_MODELS allowlist (a likely judge override). Fable 5 stays
out — priced for warn-only consumers, not a budgeted-eval panel model.
- model-pricing.test.ts: pin tests for both rows, matching the existing
Opus 4.8/4.7 pattern. Drift guards iterate the table; no changes.
All derived views (ANTHROPIC_PRICING bare view -> budget-tracker,
batch-projection, budget-meter) pick the rows up automatically.
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
resolveHardExcludes() only ran at DB-query build time (cache miss), so
query_cache rows written by a process without GBRAIN_SEARCH_EXCLUDE could
be served to a process with it (and vice versa), leaking excluded slugs.
hybridSearchCached now resolves the effective hard-exclude list exactly
as the engines' query-build path does and folds it (sorted, append-only
hx= part) into knobsHash via a new KnobsHashContext.hardExcludes field.
KNOBS_HASH_VERSION 11 -> 12: one-time global cache cold-miss on upgrade,
refills within cache.ttl_seconds.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Collector branch superseding the gateway tool-loop duplicate cluster and
adjacent provider fixes. Re-implemented from the best of each PR (deduped by
content, not file-overlap); every fix carries test coverage.
Gateway tool-loop resume (supersedes #1934#2062#2065#2112#2274#2487#2336#2257#2499#2491, test #2063):
- toolLoop now persists the tool-result user turn per round (onToolResultTurn),
so a resumed subagent job reloads a balanced transcript instead of dangling
assistant tool-calls that non-Anthropic providers reject with
AI_MissingToolResultsError.
- runSubagentViaGateway reconciles an already-corrupted transcript on resume:
it heals every dangling assistant tool-call turn (not just the tail) from the
settled subagent_tool_executions rows, re-dispatching idempotent-pending
tools and throwing on non-idempotent, mirroring the legacy Anthropic path.
Terminal early-return for a transcript that already reached end_turn.
- repairToolPairing() is a last-resort normalization at the chat() boundary
(from #2336): back-fills error stubs for any assistant tool-call still
unanswered (partial turns, provider-duplicated/dropped IDs on local models,
length-truncated batches). No-op on balanced input.
- toModelMessages is Date-safe (Postgres timestamptz -> ISO via a JSON round
trip at the SDK boundary, never a ::jsonb cast; degrades bigint/circular to a
string instead of throwing) and drops non-string text blocks reasoning models
emit that AI SDK v6 rejects (#2488).
Adjacent provider fixes:
- Model-aware default max output tokens: thinking-by-default Claude 5 models get
headroom (gateway 32000, think 16000) while everything else stays 4096/4000,
so DeepSeek/OpenAI subagents don't exceed provider caps (#2614#2806).
- DeepSeek: promote reasoning_content into content when content is empty, via a
fail-open recipe fetch shim (#2617).
- OpenRouter: map openrouter_api_key (config + env) into OPENROUTER_API_KEY
through buildGatewayConfig; register agent.use_gateway_loop, zeroentropy and
openrouter keys in KNOWN_CONFIG_KEYS so `config set` accepts them (#2572,
config key from #2112).
Preserves JSONB (no JSON.stringify into ::jsonb), engine parity, source
isolation, and trust-boundary invariants. Verified with the gbrain-pr-test-env
consumer matrix (clawlancer/Postgres, gstack + hivemindos/PGLite) baseline-FAIL
-> candidate-PASS on the same repro, plus bun run verify (31/31) and 439
targeted unit + e2e tests.
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
Co-authored-by: thomaskong119 <thomaskong119@hotmail.com>
Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com>
Co-authored-by: brettdavies <brettdavies@users.noreply.github.com>
Co-authored-by: ivandebot <187176982+ivandebot@users.noreply.github.com>
Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br>
Co-authored-by: fbal23 <fbal.public@gmail.com>
Co-authored-by: javieraldape <javieraldape@users.noreply.github.com>
Co-authored-by: David Carolan <david@joyrestart.com>
Co-authored-by: Masashi-Ono0611 <masashi.ono.0611@gmail.com>
Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com>
Co-authored-by: psam-717 <mphilannorbah@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(config): register Life Chronicle keys so the documented enable command works
The v0.42.56.0 release notes say `gbrain config set auto_chronicle true`,
but the key was never added to KNOWN_CONFIG_KEYS — the documented command
fails with 'Unknown config key' and the operator has to discover --force by
reading source. Registers 'auto_chronicle' plus the 'chronicle.' prefix
(chronicle.tz and future knobs). Same registration class as the v0.42.42.0
spend-controls fix. Regression test pins both.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk
* fix(config): register takes.bootstrap_enabled too — same unregistered-key class
Hit while enabling the takes bootstrap on a live brain: the onboard
remediation's documented enable key fails 'Unknown config key' exactly like
auto_chronicle did. Registered + pinned by the same regression test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk
---------
Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Postgres returns BIGINT file sizes as native BigInt values. Returning those values directly from file_list makes JSON serialization fail, and using them in CLI arithmetic can also throw.
Convert size_bytes to Number at the operation boundary and in the CLI display path. File sizes remain exact well beyond any practical attachment size.
Add a unit regression that exercises a native BigInt row and proves the operation result is JSON-serializable, plus a real-Postgres E2E assertion for the file_list response.
* fix(import): skip marked.lexer on fence-less pages to avoid bulk-import OOM (#2437)
extractFencedChunks() ran marked.lexer() on every page body. The lexer
allocates transient memory proportional to page size even when there is no
code fence to extract — a ~2MB table/doc page spikes ~110MB of heap to
produce zero fenced chunks. During bulk import these per-page spikes stack
on accumulated chunk/embedding memory and can OOM the worker; the existing
try/catch cannot rescue an OOM (process death, not a throw).
On a representative brain ~99% of importable pages have no fence, so the
lexer pass is pure wasted work there. Add a fast-path that returns early
when the body contains no fence marker. Matches both ``` and ~~~ so tilde
fenced code still extracts.
Scope: this removes the fence-less transient-allocation surface (the
observed incident). It does not make marked.lexer safe for pages that DO
contain a fence; an input-size/nesting cap is a sensible follow-up.
Tests: add two regression cases — tilde-fenced code still extracts, and a
large fence-less table page imports with zero fenced chunks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(import): match marked's \r normalization in the fence fast-path (#2437)
Self-review follow-up. marked normalizes `\r\n|\r → \n` before lexing, but the
no-fence fast-path probed the raw body with `(^|\n)`. A CR-only (classic-Mac)
line-ended page with a real fenced block would be skipped by the guard while
marked would have extracted it — a lost fenced_code chunk. Widen the line-start
class to `(^|[\r\n])` so the probe agrees with marked. CRLF was already covered.
Add a regression test (CR-only fenced page still extracts); it fails on the old
`(^|\n)` regex and passes on the fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Carry the caller's scalar or federated source scope through every think gather stream: hybrid page retrieval, takes keyword/vector retrieval, and graph traversal. Adds source predicates to the takes retrieval methods in both engines. Part of #2200 (the think slice; #2200 stays open as the tracking issue for the remaining by-slug read ops).
parseRowCells now splits on unescaped pipes only and decodes escaped pipes while preserving ordinary backslashes and empty cells, so facts whose text contains literal | survive the fence->DB reconcile instead of being silently deleted. Fixes#2726.
Bare names resolve only when prefix expansion finds exactly one canonical candidate; ambiguous collisions and low-specificity multi-token fuzzy matches fall through to the guarded holding path instead of confident wrong attribution. Fixes#2723.
Adds the missing timeline_entries.event_page_id forward-reference bootstrap probe + bare-column repair to both engines so pre-v121 brains can replay the current schema and reach migration v121. Fixes#2724.
* fix(ai): drop empty-string env values before merge so they can't clobber config keys (#1249)
Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls; an
unconditional process.env spread let that empty string override a valid
config.json key, breaking every gateway op with NO_ANTHROPIC_API_KEY. Filter
'' / undefined before the merge; '0' and 'false' are preserved.
* fix(ai): normalize native provider base URLs + replace embedding guard with a dims-presence check (#1250, #1292)
#1250: createAnthropic/createOpenAI were called with no baseURL, so an
env-injected bare host (e.g. ANTHROPIC_BASE_URL without /v1) 404'd. Add a
shared resolveNativeBaseUrl and pass a normalized baseURL at all anthropic +
openai native sites (google deferred until its suffix is verified).
#1292/D6: the user_provided_model_unset guard was structurally unreachable as
a no-model check (parseModelId throws on a bare provider) and only ever
false-positived for litellm:<model>, silently disabling vector search. Replace
it with a real dims-presence check for user-provided/zero-default recipes and
delete the dead branch in both consumers. Also stop configureGateway from
fabricating a default embedding_dimensions, so 'no dims set' stays honest.
* fix(ai): trust user-declared embedding dims for local recipes + litellm /v1 hint (#2271, #2209)
#2271: a new trust_custom_dims flag adds a passthrough tier so ollama /
llama-server / litellm accept a user-supplied --embedding-dimensions instead of
being hard-rejected. Fail-closed for fixed-dim providers (openai/voyage/
zeroentropy) and excludes openrouter (declares dims_options). Register modern
ollama embed model names.
#2209: litellm setup_hint now states the /v1 path convention and the docs
pointer is corrected to docs/integrations/embedding-providers.md.
* docs+test(ai): KEY_FILES current-state for provider-agnostic gateway + embed-preflight dims-unset test (#1249, #1250, #1292)
* fix(ai): point user_provided_dims_unset remediation at 'gbrain init' (config set rejects it) + coverage
Pre-landing adversarial review (P1): the new dims-unset guard told users to run
'gbrain config set embedding_dimensions <N>', which config.ts hard-rejects (it's a
schema-sizing field). Both consumer messages now point at 'gbrain init
--embedding-dimensions'. Adds: pgvector-cap-still-fires regression for the
trust_custom_dims passthrough, and a configureGateway backfill-invariant test.
* chore: bump version and changelog (v0.42.57.0)
Provider-agnostic plumbing wave: #1249 empty-env clobber, #1250 native baseURL
normalization, #1292 embedding dims-presence guard, #2271 trust_custom_dims
passthrough, #2209 litellm /v1 hint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: sync embedding-providers guide for provider-agnostic gateway wave (v0.42.57.0)
Post-ship doc drift fix for the v0.42.57.0 AI-gateway wave:
- LiteLLM section now names the /v1 base-URL convention (#2209).
- Ollama section lists the newly-registered modern embedders qwen3-embed-8b
+ snowflake-arctic-embed-l-v2, and notes dims-trust for local recipes (#2271).
- llama-server section notes gbrain trusts the user-declared dimension (#2271).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: post-ship doc sweep for v0.42.57.0 provider-agnostic gateway wave
- KEY_FILES.md types.ts entry: document EmbeddingTouchpoint.trust_custom_dims
(#2271 passthrough tier, runs after dims_options + Matryoshka allowlists)
- ENGINES.md: embedding design-choice note now names the provider-agnostic
gateway delegation instead of the stale OpenAI-only parenthetical
- embedding-providers.md: drop an exact-duplicate doctor-8c paragraph
- llms-full.txt regenerated (ENGINES.md is inlined in the bundle)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: apply codex doc-review findings for v0.42.57.0 (base-URL env note, litellm multimodal)
- embedding-providers.md OpenAI section: document OPENAI_BASE_URL /
ANTHROPIC_BASE_URL bare-host /v1 normalization (#1250 user-facing surface)
- TL;DR table: litellm multimodal is backend-permitting (recipe declares
supports_multimodal: true, routed via the openai-compat multimodal path),
not "no"
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: pin engine-find-trajectory schema to 1536 + stop gateway-state leaks across shard files
CI shard 5 failed 7 findTrajectory tests with 'expected 1280 dimensions, not
1536': engine-find-trajectory hardcodes 1536-d vectors but sizes its schema
from AMBIENT gateway state in beforeAll — which runs before the
legacy-embedding-preload's per-test 1536 restore. A preceding file that ends
with a dimensionless configureGateway (facts-extract-silent-no-op) or a bare
resetGateway poisons the next fresh initSchema down to 1280-d columns. The
new test files in this PR reshuffled shard bin-packing and exposed the trap.
- engine-find-trajectory: pin OpenAI/1536 explicitly before initSchema (the
pattern bunfig's preload documents) — deterministic regardless of neighbors
- facts-extract-silent-no-op, diagnose-embedding-dims, embed-preflight:
restore the legacy 1536 pin in afterAll instead of ending reset/dimensionless
Reproduced: synthetic dimensionless-gateway file + old victim = the exact 7
CI failures; with the pin = 0. Verified in-process pair runs both orders.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
