Reviewed (A7): extracts parseLegacyTokenScope into src/core/legacy-token-scope.ts
shared by the HTTP transport + OAuth provider (no duplicate logic). Closes the
OAuth-provider gap where legacy permissions.source_id grants collapsed to 'default'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewed (A6): omitted/empty scope both safely default to the client's
REGISTERED scope and stay clamped to it (no escalation); fixes the empty-grant
that never self-heals. Verified the result can't exceed allowedScopes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewed: doctor.ts only; the five fixes sit in different checks than the
authored #2038 timeline_dedup_index check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewed: single-statement config || patch eliminates the read-then-write
lost-update race. Touches updateSourceConfig (disjoint from the deleteFactsForPage
region the authored #1928 fix changed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewed: gateway.ts AsyncLocalStorage + search/mode.ts; disjoint from authored
fixes. Chosen over #2083 (whole openai-compatible recipe class vs ZeroEntropy-only).
Credit @billy-armstrong for the original #1400 diagnosis.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reviewed: same walker as #2052; venv glob addition. Resolved on top of #2052.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts:
# src/commands/import.ts
# src/core/sync.ts
Reviewed: config-layer only (config.ts, init.ts, migrate-engine.ts); disjoint
from the authored fixes. Prevents a cwd .env DATABASE_URL from silently
retargeting the brain at the wrong DB.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The meetings extractor's bare catch {} hid a brain-wide timeline-write failure
(0 entries, no error). It now counts + surfaces batch errors. Adds a Date-bearing
batch regression test proving the #1861 jsonb_to_recordset refactor already
fixed the original ::text[] cast failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A migration renumbered during a merge (v102) could be recorded-as-applied
without its DDL running, leaving the 3-column index so every timeline write
failed the 4-column ON CONFLICT. runMigrations now always runs a shape-keyed
drift repair (dedupe-then-rebuild) even when no migration is pending, and
doctor surfaces the drift.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A live holder's lock was force-removed after 5min age alone, letting a second
process share the single-writer data dir -> WAL corruption. The lock now
heartbeats while held; a holder is reaped only when its PID is dead OR its
heartbeat went stale past the steal grace. Pairs PID liveness with heartbeat
age to also defeat PID reuse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
put_page write-through resolved the disk target from the global sync.repo_path,
so a default-source page (local_path NULL) got written into an unrelated
federated source's working tree. Now it uses the assigned source's own
local_path; NULL local_path skips (no leak); the global path is used only as a
sole-source fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The autopilot health-probe recovery called connect() with no args after
disconnect(), losing the startup config (database_url undefined -> FATAL
restart-loop on every DB blip) and opening a null-pool window. Both call sites
now use engine.reconnect(), which restores the captured config.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fence reconcile delete-then-reinsert wiped cli:-origin facts (no fence to
recreate them); a failed-sync full walk turned it brain-wide (1829 rows, 0
reinserted, status ok). Now: exclude cli: rows from the wipe, do NOT inherit
the failed-sync->full-walk fallback for this destructive phase, and warn on
net-negative reconcile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Engine-layer API for two cycle/availability fixes that share these files:
- deleteFactsForPage gains optional excludeSourcePrefixes so the fence
reconcile can protect non-fence facts (e.g. cli: conversation facts).
- reconnect(ctx?) is now a first-class BrainEngine method on both engines
(PostgresEngine already had it; PGLite gains config capture + reconnect)
so callers stop using disconnect()+bare connect().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(extract,ingest): well-form lone UTF-16 surrogates before jsonb (#2011)
excerpt() in link-extraction.ts sliced the link-context window by raw UTF-16
index, so a boundary landing inside a non-BMP char (emoji, math, CJK) left an
unpaired surrogate half in `context`. Serialized to JSONB for the
jsonb_to_recordset batch insert, Postgres rejects it at the ::jsonb cast and
aborts the whole batch — wedging `extract --stale` because the staleness
bookmark only advances on a clean finish.
- text-safe.ts: new ensureWellFormed() (Bun isWellFormed/toWellFormed) — one
shared surrogate-cleaning primitive.
- link-extraction.ts: excerpt() well-forms the slice (root-cause fix).
- batch-rows.ts: new sanitizeForJsonb() = ensureWellFormed(stripNul(s)) applied
to every free-text body field (link context; timeline summary/detail/source;
take claim/source). Identity/security fields stay un-sanitized and fail closed.
- postgres-engine.ts + pglite-engine.ts: scalar addLink + addTimelineEntry use
sanitizeForJsonb too, matching the batch path on both engines.
- brainstorm/orchestrator.ts: consolidate hand-rolled sanitizeUnicode onto
ensureWellFormed (also fixes consecutive-lone-surrogate mishandling).
Tests: ensureWellFormed unit cases (incl. consecutive lone surrogates), an
excerpt window-split regression, PGLite + Postgres-e2e surrogate cases across
all free-text fields and scalar paths, and fail-closed identity-field tests
proving sanitization was NOT extended to slugs/holders.
* v0.42.39.0 chore: bump version and changelog (#2011)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: update project documentation for v0.42.39.0
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* v0.42.40.0 chore: re-slot release version (was 0.42.39.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: fix env-mutation isolation violations in retrieval-reflex tests
check:test-isolation (rule R1) flags direct process.env mutation in
non-serial test files — bun's parallel runner loads multiple files into
one process, so a leaked GBRAIN_RETRIEVAL_REFLEX flips reflex behavior
in unrelated tests. Both files landed via the #2019 merge; convert the
beforeEach/afterEach env juggling to the canonical withEnv() wrapper,
which restores the prior value via try/finally even on throw.
Fixes the failing `verify` CI check on #2031 (and the `test-status`
aggregate that inherits it). All 30 verify checks green locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(integrations): parameterize resolver-row fence by recipe id
The install fence was hardcoded gbrain:agent-voice:resolver-rows, so any
second copy-into-host-repo recipe wrote a mislabeled block (and refresh/
uninstall keyed on recipe id would miss it). Derive it from manifest.recipe.
* feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981)
Deterministic per-turn pointer layer in the context engine: a zero-LLM,
precision-biased scan resolves salient entities (names, @handles) to existing
brain pages and injects compact pointers (name → slug → safe synopsis). Detect
+ point, never auto-dump. Fail-open, capped, suppression on prior context only.
Engine-aware resolver ladder (no second DB connection): host ctx.brainQuery →
PGLite serve resolve IPC (unix socket) → Postgres cached direct → disabled.
Synopsis runs through get_page's privacy strip. Plus the retrieval-reflex recipe
+ policy skill, the retrieval_reflex_health doctor check, config gate, and the
init next-step hint.
* chore: bump version and changelog (v0.42.38.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(architecture): KEY_FILES entries for Retrieval Reflex surface (#1981)
Document the new src/core/context/ modules, the context-engine resolver
ladder, the serve resolve IPC, the retrieval_reflex_health doctor check,
and the recipe-id-keyed install fence. Current-state only.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## 1. Stale lock break hints cover gbrain-cycle: keys
The doctor stale-lock report only recognized `gbrain-sync:` lock prefixes;
everything else fell back to `gbrain sync --break-lock`, which is wrong for
dream/autopilot cycle locks. A `gbrain-cycle:<source>` or `gbrain-cycle`
lock now suggests `gbrain dream --break-lock [--source <name>]`, and
unknown lock shapes fall back to `gbrain doctor` instead of a
misleading sync command.
## 2. content_sanity_audit_recent counts reject and quarantine as hard failures
v0.42 renamed the hard disposition path: rejected pages emit a `reject`
event and quarantined junk pages emit `quarantine`; `hard_block` is now
only the pre-v0.42 legacy alias. The status check only counted `hard_block`,
so fresh `reject` / `quarantine` events from the new path cleared as `ok`
whenever fewer than 10 events existed. The check now sums all three for the
hard count, and `soft_block + flag` for the soft count.
## 3. graph_coverage excludes test fixture entity pages from the denominator
Brains seeded with code sources (e.g. a sync of the gbrain repo itself)
could accumulate test fixture pages typed as `entity` / `person`. Including
these in the entity-count denominator diluted coverage and produced spurious
warnings ("Entity link coverage 0%, timeline 0%") on knowledge-only brains
with no real entity pages. The check now queries a per-entity stats CTE that
excludes `tools/gbrain/test/*` slugs and the `templates/new-person` stub,
with an additional guard for the all-fixture case (`eligibleEntityCount = 0`).
## 4. process.exitCode instead of process.exit at doctor main exit point
`process.exit(hasFail ? 1 : 0)` was a hard kill that prevented cleanup
handlers (Bun unload events, open DB connections) from running. Using
`process.exitCode = hasFail ? 1 : 0` defers the actual termination until
the end of the event loop, allowing cleanup to complete.
## 5. checkSubagentCapability exported for test seams + gateway loop guard
The function was private, making it untestable in isolation. It is now
exported. Additionally, users running gbrain with a non-Anthropic chat model
via `agent.use_gateway_loop=true` no longer receive a spurious warning that
`ANTHROPIC_API_KEY` is missing — subagents route via the gateway loop in
that configuration and do not need the key directly.
## Tests
Doctor test suite: 77 pass, 0 fail (no regressions).
## Problem
`updateSourceConfig` used a read-then-write pattern: read the current
`config` row, normalize it in JavaScript, then write the merged result
back with `SET config = <normalized> || <patch>`.
Under concurrent callers (two background autopilot/cycle paths patching
different keys simultaneously), both callers can read the same stale
row. The later `SET config = ...` then clobbers the earlier patch,
silently dropping whatever keys the first caller wrote. Reproduced
at 21/25 lost-update events under real Postgres with parallel callers.
## Fix
Fold the normalization and merge into a single atomic `UPDATE … SET
config = CASE … END || patch` statement. Because the `SET` expression
evaluates against the row-locked latest version of `config`, there is
no snapshot window between the read and the write. Concurrent callers
now converge correctly (50/50 clean in reproduction test).
The `CASE` also normalizes historical bad JSONB shapes inline:
- `object` — used as-is
- `string` — double-encoded config; inner text parsed with the SQL
`IS JSON` guard (Postgres 16+) so unparseable strings fall back to
`{}` instead of raising `invalid input syntax for type json`
- `array` — array of patch objects aggregated into a flat object via
`jsonb_object_agg`
- anything else — falls back to `{}`
`pglite-engine.updateSourceConfig` already used an atomic `||` merge;
this change brings postgres-engine to parity.
## Test
Added two assertions to `test/list-all-sources.test.ts`:
1. JSONB string holding non-JSON text normalizes to `{}` (no cast throw)
2. JSONB string holding double-encoded valid JSON is parsed then merged
Marks the v0.42.20.0 'decouple the op-dispatch force-exit timer' follow-up
complete — this branch ships exactly that decoupling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review: buildFactsAlterRecipe shipped the same defect class
this branch fixes for content_chunks 350 lines up — a cross-dimension
ALTER ... USING cast that pgvector refuses while rows hold old-width
vectors. Dimension changes now wipe first (the facts pipeline re-embeds
on next write); same-dim type swaps (halfvec <-> vector) keep the
lossless cast and PRESERVE data. Both behaviors pinned by tests.
Adversarial review: addCodeEdges still wrote e.source_id ?? null, so any
future caller that forgets the field reintroduces invisible NULL edges
the day after the v116 backfill runs. A NULL source_id is invisible to
every scoped call-graph query; default to the schema-default source the
way the pages table does. Applied to both engines (parity).
Adversarial review: txOpts used truthiness while the edge stamp used
nullish — sourceId:'' put pages under 'default' but stamped edges '',
FK-violating against sources(id) and silently dropping the file's whole
call graph in the best-effort catch. The unscoped getChunks could also
fan out to same-slug chunks from another source. One normalized
edgeSourceId (sourceId || 'default') now drives both the chunk lookup
and the stamp.
Adversarial review, two findings on the new timeout path:
1. On timeout the finally drained, disconnected, then CLEARED the
hard-deadline timer — removing the only backstop while the abandoned
handler (withTimeout races, it does not cancel) can hold ref'd
sockets/SDK timers that keep Bun's loop alive: 'timed out' printed,
process immortal — the zombie class this branch exists to kill,
resurrected through its own fix. The finally now exits explicitly
after teardown completes on the timeout path.
2. makeContext does DB I/O (resolveSourceId) for EVERY op and sat
outside any bound — a pooler wedge at context build hung reads,
writes, and admin alike. It now shares the same wallclock bound.
- import-code-edges-source-id: scoped import stamps edges + scoped
getCallersOf/getCalleesOf match (verified failing pre-fix), plus the
unscoped-import case asserting 'default' stamping.
- cli-force-exit-teardown-arming: structural pin — the hard-deadline
timer arms inside the finally (teardown entry), never before the op
body; daemon guard, unref, clearTimeout intact.
- embedding-dim-check: recipe order pinned — UPDATE precedes ALTER so
the printed SQL can't drift from docs/embedding-migrations.md again.
The 'Why we don't do this automatically' list still said alter-then-wipe;
reorder to wipe-then-alter and replace the fragile 'step 3' numeric
cross-reference with a name-based one.
1. Backfill: edges written before the stamping fix sit at source_id=NULL
and stay invisible to scoped call-graph queries until repaired. Derive
each edge's source from its own from_chunk's page (pages.source_id is
NOT NULL DEFAULT 'default'). Same SQL verified live on a 2,122-edge
production brain.
2. Indexes: getCalleesOf filters both edge tables on from_symbol_qualified,
which had no index — every callee lookup was a seq scan, amplified
per-BFS-node by the recursive code walk. With NULL edges repaired,
scoped walks actually expand, so the latent cost becomes real.
Mirrored into src/schema.sql; schema-embedded.ts regenerated.
Review catch: the doc fix corrected docs/embedding-migrations.md, but
embeddingMismatchMessage still PRINTED the broken order — ALTER before
UPDATE ... SET embedding = NULL — and linked to the now-contradicting
doc. pgvector refuses to cast existing vectors across dimensions, so
the printed recipe aborted on any brain that has embeddings. Swap the
steps and say why inline.
Review catch: 'sourceId ?? null' fixed the scoped path but left the
unscoped one (reindex --code without --source, importCodeFile callers
without opts.sourceId) stranding edges at NULL while their pages land
under the schema default (pages.source_id DEFAULT 'default') — so
getCallersOf(sym, { sourceId: 'default' }) missed them. Same bug,
other door. Fallback is now 'default'.
With the hard-deadline timer correctly scoped to teardown, a genuinely
wedged read handler (hung pooler connection mid-query) would hang the
CLI forever — the #1633 zombie class the old pre-try timer accidentally
bounded at 10s. Reads now get a generous withTimeout (180s default, far
above any healthy slow-pooler run; --timeout=Ns overrides; exit 124 with
the teardown finally still draining + disconnecting). Writes/admin stay
unbounded: a long import/embed must never be killed by a default.
The Postgres recipe ordered ALTER COLUMN TYPE vector(N) before the
UPDATE that clears stale embeddings. pgvector refuses to cast existing
vectors across dimensions ('expected 1024 dimensions, not 1536'), so
the recipe as written aborts the transaction on any brain that has
embeddings — which is every brain doing this migration. Swap the steps:
NULLs cast fine.
importCodeFile built CodeEdgeInput rows without source_id, so every
edge landed NULL. getCallersOf/getCalleesOf filter
`AND source_id = <scoped>` whenever a worktree pin or --source is in
play — NULL never matches, so scoped call-graph queries silently
returned 0 rows on multi-source brains even though the edges existed
(2,122 edges, 26 targeting the probed symbol, count 0 returned).
One-line fix: carry the sourceId already in scope into the edge input.
Existing NULL rows backfill with:
UPDATE code_edges_symbol e SET source_id = p.source_id
FROM content_chunks c JOIN pages p ON p.id = c.page_id
WHERE c.id = e.from_chunk_id AND e.source_id IS NULL;
(same for code_edges_chunk). Verified: code-callers returns 21 callers
where it returned 0.
The 10s force-exit timer in the shared-op dispatch was armed BEFORE the
try block, so any op whose handler ran past 10s wall-clock was killed
mid-flight with process.exit(0) and zero stdout. On a slow Postgres
pooler (6-10s per fresh connection) a healthy `gbrain search` was
force-exited every time — an empty 'success' indistinguishable from no
results. The v0.42.20.0 exitCode honor can't help: a mid-op kill fires
before any error path sets exitCode.
Move the arming into the finally (teardown entry), matching the
fall-through owner-disconnect site later in main(): the timer still
bounds a hung drain/disconnect (the C13 contract) but can no longer
kill a slow-but-progressing op. Verified on a transaction-pooler
Supabase brain: search went from 0 bytes/exit 0 at 10s to real results
at ~21s.
Bun merges .env files from the process cwd into process.env before any
user code runs. loadConfig() prefers env DATABASE_URL over
~/.gbrain/config.json, so any gbrain invocation from inside a web-app
checkout silently retargets the brain at that app's database — reads go
to the wrong DB and apply-migrations can write gbrain's schema into a
production app database (#427).
effectiveEnvDatabaseUrl() re-parses the .env files Bun auto-loads from
cwd and treats a DATABASE_URL whose value matches one of them as
file-origin: ignored, with a one-time stderr notice. GBRAIN_DATABASE_URL
and genuinely exported DATABASE_URLs are honored unchanged, so the
operator escape hatch and the e2e suite's env-provided URL keep working.
Applied at loadConfig, getDbUrlSource (doctor parity), init
--non-interactive, and migrate --to.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
collectSyncableFiles (the full-sync / dry-run enumerator) reimplemented its
own directory skip list inline (node_modules || ops), bypassing the canonical
pruneDir gate and ignoring .gitignore entirely. On a Laravel/PHP repo this
descended into vendor/ (~50k Composer files), storage/, and public/build/,
trying to import 52k dependency/build files and flooding the index with
library internals (a 35-min sync that never finished, killed by the watchdog
at 3%).
- collectSyncableFiles now enumerates via `git ls-files --cached --others
--exclude-standard` when dir is a git work tree, so the walk honors
.gitignore (tracked + untracked-not-ignored). Falls back to the FS walk for
non-git dirs. EroLab: 52164 -> 1028 files.
- The FS fallback now prunes through the canonical pruneDir() instead of a
drifted inline list, so the two skip lists can't diverge again.
- PRUNE_DIR_NAMES gains vendor/dist/build (dependency + build-output trees).
Addresses #1483 (.gbrainignore), #1159 (--respect-gitignore), and the
maintainer's #1942 vendor/dist/build prune. Walker regression suites
(sync-walker-symlink, brain-writer-walk-prune, sync, sync-walker-submodule)
green: 90 pass.
* fix(jobs): reap stale dead-holder cycle/sync locks (#1972)
A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row
until something contended for it — reclaim was on-contention only. Add a
host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder
PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle*
namespaces only (never elections/supervisor/reindex), with a snapshot-matched
delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse.
Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now
auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now
back inspectLock + listStaleLocks (killed the triplication).
* fix(db): bound pool disconnect so teardown can't eat CLI output (#1972)
pool.end() against PgBouncer transaction-mode never drained, so disconnect
blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write,
truncating stdout (e.g. #1959's relational query returned empty). Add a
gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against
a hard timer, so teardown is bounded regardless of what postgres.js does and is
testable. connection-manager ends its direct + read pools concurrently so the
per-pool bounds don't stack. PGLite disconnect is unaffected.
* fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972)
v0.42.29 made only the embed phase honor the abort signal; a 24h pull still
showed force-evicts from a long non-embed phase ignoring it. Thread the signal
into every cycle-reachable long loop: extract (extractForSlugs + the full-walk
extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop +
embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket
loop. Add a terminal abort check so an aborted cycle never stamps
last_full_cycle_at as a completed run (Codex #9). lint now yields + checks
abort every 200 pages (it's synchronous; the yield is what lets the signal
land). New phase-duration force-evict attribution log names any phase that
crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start.
* chore: bump version and changelog (v0.42.38.0)
#1972 — stale-lock reaper, bounded pool disconnect, and complete
cooperative-abort coverage across cycle phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage
document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY),
db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/
extract_facts/consolidate/lint + terminal guard) entries to current truth.
* test(isolation): fix shard-order flakes exposed by #1972's new test files
Adding 3 new test files reshuffled the hash-based shards, exposing two
pre-existing test-isolation bugs:
- cycle-consolidate.test.ts assumed the global legacy-embedding preload's
1536-d gateway config still held at initSchema, but a co-sharded test that
calls resetGateway() in teardown nulls it, so initSchema fell back to the
1280-d default and built a halfvec(1280) facts column its 1536-d fixtures
can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern
legacy-embedding-preload.ts documents for 1536-d fixture tests).
- db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated
process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation
rule R1. Convert to withEnv().
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dimsProviderOptions() threads input_type ('query' | 'document') into
providerOptions.openaiCompatible for asymmetric models (ZE zembed-1,
Voyage v3+), but the AI SDK's openai-compatible adapter validates
providerOptions against a fixed schema and silently drops the field
before building the HTTP body. Every embedQuery() was therefore encoded
document-side: the ZE shim's hard default fired ('document'), Voyage and
local openai-compat servers got no input_type at all, and asymmetric
retrieval silently collapsed toward surface-token overlap — while the
providerOptions-level contract test stayed green.
Fix: an AsyncLocalStorage (same pattern as __budgetStore) populated in
embedSubBatch() only when providerOptions actually threads an
input_type, read at body-rewrite time by the fetch shims:
- zeroEntropyCompatFetch: recovers the threaded value; document default
preserved for ingest paths.
- voyageCompatFetch: opt-in like the dims.ts Voyage branch — inject only
when threaded; the field stays off the wire otherwise.
- NEW openAICompatAsymmetricFetch: fallthrough default for every other
openai-compatible recipe (llama-server, litellm, ollama, ...) — the
canonical local/proxy paths for asymmetric models. Strict pass-through
when nothing was threaded, so symmetric deployments see zero wire
change; recipes with their own compat fetch (azure) keep it via the
compat.fetch ?? precedence.
KNOBS_HASH_VERSION bumped 10→11: cached query_cache rows were keyed on
document-side query vectors; pre-fix rows must not be served to post-fix
lookups (same convention as the v=3 embedding-provider bump). One-time
global cold-miss on upgrade; refills within cache.ttl_seconds.
Tests: test/embed-input-type-wire.test.ts runs the REAL SDK transport
with a mocked global fetch and asserts on the outbound body — the only
layer where this regression is observable. Covers ZE hosted, llama-server,
litellm, ollama (query + document sides) and pins the pass-through for
non-asymmetric models and Voyage's opt-in shape. 4 of the original 7
assertions fail on master, proving the pin. One structural pin in
test/ai/zeroentropy-compat-fetch.test.ts updated to the new line shape
(same semantic); KEY_FILES.md gateway.ts entry updated to the new truth.
Supersedes #1400 (closed unmerged) — same ALS mechanism, extended to
Voyage + all openai-compatible recipes. Credit to @billy-armstrong for
the original diagnosis.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
collectSyncableFiles (first-sync walker) and the incremental PRUNE_DIR_NAMES
set skipped node_modules but not Python venv/. On a Python repo the walker
descended into venv/ (thousands of files); the resulting slug collisions
crashed putPage's INSERT ... ON CONFLICT ... RETURNING with
"undefined is not an object (evaluating 'row.deleted_at')".
Add `venv` alongside node_modules in both the import.ts inline skip and
PRUNE_DIR_NAMES. venv is the Python equivalent of node_modules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a client omits `scope` on /authorize, the authorize() grant computed
`(params.scopes || []).filter(...)` → the empty set. That empty grant was
written to oauth_codes and propagated into the access AND refresh tokens, so
every request failed `insufficient_scope` even though the client was
registered with e.g. `read write`. Because refresh inherits the stored grant,
it never self-healed — reconnecting just minted another empty-scoped token.
Some MCP connectors (observed with Claude Desktop) omit `scope` on /authorize,
so they hit this on every connection.
Fix: when no scope is requested, default to the client's full registered scope
(RFC 6749 §3.3 permits a server default). This mirrors exchangeClientCredentials,
which already does `requestedScope ? ... : allowedScopes`. The result is still
clamped to the allowed set, so an explicit over-broad request cannot escalate.
Adds test/oauth-authorize-scope-default.test.ts covering: omitted/empty →
inherits full grant; explicit subset honored; clamp preserved (over-broad and
disallowed-only requests cannot escalate or trigger inheritance).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): scope cross-source reads to the caller grant; close get_page exact-path leak
One shared resolveRequestedScope() routes every source-scoped read op
(query, code_callers/callees, search_by_image, code_blast/flow, get_page)
through a single fail-closed trust+grant ladder: a remote caller's __all__
collapses to its granted sources (never the whole brain) and an explicit
out-of-grant source_id is rejected. get_page's exact-match path now honors a
federated grant via getPage(sourceIds[]) in both engines. Legacy bearer tokens
carry their stored permissions.source_id grant (bounded, never widened). Also
retries getConfig on transient connection loss.
Closes#1924, #1371, #1393, #1336, #1603.
* fix(ingest): non-string frontmatter no longer aborts lint/sync; embed/hook/catalog papercuts
Parser coerces a non-string title to a string and falls back to inference for
slug/type (never fabricating a "123" slug), with a lint NON_STRING_FIELD finding
surfacing the malformed frontmatter; a defensive guard in content-sanity stops a
non-string title from crashing the whole lint/sync run brain-wide. Plus: embed
--catch-up no longer arms the overflowed 32-bit budget timer (and surfaces
unembeddable chunks); the frontmatter pre-commit hook ships a correct .md/.mdx
regex; and the skill catalog parses YAML block-scalar descriptions.
Closes#1883, #1658, #1556, #1948, #1946, #1840, #1711.
* v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add NON_STRING_FIELD frontmatter validation class to docs for v0.42.37.0
The v0.42.37.0 non-string-frontmatter fix added an eighth validation
class (NON_STRING_FIELD / lint code frontmatter-non-string-field). Update
the two current-state docs that enumerate the validation classes:
- skills/frontmatter-guard/SKILL.md (seven->eight + table row)
- docs/integrations/pre-commit.md (seven->eight + table row)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970)
When a source's history is rewritten (force-push, master→main consolidation,
squash), the recorded last_commit can fall outside HEAD's history. The old
guard sent both "object missing" and "not an ancestor" to performFullSync — a
full repo re-walk that never advances the bookmark under a cron timeout on a
large cross-region brain, so the source goes silently stale.
Fix: only a truly-absent object forces a full reconcile. A present-but-non-
ancestor bookmark is diffed tree-to-tree directly (git diff A..B needs no
ancestry), importing only the real delta. Adds: oversized-diff fallback to full
reconcile (F-B); performFullSync now purges deleted files, gated to file-backed
pages by source_path so manual/put_page and metafile pages are spared (F-A);
rename-to-unsyncable deletes the stale old page (F-C). 7 new PGLite e2e tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(architecture): record #1970 sync bookmark recovery + full-sync delete reconcile in KEY_FILES
Update the sync.ts entry to current truth: entry-time bookmark-reachability
guard (gc'd anchor → full reconcile; non-ancestor-but-present → direct
tree-to-tree diff), oversized-diff fallback, performFullSync now authoritative
for deletes (file-backed pages by source_path), and rename-to-unsyncable delete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(search): deterministic relational-query parser
Pure, ReDoS-bounded parser that detects relationship queries ("who invested
in X", "who at X works on Y", "who introduced me to X", "what connects A and
B") and maps them to typed edges. Schema-pack-extensible vocab with subset
validation against the link types ingest produces, so query-side and
ingest-side relation vocabularies can't drift. No-match / pronoun-seed /
adjacency guards keep it precision-first (a candidate only; the arm fires
only when a real seed also resolves).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(engine): relationalFanout typed-edge fan-out (both engines)
Generalizes traversePaths to a SEED ARRAY, aggregating to ranked NODES
(shortest hop, edge-richness count, via-link-types, shortest connecting
path, canonical chunk id) instead of edges. Within-source traversal (never
crosses a source boundary even across a federated scope), link_source=
'mentions' excluded by default, deleted_at filtered at seed + every neighbor
+ every node, bounded depth (<=3) + candidate cap. Adds RelationalFanoutRow
/ RelationalFanoutOpts + the relational SearchResult/SearchOpts fields to
types.
Lands in lockstep in postgres + pglite engines, pinned by a DATABASE_URL-
gated parity block in engine-parity.test.ts; a PGLite unit test exercises
the SQL (typed-edge filter, mentions exclusion, deleted_at, canonical chunk,
multi-seed connects, determinism) in default CI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(search): relational recall arm + federation key hardening
Wires edge-derived candidates into bare hybridSearch as a FOURTH RRF arm
(relational-recall.ts): parse the original query -> scope-aware,
confidence-gated seed resolution (drops fallback_slugify; never traverses
from a guess) -> relationalFanout -> batch-hydrate, reinforcing each page's
REAL canonical chunk (page-level key for chunkless entity pages) ->
--explain attribution + fail-open audit row. Text-only (no-op in image
mode); pure no-op for non-relational queries; rides every downstream stage
(cosine, post-fusion boosts, dedup, reranker, autocut, token budget).
Mode wiring: relationalRetrieval + relational_retrieval_depth knobs
(conservative off; balanced/tokenmax on; depth 2), per-call thread-through
in both bare + cached paths, KNOBS_HASH_VERSION 9->10 (rel=/reld=), config
keys, modes-dashboard descriptions, and a `relational` param on the query op.
Federation hardening (structural, engine-wide): the RRF/dedup key now
carries source_id via a shared rrfKey() (fixes a latent cross-source
collapse where same-slug pages in different sources merged), and the query
cache scopes by a canonical source-set key (cacheScopeKey) so a federated
read can't be served a single-source row.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(eval): relational benchmark + recall@k harness metrics
NamedThingBench harness gains recall@k / recall@10 (the relational headline
metric) on QuestionResult + FamilyReport, plus typed seed/linkTypes/kind on
NamedThingQuestion so the graph-relationship family is machine-checkable.
Adds the relational benchmark corpus (test/fixtures/retrieval-quality/
relational/): a small entity graph whose answers are LEXICALLY UNRECOVERABLE
— every page body is generic and never names the entity it relates to, so
only the typed edge connects query to answer. corpus.ts is the canonical
source for both the seed loader and the 38-question gold set; relational.jsonl
is generated from it (a drift test pins them equal).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(eval): relational A/B proof + arm fires on all retrieval paths
Fixes the integration bug the eval caught: the relational arm was only
injected on the main RRF path, so it silently did nothing whenever vector
was unavailable — no embedding provider configured (the default in many
deployments) OR embed failure. The arm is now built once and fused via RRF
on ALL THREE hybridSearch return paths (no-embedding-provider, embed-failed
keyword fallback, main path). Without this it would have been dead in
exactly the setups that most need it.
Adds `gbrain eval retrieval-quality --ab-relational`: runs the gold set
twice (arm off vs on) in a fixed mode and reports the graph-relationship
recall@10 lift + Hit@3 + latency add. The CI A/B test pins the headline
result — recall@10 jumps from <25% (lexically unrecoverable) to >75% with
the arm on — and a non-relational query returns identical results arm-on vs
off (the no-op / no-regression gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.34.0)
Relational retrieval feature: typed-edge recall arm + federation key
hardening. Also updates the KNOBS_HASH_VERSION 9→10 assertions across the
remaining search test files (the bump invalidates relational-off cache rows).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document typed-edge relational retrieval (v0.42.34.0)
CLAUDE.md Search Mode: add relationalRetrieval to the knob table, the
knobs_hash v=9→10 note, and a relational-retrieval summary. RETRIEVAL.md:
add the relational recall arm to the pipeline diagram. Regenerate llms
bundles (build:llms).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(test): size relational fixtures to the actual embedding-column dim
CI shard runs with the ZeroEntropy gateway default (1280-d), but the
relational test fixtures hardcoded 1536-d embeddings, so chunk inserts were
rejected with "expected 1280 dimensions, not 1536" (CheckExpectedDim) — the
`test (6)` shard + `test-status` failures. The column width tracks the
configured gateway default and can shift with shard order, so fixtures now
probe `content_chunks.embedding`'s actual `atttypmod` after initSchema and
size embeddings to it (the pglite-engine.test.ts pattern), via a shared
`probeEmbeddingDim` helper. Verified passing at a forced 1280-d column.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Curated, high-bar diary of the valuable ideas surfaced by the community-PR
wave, grouped into 10 themes with contributor credit and OPEN/CLOSED/HELD
status, so good thinking survives PR closure. Captured during a full triage
+ hygiene pass over the open-PR backlog. Pure docs; no code impact.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (#1881)
recloneIfMissing deleted local_path whenever a source had a remote_url and a
non-healthy on-disk state, with no check that gbrain actually created the clone.
A source whose local_path was a user's live working tree (remote_url set, no
gbrain-created clone) could have its directory removed and re-cloned over.
- isOwnedClone(): ownership, not path-containment. True only for a config
.managed_clone marker (written by addSource --url) or exact normalized-path
equality with defaultCloneDir(id) (back-compat for pre-marker default clones).
- recloneIfMissing: ownership guard aborts before ANY filesystem op; EXDEV-safe
sibling-temp clone + atomic swap (old aside -> new in -> drop old) with
best-effort restore + a message naming where the original is preserved;
symlink-leaf reject before the destructive rename.
- sync.ts validate_repo_state guards reclone on isOwnedClone (no per-sync warn).
- sources restore degrades to a warning for an unowned source instead of the
misleading "missing clone, try sync" hint.
Tests: #1881 regression (tree survives), isOwnedClone matrix, symlink reject,
EXDEV swap residue-free, --clone-dir owned-via-marker, restore CV3, unownedHint
healthy/degraded, sync-level refusal.
* chore: bump version and changelog (v0.42.33.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document sources-ops reclone-ownership invariant for v0.42.33.0 (#1881)
Add the missing src/core/sources-ops.ts entry to KEY_FILES.md capturing the
must-never-violate reclone-ownership guarantee: gbrain only deletes/re-clones a
clone it created (isOwnedClone), never a user working tree. Covers managed_clone
marker, defaultCloneDir back-compat, EXDEV-safe swap, TOCTOU + symlink-leaf
guards, unmanaged_path SourceOpError, and the read-only sources restore path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(import): coerce non-string frontmatter title/slug/type (#1939)
YAML `title: 2024-06-01` parses to a Date and `title: 1458` to a number;
the old `(frontmatter.X as string)` cast was a compile-time lie, so
downstream `.toLowerCase()` threw and (via the importer failure gate)
could wedge sync indefinitely. parseMarkdown now coerces via
coerceFrontmatterString (Date -> UTC ISO date, deterministic), and the
pure assessContentSanity self-protects against a non-string title.
* feat(sync): bounded auto-skip failure ledger; poison file can't wedge indexing (#1939)
New src/core/sync-failure-ledger.ts owns the failure store + a crash-safe,
multi-source, concurrent bounded auto-skip valve. A file that fails N
consecutive syncs (GBRAIN_SYNC_AUTOSKIP_AFTER, default 3) auto-skips so it
can't freeze all indexing forever, while fresh failures still fail-closed
and a `<head>` history-rewrite sentinel hard-blocks even with --skip-failed.
- (source_id, path) keying — failures never merge across sources
- success clears a path so attempts are truly consecutive
- advance-before-ack ordering (a crash can't mark a file skipped while wedged)
- shared applySyncFailureGate used by BOTH the incremental and full-sync gates
- legacy-row normalization + duplicate collapse on load
- cross-process lock + atomic temp-rename, age-based stale-lock break
sync.ts re-exports the ledger for existing callers; import.ts records
source-scoped and defers the bookmark to the gate under managedBookmark.
* fix(doctor): sync_failures severity via one shared decision on both surfaces (#1939)
Local buildChecks and remote doctorReportRemote now both route through
decideSyncFailureSeverity, so a stuck bookmark escalates WARN -> FAIL
consistently (oldest-open age > fail cadence, or large unresolved count),
auto-skipped pages stay visible (WARN, not hidden), and the
acknowledged/acknowledged_at field-split that caused drift is gone. The
remote surface stays subprocess-free (file read + Date.parse only).
* chore(test): add trailing newline to e5-lease-cap-ab baseline fixture
* fix(sync): address adversarial review findings on the failure ledger (#1939)
- #1: a parse-failed file that is later deleted/renamed-away no longer leaves
a permanent open ledger row. Removed paths (filtered.deleted, renamed-from,
and the "gone from disk" forward-delete skip branch) are treated as resolved
so the ledger self-heals instead of aging doctor to a stuck FAIL.
- #3: decideSyncFailureSeverity escalates to FAIL on OPEN (blocking) failures
only — auto_skipped rows already advanced the bookmark, so they stay
WARN-visible regardless of count, matching the state-machine contract.
* chore: bump version and changelog (v0.42.30.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document sync-failure ledger + auto-skip valve for v0.42.30.0
KEY_FILES.md: new src/core/sync-failure-ledger.ts entry (bounded auto-skip
state machine, decideGateAction/decideSyncFailureSeverity/applySyncFailureGate,
GBRAIN_SYNC_AUTOSKIP_AFTER); update sync.ts (failure store moved to ledger,
re-exported), doctor.ts (sync_failures severity via shared rule on both
surfaces), markdown.ts (coerceFrontmatterString), import.ts (managedBookmark).
live-sync.md: poison-file auto-skip tricky-spot. Regenerated llms-full.txt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: re-bump to v0.42.31.0 (queue collision on 0.42.30.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: re-bump to v0.42.32.0 (queue collision)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Garry Tan