mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
06f58c2b32398b238bb5b153293f6887bf55a14f
303
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 spiky02plateauandGitHub
|
06f58c2b32 |
feat(gateway): config-driven provider_chat_options passthrough (fixes #2577) (#2857)
Add provider_chat_options alongside provider_base_urls and thread it through the gateway config path into chat(). The chat request now deep-merges provider-scoped options and model-scoped overrides into providerOptions keyed by recipe id, preserving existing gateway-built options such as Anthropic cacheControl and leaving absent-config behavior unchanged. This lets operators disable thinking for small-budget hybrid-reasoning utility calls without hardcoding that behavior for every use of those models. |
||
|
|
6abab9d584 |
fix(facts): durable facts-absorb jobs for one-shot CLI processes + source-scoped fence paths (#2104)
* fix(facts): durable facts-absorb jobs for one-shot CLI processes Every gbrain capture/put from a short-lived CLI enqueued the facts:absorb chat into the in-process FactsQueue, then the exit teardown drained for 1-2s and aborted the in-flight call — logging 'pipeline_error: [chat(...)] The operation was aborted.' on every eligible CLI page write and never extracting facts. cli.ts now marks one-shot processes (everything except serve/jobs/ autopilot); runFactsBackstop's queue mode submits a durable facts-absorb minion job for the long-lived jobs worker instead, with content-hash idempotency and fallback to the in-process queue if submission fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(facts): fence-write resolves source-scoped page path writeFactsToFence joined local_path + slug directly, writing main-source fences to the repo ROOT (the default source's tree) and polluting ~/brain with stray root-level fence files. Route through resolvePageFilePath — the same helper the put_page write-through and dream-cycle reverse-render use — so non-default sources fence into .sources/<id>/<slug>.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Ragnar Åström <reghar@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ziyang GuoandGitHub
|
e0ca74200a | fix(timeline): expose date window filters (#2694) | ||
|
|
9315fd0746 |
fix(conversation-parser): read raw_transcript sidecar + parse plain Speaker A/B lines (#1898)
The parser/doctor/extractor read the polished page body (compiled_truth + timeline) instead of the raw turn-by-turn transcript that meeting pages store in a `raw_transcript` frontmatter sidecar, and the brain's actual raw format (`Speaker A: ...` / `Speaker B: ...`) had no built-in pattern. Result: scan returned no_match / 0 messages and conversation-fact extraction produced 0 segments / 0 facts. - new readConversationBodyForParsing(): prefer the raw_transcript sidecar when present, fall back to compiled_truth + timeline (src/core/conversation-parser/body.ts) - wire it into conversation-parser scan, doctor coverage check, and extract-conversation-facts (drops the old readPageBody helper) - add a built-in `speaker-letter-no-time` pattern for plain `Speaker A:` lines Verified: scan now returns phase=regex_match (speaker-letter-no-time), and the Ben page extracts 61 facts / 7 segments. Tests added; suite green. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
d33aee843b |
query: filter since/until on effective date, not updated_at (#1706)
since/until range filters were applied against updated_at, so edited-but-old entries leaked into time-bounded queries. Filter on the effective date instead. Closes #1520 Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
maxpetrusenkoagentandGitHub
|
323d7d6336 | test(ai): pin gateway tool schema conversion (#2063) | ||
|
|
9ceca6063b |
book-mirror: emit HTML <table valign=top> instead of markdown pipe tables (#2270)
Markdown pipe tables have no vertical-align control, so every renderer except GitHub middle-aligns rows — unreadable when the two columns differ in length. Switch the per-chapter prompt + frontmatter tag to the HTML <table> form with valign=top on every cell (matches the 20 hand-built mirror pages that already render correctly). Co-authored-by: garrytan-agents <agents@garrytan.dev> |
||
|
|
e538051401 |
feat(extract): recognize inline [Source: ..., YYYY-MM-DD] citations as timeline entries (#2524)
* feat(extract): recognize inline [Source: ..., YYYY-MM-DD] citations as timeline entries gbrain's own quality conventions (skills/conventions/quality.md) require a dated [Source: ..., YYYY-MM-DD] citation on every brain write, so curated pages are full of dated evidence — but extractTimelineFromContent only recognized the timeline-bullet and date-header formats. A page whose dates all live in citations scored zero timeline coverage in brain_score, and doctor pointed users at a formatting convention their own citations already satisfied in spirit. Format 3 files one entry per citation: date and source from the marker, summary from the annotated line with citation markers stripped. Lines already captured by Format 1 are skipped so a timeline bullet carrying its own citation is not double-filed. Bare citations with no surrounding text are ignored. Idempotency is unchanged: persistence already dedupes at the DB layer. * fix(extract): Format 3 citations also in parseTimelineEntries (db-source + ingest path) The first commit only taught extractTimelineFromContent (fs-source) the citation format; the db-source extract and the ingest path parse through parseTimelineEntries in core/link-extraction.ts, which still could not see citations. Same rules as the fs parser: bullet-captured lines skipped, bare citations ignored, invalid calendar dates rejected; the citation source is preserved in the entry detail. --------- Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com> |
||
|
|
7202ebf3da |
fix(takes): bootstrap runs progress through the corpus instead of rescanning the newest slice (#2638)
extractTakesFromPages selected pages by updated_at DESC + LIMIT with no exclusion of pages that already hold takes. The CLI clamps --max-pages to 1000, so on a corpus larger than one run every re-run rescanned the same most-recent 1000: the older tail could never be bootstrapped, and each rescan re-spent Haiku budget producing upsert-identical rows. Seen live on a 2,311-eligible-page brain — a second run would have covered 0 new pages. Covered pages are now skipped by default (NOT EXISTS on takes.page_id), so repeat runs sweep a large corpus in slices; --include-covered restores the full rescan for refresh use. Usage text documents both plus the 1000 clamp. Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
f981f70a2f |
fix(extract): deterministic atom slug — stop cross-day + trailing-dash duplicate atoms (#2482)
extract_atoms minted duplicate atom pages two ways: A. Trailing-dash twins. The local slugger truncated the title at 60 chars with no re-strip, so a cut landing on a hyphen left a trailing dash (`…would-`). The FS-import normalizer (slugifySegment) strips it (`…would`), so the same atom persisted under two slugs and the dedup-by-slug check never collapsed them. B. Cross-day re-mint. The slug used the run date (todayDate()), while the idempotency guard keys on the whole-file source_hash. Append-only sources (chat/transcript exports) grow daily, so the file hash changes, the guard never matches, the source is re-extracted, and each re-mint lands under a new date prefix → a new slug → no upsert → a duplicate. Fix: make the atom slug a deterministic function of stable inputs — `atoms/<source-date>/<stem>-<title-hash>`: - source date is parsed from the source ref (transcript filename / page slug), not the run date, so re-extraction converges on the same slug and putPage upserts instead of duplicating; - the 6-char title hash keeps two atoms whose titles share the first 60 chars on distinct slugs (no silent clobber of a different atom); - the stem routes through the canonical slugifySegment and re-strips a trailing dash, so the two write paths can no longer disagree. The whole-file source_hash batch check is retained only as a cost fast-path (skip re-running the model on an unchanged source); correctness no longer depends on it. Adds a hermetic PGLite regression test (no DATABASE_URL) asserting the source-dated prefix, title-hash suffix, trailing-dash strip, and upsert on re-extraction of a grown append-only transcript. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
vinsewandGitHub
|
34c0ff0c56 |
fix(autopilot): verify lock holder process before exiting (#477)
A lock file can outlive its autopilot process after a crash or forced termination. The previous mtime-only check treated that file as proof that another instance was running, so supervisor restarts could exit repeatedly until the file aged out. Read the holder PID and probe it with signal 0. Keep the lock whenever that process is alive; take over only dead, malformed, empty, or self-owned locks. EPERM remains conservative and counts as alive, preventing two autopilot instances from running concurrently. Add hermetic tests for missing, live, dead, malformed, and empty lock states. |
||
|
|
e1e1f3bac2 |
feat(extract_atoms): honor pack manifest extractable flag in page discovery (#2615)
Atom-extraction page discovery hardcoded EXTRACTABLE_PAGE_TYPES and ignored the active pack's `extractable: true` flags, so a type declared extractable in the manifest (e.g. `note`) never actually extracted. Closes the D2 TODO the code already flagged (extract-atoms.ts: 'future pack-aware refactor ... pull from the active pack manifest'). Resolve the allowlist as: legacy hardcoded floor UNION the pack's extractable types, MINUS synthesis outputs (atom, concept — extracting from these would loop, since concepts are synthesized from atoms). Mirrors facts/eligibility.ts, which excludes concept the same way. Back-compat: gbrain-base brains keep every legacy target via the union; fail-soft falls back to the legacy floor if the pack can't load. - pure unionExtractableTypes() policy, unit-tested - discoverExtractablePages + countExtractAtomsBacklog resolve from the pack - page-discovery fixture updated (note now extracts; concept stays excluded) Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
a12db46350 |
schema: resolve all bundled packs in schema use, not just gbrain-base (#1707)
`schema use` hardcoded `gbrain-base` (and a fixed bundled list), so other bundled packs could not be selected by name. Use the shared BUNDLED_PACK_NAMES set and resolve `<name>.yaml` generically so every bundled pack activates. Closes #1574 Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
|
9f313db374 |
fix(pricing): add Sonnet 5 and Fable 5 to the canonical chat-pricing table (#2799)
claude-sonnet-5 and claude-fable-5 are GA Anthropic models, but neither was in CANONICAL_PRICING. A brain routing a tier to them (e.g. models.tier.reasoning = anthropic:claude-sonnet-5) ran with cost telemetry blind on that tier: canonicalLookup missed, the budget meter logged BUDGET_METER_NO_PRICING and disabled the gate, and cost views under-reported spend. - model-pricing.ts: anthropic:claude-sonnet-5 at $3/$15 and anthropic:claude-fable-5 at $10/$50. Sonnet 5's launch intro discount ($2/$10 through 2026-08-31) is deliberately not modeled — the table carries standard rates so estimates stay conservative and the entry needs no time-bombed edit when the promo lapses. - takes-quality-eval/pricing.ts: claude-sonnet-5 added to the curated SUPPORTED_MODELS allowlist (a likely judge override). Fable 5 stays out — priced for warn-only consumers, not a budgeted-eval panel model. - model-pricing.test.ts: pin tests for both rows, matching the existing Opus 4.8/4.7 pattern. Drift guards iterate the table; no changes. All derived views (ANTHROPIC_PRICING bare view -> budget-tracker, batch-projection, budget-meter) pick the rows up automatically. Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
bb417051fe |
fix(search): fold hard-exclude/include prefixes into knobs_hash — stop cross-process cache leak of excluded slugs (#2825) (#2885)
resolveHardExcludes() only ran at DB-query build time (cache miss), so query_cache rows written by a process without GBRAIN_SEARCH_EXCLUDE could be served to a process with it (and vice versa), leaking excluded slugs. hybridSearchCached now resolves the effective hard-exclude list exactly as the engines' query-build path does and folds it (sorted, append-only hx= part) into knobsHash via a new KnobsHashContext.hardExcludes field. KNOBS_HASH_VERSION 11 -> 12: one-time global cache cold-miss on upgrade, refills within cache.ttl_seconds. Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
+5
|
86962b242b |
fix(gateway): consolidate tool-loop resume + provider fixes (fix-wave A) (#2820)
Collector branch superseding the gateway tool-loop duplicate cluster and adjacent provider fixes. Re-implemented from the best of each PR (deduped by content, not file-overlap); every fix carries test coverage. Gateway tool-loop resume (supersedes #1934 #2062 #2065 #2112 #2274 #2487 #2336 #2257 #2499 #2491, test #2063): - toolLoop now persists the tool-result user turn per round (onToolResultTurn), so a resumed subagent job reloads a balanced transcript instead of dangling assistant tool-calls that non-Anthropic providers reject with AI_MissingToolResultsError. - runSubagentViaGateway reconciles an already-corrupted transcript on resume: it heals every dangling assistant tool-call turn (not just the tail) from the settled subagent_tool_executions rows, re-dispatching idempotent-pending tools and throwing on non-idempotent, mirroring the legacy Anthropic path. Terminal early-return for a transcript that already reached end_turn. - repairToolPairing() is a last-resort normalization at the chat() boundary (from #2336): back-fills error stubs for any assistant tool-call still unanswered (partial turns, provider-duplicated/dropped IDs on local models, length-truncated batches). No-op on balanced input. - toModelMessages is Date-safe (Postgres timestamptz -> ISO via a JSON round trip at the SDK boundary, never a ::jsonb cast; degrades bigint/circular to a string instead of throwing) and drops non-string text blocks reasoning models emit that AI SDK v6 rejects (#2488). Adjacent provider fixes: - Model-aware default max output tokens: thinking-by-default Claude 5 models get headroom (gateway 32000, think 16000) while everything else stays 4096/4000, so DeepSeek/OpenAI subagents don't exceed provider caps (#2614 #2806). - DeepSeek: promote reasoning_content into content when content is empty, via a fail-open recipe fetch shim (#2617). - OpenRouter: map openrouter_api_key (config + env) into OPENROUTER_API_KEY through buildGatewayConfig; register agent.use_gateway_loop, zeroentropy and openrouter keys in KNOWN_CONFIG_KEYS so `config set` accepts them (#2572, config key from #2112). Preserves JSONB (no JSON.stringify into ::jsonb), engine parity, source isolation, and trust-boundary invariants. Verified with the gbrain-pr-test-env consumer matrix (clawlancer/Postgres, gstack + hivemindos/PGLite) baseline-FAIL -> candidate-PASS on the same repro, plus bun run verify (31/31) and 439 targeted unit + e2e tests. Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: thomaskong119 <thomaskong119@hotmail.com> Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> Co-authored-by: brettdavies <brettdavies@users.noreply.github.com> Co-authored-by: ivandebot <187176982+ivandebot@users.noreply.github.com> Co-authored-by: Rafael Reis <rafael.reis@contabilizei.com.br> Co-authored-by: fbal23 <fbal.public@gmail.com> Co-authored-by: javieraldape <javieraldape@users.noreply.github.com> Co-authored-by: David Carolan <david@joyrestart.com> Co-authored-by: Masashi-Ono0611 <masashi.ono.0611@gmail.com> Co-authored-by: spiky02plateau <155588579+spiky02plateau@users.noreply.github.com> Co-authored-by: psam-717 <mphilannorbah@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ziyang GuoandGitHub
|
ec3910afc4 | fix(import): route image pages by source (#2718) | ||
|
Ziyang GuoandGitHub
|
7a275bf0b5 | fix(takes): scope page lookup by source (#2698) | ||
|
|
285cf39f9a |
fix(config): register Life Chronicle keys so the documented enable command works (#2632)
* fix(config): register Life Chronicle keys so the documented enable command works The v0.42.56.0 release notes say `gbrain config set auto_chronicle true`, but the key was never added to KNOWN_CONFIG_KEYS — the documented command fails with 'Unknown config key' and the operator has to discover --force by reading source. Registers 'auto_chronicle' plus the 'chronicle.' prefix (chronicle.tz and future knobs). Same registration class as the v0.42.42.0 spend-controls fix. Regression test pins both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk * fix(config): register takes.bootstrap_enabled too — same unregistered-key class Hit while enabling the takes bootstrap on a live brain: the onboard remediation's documented enable key fails 'Unknown config key' exactly like auto_chronicle did. Registered + pinned by the same regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk --------- Co-authored-by: Paolo Belcastro <p3ob7o@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Matt GunninandGitHub
|
836d83012d | fix(orphans): exclude generated corpus roots (#2068) | ||
|
vinsewandGitHub
|
d0447a597b |
fix(files): normalize bigint sizes before JSON serialization (#472)
Postgres returns BIGINT file sizes as native BigInt values. Returning those values directly from file_list makes JSON serialization fail, and using them in CLI arithmetic can also throw. Convert size_bytes to Number at the operation boundary and in the CLI display path. File sizes remain exact well beyond any practical attachment size. Add a unit regression that exercises a native BigInt row and proves the operation result is JSON-serializable, plus a real-Postgres E2E assertion for the file_list response. |
||
|
|
659b6e9b4d |
fix(import): skip marked.lexer on fence-less pages to avoid bulk-import OOM (#2437) (#2440)
* fix(import): skip marked.lexer on fence-less pages to avoid bulk-import OOM (#2437) extractFencedChunks() ran marked.lexer() on every page body. The lexer allocates transient memory proportional to page size even when there is no code fence to extract — a ~2MB table/doc page spikes ~110MB of heap to produce zero fenced chunks. During bulk import these per-page spikes stack on accumulated chunk/embedding memory and can OOM the worker; the existing try/catch cannot rescue an OOM (process death, not a throw). On a representative brain ~99% of importable pages have no fence, so the lexer pass is pure wasted work there. Add a fast-path that returns early when the body contains no fence marker. Matches both ``` and ~~~ so tilde fenced code still extracts. Scope: this removes the fence-less transient-allocation surface (the observed incident). It does not make marked.lexer safe for pages that DO contain a fence; an input-size/nesting cap is a sensible follow-up. Tests: add two regression cases — tilde-fenced code still extracts, and a large fence-less table page imports with zero fenced chunks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(import): match marked's \r normalization in the fence fast-path (#2437) Self-review follow-up. marked normalizes `\r\n|\r → \n` before lexing, but the no-fence fast-path probed the raw body with `(^|\n)`. A CR-only (classic-Mac) line-ended page with a real fenced block would be skipped by the guard while marked would have extracted it — a lost fenced_code chunk. Widen the line-start class to `(^|[\r\n])` so the probe agrees with marked. CRLF was already covered. Add a regression test (CR-only fenced page still extracts); it fails on the old `(^|\n)` regex and passes on the fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
f15163f727 |
fix(sync): normalize path separators + add mass-delete safety valve to full-sync reconcile (#2828) (#2836)
On a Windows checkout, path.relative yields backslash-separated paths while a page's stored source_path can hold forward slashes (e.g. git-derived). The full-sync reconcile compared the two without normalizing separators, so every file-backed page looked stale and the reconcile deleted the entire source. - Normalize separators on BOTH sides of the membership test (shared .replace(/\/g, '/')), so pages written with either separator match on any OS. - Add a mass-delete safety valve: when a reconcile would sweep > 50% of the file-backed pages a strategy manages on a source with > 20 of them, skip the delete and surface a loud warning instead of silently wiping the brain. GBRAIN_ALLOW_MASS_RECONCILE=1 restores the old behavior. - Factor the decision into pure, exported helpers (planReconcileDeletes, massReconcileAllowed) and unit-test the separator matching, the valve threshold, and the env override without a live engine. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
bb3376e3b0 |
fix(security): prevent leaking admin bootstrap token to non-TTY stdout (#2625)
* fix(security): #2624 don't print admin bootstrap token on non-TTY (log-leak) serve --http printed the generated admin token in the startup banner unconditionally. In containerized deploys stderr ships to centralized log storage, turning the token into a standing secret in logs. Fail-safe default: the generated token now prints only when stderr is an interactive TTY. Non-TTY starts hide it (--print-admin-token forces it; $GBRAIN_ADMIN_BOOTSTRAP_TOKEN + --suppress-bootstrap-token already existed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(security): #2624 banner shows 'from env' before non-TTY hidden guard Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Time AttakcandGitHub
|
010847c020 |
fix(think): enforce source scope across gather (#2200) (#2739)
Carry the caller's scalar or federated source scope through every think gather stream: hybrid page retrieval, takes keyword/vector retrieval, and graph traversal. Adds source predicates to the takes retrieval methods in both engines. Part of #2200 (the think slice; #2200 stays open as the tracking issue for the remaining by-slug read ops). |
||
|
Time AttakcandGitHub
|
8e84c5b4a1 |
fix(facts): parse escaped pipes in facts fence round-trips (#2726) (#2738)
parseRowCells now splits on unescaped pipes only and decodes escaped pipes while preserving ordinary backslashes and empty cells, so facts whose text contains literal | survive the fence->DB reconcile instead of being silently deleted. Fixes #2726. |
||
|
Time AttakcandGitHub
|
68ed7bafa4 |
fix(facts): quarantine ambiguous entity matches (#2723) (#2737)
Bare names resolve only when prefix expansion finds exactly one canonical candidate; ambiguous collisions and low-specificity multi-token fuzzy matches fall through to the guarded holding path instead of confident wrong attribution. Fixes #2723. |
||
|
Time AttakcandGitHub
|
42ab0956a4 |
fix(migrate): preserve sources and scope resume targets (#2677) (#2736)
migrate --to now copies the complete source catalog before pages (fixes the pages_source_id_fkey failure on multi-source brains), and resume manifests carry an opaque target identity so a checkpoint from one target is discarded for a different target. Fixes #2677. |
||
|
Time AttakcandGitHub
|
2fca124468 |
fix(schema): unblock pre-v121 schema replay (#2724) (#2735)
Adds the missing timeline_entries.event_page_id forward-reference bootstrap probe + bare-column repair to both engines so pre-v121 brains can replay the current schema and reach migration v121. Fixes #2724. |
||
|
|
a25209bbb2 |
v0.42.58.0 fix(ai): provider-agnostic gateway — env clobber, base-URL /v1, embedding dims (#1249 #1250 #1292 #2271 #2209) (#2627)
* fix(ai): drop empty-string env values before merge so they can't clobber config keys (#1249) Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls; an unconditional process.env spread let that empty string override a valid config.json key, breaking every gateway op with NO_ANTHROPIC_API_KEY. Filter '' / undefined before the merge; '0' and 'false' are preserved. * fix(ai): normalize native provider base URLs + replace embedding guard with a dims-presence check (#1250, #1292) #1250: createAnthropic/createOpenAI were called with no baseURL, so an env-injected bare host (e.g. ANTHROPIC_BASE_URL without /v1) 404'd. Add a shared resolveNativeBaseUrl and pass a normalized baseURL at all anthropic + openai native sites (google deferred until its suffix is verified). #1292/D6: the user_provided_model_unset guard was structurally unreachable as a no-model check (parseModelId throws on a bare provider) and only ever false-positived for litellm:<model>, silently disabling vector search. Replace it with a real dims-presence check for user-provided/zero-default recipes and delete the dead branch in both consumers. Also stop configureGateway from fabricating a default embedding_dimensions, so 'no dims set' stays honest. * fix(ai): trust user-declared embedding dims for local recipes + litellm /v1 hint (#2271, #2209) #2271: a new trust_custom_dims flag adds a passthrough tier so ollama / llama-server / litellm accept a user-supplied --embedding-dimensions instead of being hard-rejected. Fail-closed for fixed-dim providers (openai/voyage/ zeroentropy) and excludes openrouter (declares dims_options). Register modern ollama embed model names. #2209: litellm setup_hint now states the /v1 path convention and the docs pointer is corrected to docs/integrations/embedding-providers.md. * docs+test(ai): KEY_FILES current-state for provider-agnostic gateway + embed-preflight dims-unset test (#1249, #1250, #1292) * fix(ai): point user_provided_dims_unset remediation at 'gbrain init' (config set rejects it) + coverage Pre-landing adversarial review (P1): the new dims-unset guard told users to run 'gbrain config set embedding_dimensions <N>', which config.ts hard-rejects (it's a schema-sizing field). Both consumer messages now point at 'gbrain init --embedding-dimensions'. Adds: pgvector-cap-still-fires regression for the trust_custom_dims passthrough, and a configureGateway backfill-invariant test. * chore: bump version and changelog (v0.42.57.0) Provider-agnostic plumbing wave: #1249 empty-env clobber, #1250 native baseURL normalization, #1292 embedding dims-presence guard, #2271 trust_custom_dims passthrough, #2209 litellm /v1 hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync embedding-providers guide for provider-agnostic gateway wave (v0.42.57.0) Post-ship doc drift fix for the v0.42.57.0 AI-gateway wave: - LiteLLM section now names the /v1 base-URL convention (#2209). - Ollama section lists the newly-registered modern embedders qwen3-embed-8b + snowflake-arctic-embed-l-v2, and notes dims-trust for local recipes (#2271). - llama-server section notes gbrain trusts the user-declared dimension (#2271). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: post-ship doc sweep for v0.42.57.0 provider-agnostic gateway wave - KEY_FILES.md types.ts entry: document EmbeddingTouchpoint.trust_custom_dims (#2271 passthrough tier, runs after dims_options + Matryoshka allowlists) - ENGINES.md: embedding design-choice note now names the provider-agnostic gateway delegation instead of the stale OpenAI-only parenthetical - embedding-providers.md: drop an exact-duplicate doctor-8c paragraph - llms-full.txt regenerated (ENGINES.md is inlined in the bundle) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply codex doc-review findings for v0.42.57.0 (base-URL env note, litellm multimodal) - embedding-providers.md OpenAI section: document OPENAI_BASE_URL / ANTHROPIC_BASE_URL bare-host /v1 normalization (#1250 user-facing surface) - TL;DR table: litellm multimodal is backend-permitting (recipe declares supports_multimodal: true, routed via the openai-compat multimodal path), not "no" Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: pin engine-find-trajectory schema to 1536 + stop gateway-state leaks across shard files CI shard 5 failed 7 findTrajectory tests with 'expected 1280 dimensions, not 1536': engine-find-trajectory hardcodes 1536-d vectors but sizes its schema from AMBIENT gateway state in beforeAll — which runs before the legacy-embedding-preload's per-test 1536 restore. A preceding file that ends with a dimensionless configureGateway (facts-extract-silent-no-op) or a bare resetGateway poisons the next fresh initSchema down to 1280-d columns. The new test files in this PR reshuffled shard bin-packing and exposed the trap. - engine-find-trajectory: pin OpenAI/1536 explicitly before initSchema (the pattern bunfig's preload documents) — deterministic regardless of neighbors - facts-extract-silent-no-op, diagnose-embedding-dims, embed-preflight: restore the legacy 1536 pin in afterAll instead of ending reset/dimensionless Reproduced: synthetic dimensionless-gateway file + old victim = the exact 7 CI failures; with the pin = 0. Verified in-process pair runs both orders. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
058f448b9a |
v0.42.57.0 fix(pglite): incident — never steal a live data-dir lock + corrupted-store recovery hint (#2348) (#2400)
* fix(pglite): never steal the data-dir lock from a live holder (#2348) A busy `gbrain dream`/`embed` holder whose 30s heartbeat lapsed (the JS event loop is blocked during long synchronous WASM imports/CHECKPOINTs) used to get its lock reaped past the steal-grace window. PGLite/WASM is strictly single-writer, so a second OS process then opened the same data dir and corrupted the catalog + pgvector extension state (58P01 / internal_load_library / "type vector does not exist"), recoverable only by wipe+restore. Reap ONLY a dead PID; a live holder is never stolen — a wedged-but-alive or PID-reused holder makes the acquire time out with a message naming the PID. Removes the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS knob (no longer meaningful). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pglite): point a corrupted store at reinit-pglite recovery (#2348) classifyPgliteInitError gains a `corrupt` verdict for the 58P01 / internal_load_library / "vector does not exist" / "content_chunks does not exist" signature (beating the generic wasm-runtime match), so an already-damaged store gets actionable recovery (gbrain reinit-pglite / restore a backup) instead of the wrong macOS-WASM hint. Updates KEY_FILES.md to current lock behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.55.0 fix(pglite): incident — never steal a live lock + corrupted-store recovery hint (#2348) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.56.0 chore: re-bump past #2399 version collision + refresh ownerToken comment #2399 (security wave) claimed 0.42.55.0; take the next slot. Also updates the LockHandle.ownerToken JSDoc to current #2348 behavior (live holders are never reaped, so reap-then-reacquire is dead-holder-only; token guard stays as defense-in-depth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
646179047a |
v0.42.56.0 feat(chronicle): Life Chronicle — temporal timeline + thought diary + bi-temporal per-entity ontology (#2390) (#2533)
* feat(chronicle): register event + diary page types (#2390) Life Chronicle Phase A.1. Adds `event` (timeline atom) and `diary` (first-person interiority) as temporal-primitive page types under life/events/ and life/diary/, extractable:false — registered in ALL_PAGE_TYPES, both base schema packs, and the inferType prefix table, with parity fixtures. Also lands the chronicle read result types (ChronicleTimelineRow/ChronicleTimelineOpts/LastSeenResult) consumed by Phase A.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): event_page_id timeline projection + day/since/last-seen reads (#2390) Life Chronicle Phase A.2. Adds a nullable event_page_id FK to timeline_entries (migration v120; mirrored in schema.sql, pglite-schema, schema-embedded) so a type:event page projects ONE date-index row keyed to its depth page; a partial UNIQUE(event_page_id, date) makes re-extraction with a changed summary an update, not a duplicate. Dual-engine getTimelineForDate / getSince / getLastSeen filter the depth page on deleted_at, hide soft-deleted event projections at READ time (not just doctor), order by event effective_date for intra-day sequence, and honor source scope (sourceIds[] > sourceId). Ops surface as `gbrain day <date> [--week]`, `gbrain since <date> [--kind]`, `gbrain last-seen <entity>`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): auto-emit extractor — backstop + chronicle_extract job (#2390) Life Chronicle Phase A.3. A put_page backstop (gated on status==='imported', the auto-link/timeline trust gate, and the default-OFF auto_chronicle flag; diary + event pages never eligible) enqueues a chronicle_extract minion job. The job runs the extractor off the write path: deterministic when/who, an injectable LLM judge (default = chat gateway; no-op when no gateway), an all-or-nothing parse barrier (a malformed proposal writes NOTHING), then content-addressed event pages + a timeline projection via the new dual-engine upsertEventProjection (idempotent — re-run yields one event + one projection). New: src/core/chronicle/{eligibility,config,extract-events,backstop}.ts, engine.upsertEventProjection (both engines), the jobs.ts handler + a 10-min timeout. 14 unit tests (eligibility, idempotency, parse barrier, backstop gating + enqueue) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): quick-capture for diary + manual events (#2390) Life Chronicle Phase A.5. `gbrain capture` now routes the default slug by type (diary → life/diary/, event → life/events/, else inbox/) and accepts --who/--what/--where/--kind/--depth to assemble the `event:` frontmatter block for `--type event`. User-declared event keys win per-key over the flags. Goes through the existing put_page → write-through → embed path. 6 new unit tests; existing capture tests stay green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): bi-temporal per-entity ontology on the facts table (#2390) Life Chronicle Phase B.10 — the feature's differentiator. Rather than a parallel store, the open-world per-entity ontology EXTENDS the existing `facts` table (eng-review G1): migration v121 adds dimension/value/value_hash /dim_status columns + a deterministic partial-UNIQUE dedup key + an asof read index. facts already supplies bi-temporal validity, supersession (superseded_by), visibility (remote redaction), confidence, provenance, and embedding — all inherited. Dual-engine methods: mergeOntologyFact (deterministic value_hash dedup → idempotent retry; same value corroborates; a different value forward-closes the prior row's valid_until + superseded_by; a BACKDATED conflicting value is recorded WITHOUT rewriting, surfaced by findOntologyConflicts), getOntology with `--asof` valid-time travel (expired_at + status + validity-window in the predicate so quarantined/superseded never leak), discoverOntologyDimensions, findOntologyConflicts. Novel/LLM-proposed dimensions quarantine; a seed lexicon canonicalizes name drift (job_role → role). 9 unit tests cover the full lifecycle; typecheck pins both engines to the interface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): ontology ops — get/add/dimensions/contradictions (#2390) Life Chronicle Phase B.11. Exposes the bi-temporal ontology over CLI + MCP (contract-first, auto-generated): `gbrain ontology <entity> [--asof]`, `gbrain ontology-add <entity> <dim> <value>`, `gbrain ontology-dimensions` (meta-ontology rollup), `gbrain ontology-contradictions`. All reads route through sourceScopeOpts. Privacy: ontology_get redacts diary-sourced observations (source under life/diary/) for untrusted (remote) callers. 3 op tests (incl. the remote-redaction path); 47 op-registry/tool-def/description tests stay green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): agent-context loader — volunteer_chronicle (#2390) Life Chronicle Phase B.12. `loadChronicleContext` hands an agent the recent timeline (last N days) + the validity-resolved current ontology for the entities in play, in one zero-LLM payload, so it orients before acting — the exact gap behind fumbled chronology. Pure composition over getSince + getOntology (no new SQL). Exposed as the `volunteer_chronicle` read op (`gbrain orient [--days] [--entities a,b]`); diary-sourced ontology is redacted for remote callers. 2 loader tests + op-registry green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): backfill op — sweep existing meetings into events (#2390) Life Chronicle Phase A.8. `chronicle_backfill` (local-only admin op; `gbrain chronicle-backfill [--since] [--limit] [--dry-run]`) lists existing meeting/conversation/calendar pages (source-scoped via listPages), filters through the chronicle eligibility predicate, and enqueues one chronicle_extract job per eligible page so existing brains populate the timeline. --dry-run counts only; per-page enqueue failures are surfaced in `errors`, never swallowed. 2 op tests (dry-run count + enqueue). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): delight — on-this-day + narrative rendering (#2390) Life Chronicle Phase A.6 (delight). Dual-engine getOnThisDay (events from the same month-day in prior years; `gbrain on-this-day [--date]`) reusing the chronicle JOIN shape (deleted-event hiding + source scope). A pure renderTimelineNarrative turns timeline rows into prose; `gbrain day --narrative` returns it alongside the events. 5 tests. (Coverage gap-detection ships with the advisor collectors next.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): proactive advisor collector (#2390) Life Chronicle Phase A.7. A brain-state advisor collector surfaces two proactive signals in `gbrain advisor`: unresolved ontology conflicts (warn, → `gbrain ontology-contradictions`) and recent meetings with no timeline coverage (info, → `gbrain chronicle-backfill`). Advisory-only (no dispatch_id); runs over MCP too (not workspace-dependent); tolerant of pre-migration brains. 3 tests + advisor-op-gate green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): doctor chronicle_projection_health probe (#2390) Life Chronicle Phase B.13. An always-run doctor check (keyed off the event_page_id schema, NOT a migration verify-hook) counts timeline projections whose event page is soft-deleted — hidden at read time, surfaced here as a cleanup backlog (`gbrain integrity auto`). Tolerant of pre-migration brains. 1 detection test. (auto_chronicle / chronicle.tz flags already work via getConfig defaults; their docs land with document-release at ship.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): E1 temporal recall — chronicle-type boost on temporal queries (#2390) Life Chronicle Phase A.4 (E1, ambient temporality). Rather than a separate RRF arm (which needs chunk hydration + risks the fusion path), E1 is a bounded post-fusion boost: applyChronicleTypeBoost lifts `event`/`diary` results on temporal queries, wired INSIDE runPostFusionStages' `recency !== 'off'` branch so it fires ONLY on temporal intent — non-temporal search is bit-for-bit unchanged (proven by 110 passing search-path tests). Bounded ([1.0,1.25]) + floor-gated like the other metadata stages; attribution via `chronicle_boost`. 3 unit tests. (Empirical precision/negative measurement lands with the eval.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): feature eval — gbrain eval chronicle (PRIMARY proof) (#2390) Life Chronicle Phase A.9, the North-Star proof. A deterministic, CI-safe eval (brings its own in-memory PGLite; no LLM, no gateway) builds a synthetic month corpus with a known gold chronology + a planted ontology supersession + a planted conflict, then scores the chronicle layer on six gold tasks: day reconstruction (intra-day order), last-seen exact date, ontology supersession, --asof valid-time travel, contradiction surfacing, and source isolation. `gbrain eval chronicle [--json]` exits 0 iff all pass — currently 6/6. The OFF baseline (raw meeting pages) structurally can't order intra-day events or time-travel ontology; the ON path does. (The live-LLM OFF-vs-ON agent arm + LongMemEval temporal slice are a follow-up; this deterministic bar gates CI.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chronicle): pre-landing review — conflict validity, parse-barrier date, remote conflict redaction (#2390) Three bugs caught by the codex pre-landing review on the diff: 1. findOntologyConflicts ignored valid_until, so a normal forward supersession (founder→advisor from two sources) falsely reported as a live conflict. Now restricted to currently-open rows (valid_until IS NULL) in both engines. 2. The extractor parse barrier accepted any when-string >= 4 chars; a non-date value slipped past, wrote the event page, then threw on the projection's ::date cast (partial write). isValidProposal now requires a real parseable date. 3. The ontology_conflicts op had no remote diary redaction (ontology_get did); remote callers now get diary-sourced values filtered, and conflicts that lose their disagreement after redaction are dropped. Three regression tests added; 29 chronicle tests + eval 6/6 green; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.56.0 feat(chronicle): Life Chronicle — temporal timeline + diary + bi-temporal ontology (#2390) Bump VERSION + package.json to 0.42.56.0 and add the CHANGELOG entry for the Life Chronicle feature (#2390, closes duplicate #2388). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chronicle): register #2390 surfaces with the five CI guard suites (#2390) CI caught five guard tests that pin registration invariants my diff tripped: - schema-bootstrap-coverage: the four v122 facts ontology columns join COLUMN_EXEMPTIONS (facts is migration-created; the partial indexes live inside v122; every reader filters dimension IS NOT NULL — same precedent as facts.claim_metric et al). - no-valid-until-write (R8): both engines' mergeOntologyFact forward- supersession is a deliberate, documented valid_until write authority (engine-layer, dimension IS NOT NULL only; the contradiction probe still never mutates). - doctor-categories: chronicle_projection_health registered under BRAIN_CHECK_NAMES (same class as child_table_orphans). - checkTypeProliferation: the test is now threshold-relative (computes declared from the active pack, seeds declared+6) so base-pack growth can't silently move the fixed threshold again. - schema-cli: gbrain-base page-type count 25 → 27 (event + diary), with assertions on both new types. All 41 guard tests green; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: file Life Chronicle follow-up TODOs (v0.42.56.0, #2390) Eight deferred items from the CEO/eng review decisions (auto-emit default-flip fast-follow, live eval arm + LongMemEval slice, passive diary consent, interval-splitting, federation, place-as-entity, meta-ontology dashboard, materialized daily pages), each with decision provenance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): KEY_FILES entry for the Life Chronicle module (#2390) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
dde1132a29 |
v0.42.55.0 fix(security): dotfile/skills/slug confinement + DCR consent default + schema-lint migration (#418 #419 #245 #1353 #1647 #171 #1385) (#2399)
* fix(security): confine routing dotfiles, skills dir, slugs, and transcription exec Shared src/core/path-confine.ts consolidates the realpath-containment idiom (moved from sources-ops.ts) and adds isTrustedDotfile + isWriteTargetContained. - .gbrain-source (source-resolver) and .gbrain-mount (brain-resolver) walk-up dotfiles are now lstat trust-gated: a symlink, foreign-owned, or world-writable file is refused on multi-user hosts (#418), fail-closed on stat error. - resolveWorkspaceSkillsDir + every skills-dir tier (env, cwd_walk_up, repo_root, cwd_skills, install_path) route through realpath containment so a symlinked workspace/skills can't escape the declared workspace (#419). - resolveSourceId/resolveBrainId realpath both sides of the registered local_path / mount prefix match so a symlinked cwd can't misattribute source/brain. - validateSlug rejects NUL/control, bidi/RTL overrides, backslashes, and URL-encoded path separators at the shared putPage/updateSlug chokepoint; write-through confirms the file path stays within the source tree. - transcribeLargeFile uses execFileSync arg-arrays + fs.rmSync (no shell), so a path with shell metacharacters is never parsed by a shell (#245). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): default dynamic-registration clients to authorization_code Self-registered DCR clients (the unauthenticated network registration path) previously defaulted to the client_credentials grant, which bypasses the /authorize consent screen. They now default to authorization_code; an explicit client_credentials request is rejected with invalid_client_metadata unless the operator opts in with the new --enable-dcr-insecure flag. A loud stderr WARNING prints at startup whenever DCR is enabled (#1353). Manual CLI/admin client registration is unchanged (operator-trusted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): schema-lint hardening migration (search_path + view security_invoker) Migration v120 brings existing brains to the same posture as fresh installs: - ALTER VIEW page_links SET (security_invoker = on) on Postgres so the view honors the caller's RLS instead of the owner's (the view-through-RLS bypass). - ALTER FUNCTION ... SET search_path on the gbrain-owned trigger/event functions (both engines, IF EXISTS so engine-only functions are skipped; body untouched, so the load-bearing auto_enable_rls event trigger is unchanged). Closes #171. - Broaden the BYPASSRLS preflight in the historical RLS migration gates to honor superuser and inherited-role BYPASSRLS, so a superuser-connected fresh install no longer aborts (#1385). Fresh-install function definitions in schema.sql / pglite-schema.ts are born-correct (regenerated schema-embedded.ts). scripts/check-search-path.sh is a new CI guard (wired into verify) that fails if a trigger function in the schema base files is added without SET search_path. Postgres-only assertions live in the bootstrap E2E; the PGLite path is covered by test/migration-v120.test.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.55.0 fix(security): dotfile/skills/slug confinement, DCR consent default, schema-lint migration Bump VERSION + package.json to 0.42.55.0 and add the CHANGELOG entry for the security-hardening wave (#418 #419 #245 #1353 #1647 #171 #1385). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(security): note the DCR consent default in SECURITY.md (#1353) The "disable client_credentials, only allow authorization_code" guidance is now the built-in DCR default; document the new --enable-dcr-insecure escape hatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(todos): add takes_search + code_def to the federated by-slug P1 (#2200) The v0.42.55.0 eng-review codex pass flagged takes_search (holder-allowlist only) and code_def (brain-wide raw SQL over content_chunks) as remaining same-class surfaces. Noted on the existing #2200 P1 follow-up, with the caveat that the #2399 close-list deliberately keeps #1371/#2200 open until this lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): correct plpgsql alias collision in #1385 BYPASSRLS gate (real-PG) The broadened BYPASSRLS preflight aliased `pg_roles r`, but several RLS DO-blocks already declare `r record` for their backfill FOR loop, so plpgsql resolved `r.oid`/`r.rolbypassrls` to the unassigned record variable → "record \"r\" is not assigned yet" on real Postgres (PGLite tolerated it; the DATABASE_URL-gated e2e jobs are the backstop). Renamed the subquery alias to `pr` at all 10 migrate.ts sites; also broadened the schema.sql base RLS gate the same way (with the `pr` alias) for #1385 consistency on superuser fresh installs, and regenerated schema-embedded.ts. Also fixes a PRE-EXISTING engine-parity bug (confirmed failing on clean origin/master): the relationalFanout shape compared `canonical_chunk_id`, a serial id that diverges between a fresh PGLite engine and a shared Postgres DB (setupDB TRUNCATEs without RESTART IDENTITY). Compare its presence, not the exact value. Validated on real Postgres (pgvector/pg16): migration v120 applies, the v35 RLS backfill runs, and engine-parity + postgres-bootstrap + jsonb-parity are green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
814258dda6 |
v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard (#2375)
* fix(sync): op_checkpoints pin write double-encodes jsonb — every sync aborts (#2339) recordCompleted bound JSON.stringify(array) to a $3::jsonb param via postgres.js .unsafe(), double-encoding it into a jsonb string scalar that violates the v119 op_checkpoints_completed_keys_array CHECK — aborting every multi-source sync on real Postgres at the first checkpoint write. PGLite parses the string silently, which is why unit tests stayed green and it shipped. Cast through $3::text::jsonb so the text->jsonb cast parses a genuine array. Adds a DATABASE_URL-gated parity test + a dedicated Postgres CI job so the guard can never silently skip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db): sweep positional jsonb double-encode sites + AST CI guard (#2324) Every executeRaw/.unsafe site that bound JSON.stringify(x) to a bare positional jsonb cast double-encodes on real Postgres (same class as #2339). Sweep them all to the text::jsonb form across query-cache, sources-ops, llm-base, calibration-profile, impact-capture, subagent, receipt-write, traversal-cache, symbol-resolver, and the agent/sources commands. Adds scripts/check-jsonb-params.mjs (AST-lite scanner for the positional form the legacy template grep misses, incl. generic-typed calls), wired into check-jsonb-pattern.sh, with a self-test. PGLite's native db.query is not scanned — it parses text to jsonb natively, so the bug can't occur there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(search,eval): alias-hop injected results carry page_id (contradiction-probe crash) applyAliasHop injected synthetic SearchResults without page_id (the `as SearchResult` cast hid the missing field), so listActiveTakesForPages bound undefined/NaN into ANY($1::int[]) and crashed the whole contradiction probe on real Postgres. Stamp page_id=page.id at the injection site and add a finite-id filter in generateIntraPagePairs as a defensive backstop (mirrors hybrid.ts:63). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(engines): positional jsonb binding rule (text::jsonb vs the double-encode trap) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard Bumps VERSION + package.json to 0.42.53.0, adds the CHANGELOG entry, and regenerates llms-full.txt. Ships the #2339 sync-abort hotfix, the repo-wide positional jsonb double-encode sweep, the alias-hop contradiction-probe crash fix, and the new positional-form CI guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: post-ship sync — jsonb invariant now covers the positional form + new guard CLAUDE.md JSONB invariant + KEY_FILES (sql-query, check-jsonb-pattern, op-checkpoint) now describe the #2339 positional double-encode class, the $N::text::jsonb fix, and the new check-jsonb-params.mjs guard. Regenerates llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bb2e88c42a |
v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984) (#2287)
* test(supervisor): pin LOCK_HELD fence-exit is never counted as a crash (#2227) A duplicate supervisor loses the queue-scoped DB singleton lock (#1849) and exits LOCK_HELD before spawning a worker or emitting 'started'. summarizeCrashes counts only worker_exited, so the fence path is structurally uncountable. Pin it so a future refactor that logs worker_exited on the fence path fails here instead of silently re-introducing the crash-budget breaker-trip loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(autopilot): per-source cycle binds FS phases to source.local_path, not global repo (#2194 #2227) A per-source autopilot-cycle inherited the global sync.repo_path as brainDir while stamping DB freshness for source_id — mixed scope. FS phases (sync/lint/extract) ran against the wrong tree, so the failure-cooldown and freshness gates would attribute work to the wrong source. Resolve the source's local_path in the handler (reuse the archive-recheck SELECT) and bind brainDir to it; a pure-DB source gets null (FS phases skip) instead of falling through to the global checkout. Legacy no-source dispatch keeps the global repoPath. Prerequisite for the cooldown/split commits (codex outside-voice #8). Resolves TODOS:634. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(supervisor): detect a live supervisor via the DB lock under split $HOME (#2227) jobs supervisor status + doctor read the HOME-derived pidfile, so a supervisor started under a different $HOME (keeper=/root vs ops=/data) read as 'not running' while healthy — the false signal that drives an operator to spawn a duplicate. Both surfaces now fall back to the queue-scoped DB singleton lock (#1849), the HOME-independent authority, when the pidfile shows nothing. New isLockHolderLive keys on lock freshness (ttl + heartbeat steal-grace), never process.kill, so PID reuse can't false-positive (pid-liveness-alone-pid-reuse). Status surfaces the holder host/pid + recorded concurrency/max-rss from the latest started event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(supervisor): degraded retry instead of permanent give-up on crash storm (#1994 #2227) max_crashes_exceeded gave up forever, so a transient DB-pooler blip that tripped the soft budget wedged the queue until a human restart (#2227's breaker-trips tail). Crossing the soft budget now enters degraded mode: keep respawning with capped exponential backoff (60s cap — a paced retry, not a hot loop) and emit a loud crash_budget_degraded health_warn. The existing stable-run reset clears the count once a respawn survives >5min, so a recovered DB self-heals. Permanent give-up fires only at a much-higher hard ceiling (maxCrashes × 10), tunable/disablable via GBRAIN_SUPERVISOR_HARD_STOP_CRASHES (0 = never). Resolves TODOS:92. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): clamp fan-out to worker concurrency + doctor warning (#2194) Fan-out resolved to 4 (Postgres) regardless of worker --concurrency, so surplus cycles queued behind the worker and raced the stalled-sweeper. Two fixes for the same mismatch: - resolveEffectiveFanoutMax clamps to max(1, concurrency-1) (reserve a slot), gated on a LIVE DB-lock holder so a stale started-audit row can't shrink throughput (codex #9/D5); no live holder → unknown → unclamped base. Escape hatch autopilot.fanout_clamp_to_concurrency. - doctor's autopilot_fanout_concurrency check warns when fan-out exceeds effective slots — the misconfig was silent before. Advisory (started-event concurrency), wired into both doctor surfaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): per-source failure cooldown — break the dead-job storm (#2194) Only SUCCESS gated dispatch, so a source whose cycle kept failing/timing-out re-fanned-out every 5-min tick forever (200+ dead jobs/24h). Now a failed source backs off with bounded exponential cooldown (10→120min). Read at DISPATCH from minion_jobs dead/failed rows (timeouts/RSS-kills dead-letter via SQL and never run handler code, so a write-only hook would miss them) AND re-checked at CLAIM time in the handler (codex #5: already-queued/retrying jobs). A success clears it (codex #7); null-source rows excluded (codex #6); engine-parity via executeRaw. Disable with autopilot.failure_cooldown_min=0. Fail-open if config/history reads error. Surfaced via fanout_cooldown_skipped + the fanout summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): split the cycle — per-source phases + one global-maintenance job (#2194 #2227) N per-source cycles each ran the brain-wide global phases (embed-all/orphans/ purge/…) concurrently, thrashing the same rows and taking the worker 4→10GB in <60s → RSS-kill → orphaned stalls. Split them: per-source jobs now run only source-scoped (+ mixed) phases and stamp last_source_cycle_at; a new autopilot-global-maintenance job runs the global phases ONCE per window (idempotency_key + maxWaiting:1 = structural single-flight) and stamps autopilot.last_global_at. This is the codex-endorsed design that replaced the rejected skip-and-stamp-fresh approach (codex #1/#2): no freshness poisoning, no starvation — global work always runs as its own job, never marked done when it wasn't. PHASE_SCOPE is now a runtime partition (GLOBAL ∪ NON_GLOBAL == ALL). last_full_cycle_at still written for doctor/legacy (no longer a global gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): guard nullable engine in supervisor DB-lock fallback (#2227) Follow-up to the supervisor-visibility commit: doctor's engine binding is BrainEngine | null, so the inspectLock fallback must guard on a non-null engine (tsc TS2345). No behavior change — a null engine simply skips the DB-lock probe and falls back to the pidfile reading, as before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): categorize autopilot_fanout_concurrency check as ops (#2194) Follow-up to the fan-out/concurrency commit: the doctor-categories drift guard requires every check name in doctor.ts to belong to exactly one category set. Add the new autopilot_fanout_concurrency check to OPS_CHECK_NAMES (infrastructure liveness, alongside wedged_queue/supervisor). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update KEY_FILES for the autopilot cycle split + supervisor degraded-retry (#2194 #2227) Post-ship document-release: refresh the KEY_FILES current-state entries that drifted — cycle.ts (GLOBAL/NON_GLOBAL phase split + last_source_cycle_at / autopilot.last_global_at), jobs.ts (per-source local_path brainDir, claim-time cooldown, autopilot-global-maintenance handler), supervisor.ts + child-worker (degraded retry instead of permanent give-up; hard ceiling), db-lock.ts (isLockHolderLive), handler-timeouts (new handler). Regenerated llms bundle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minions): handleTimeouts counts the timed-out run as a spent attempt (#1737) The per-job timeout_at dead-letter (handleTimeouts) set status='dead' without incrementing attempts_made, unlike the wall-clock and stall dead-letter siblings. It is the FIRST killer to fire for the long-lane handlers (subagent / embed-backfill / autopilot-cycle) because timeout_ms is stamped at submit, so a timed-out long job reported `attempts: 0/N (started: N)`. Mirror the siblings with attempts_made + 1 (terminal, no retry). Safe against double-count: the worker sweep runs handleStalled -> handleTimeouts -> handleWallClockTimeouts sequentially and awaited, each guarded on status='active', so the first to dead-letter excludes the row from the rest. Regression assertions added (test/minions.test.ts + e2e/minions-resilience.test.ts) so the increment can't be silently dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(agent): recognize trailing switches in `agent run`, keep prompts freeform (#1738) parseRunFlags() broke flag parsing at the first positional token, so any flag after the prompt (`gbrain agent run "do X" --detach`) was swallowed into the prompt string and silently ignored. Now the no-value switches --detach/--follow/ --no-follow are hoisted when they trail the prompt, while everything else stays verbatim: an unknown --word is treated as prompt text (no "unknown flag" throw), a --switch mid-prompt is preserved, and `--` suppresses hoisting entirely for a literal escape. Value-flags now reject a missing or flag-shaped value (and --max-turns/--timeout-ms a non-number) instead of capturing undefined/NaN. Contract change: a prompt that starts with or trails an unguarded --word no longer errors; a literal trailing --detach needs `--`. Help text updated; tests revised + extended (test/agent-cli.test.ts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): honest live-sync status + progress-aware stall-abort (#1950) Finishes the #2255 honest-freshness story for two gaps it left. (a) `gbrain sources status` printed "idle" while a sync proc held the per-source lock (the reported bug). New shared liveSyncStatus() helper in db-lock.ts reads the SAME live-lock signal `gbrain doctor` uses; runStatus now shows "running" (BACKFILL column + a sync_running field in --json) and suppresses the misleading "never synced" warning while a sync is live. One helper, so the surfaces can't drift (doctor/status retrofit tracked as a follow-up). (b) A sync wedged-but-alive kept refreshing its lock heartbeat (it fires on its own timer) and hadn't hit the wall-clock deadline, so only a manual pkill freed it. New in-band stall watchdog keys off FORWARD IMPORT PROGRESS (progress.tick), not the heartbeat: if no file completes for GBRAIN_SYNC_STALL_ABORT_SECONDS (default 900s), it aborts via a controller composed into opts.signal, so the drain returns partial() (last_commit unchanged, next run resumes from the checkpoint) and withRefreshingLock releases the lock. Limits, documented in code: a single file slower than the window trips it; a fully starved event loop won't fire the timer (the wall-clock hard deadline is that backstop). Tests: liveSyncStatus (live/expired/none/per-source) in db-lock-inspect; the resolveStallAbortSeconds env matrix in sync-hard-deadline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(status): version field + per-section --deadline-ms budget (#1984) `gbrain status` had no version in its JSON envelope and could hang on a slow connection with no way to get a partial answer. Two additions: - version: the StatusReport JSON now carries the local gbrain CLI version so a poller can pin behavior to a build. Thin-client also surfaces remote_version (the brain server's version), and the get_status_snapshot MCP op reports its version for that parity. - --deadline-ms=N / --fast: a shared wall-clock budget. Each section is raced against the REMAINING budget via Promise.race (NOT process-watchdog, which SIGKILLs and can't return partial output), so one slow/hung section can't strand the snapshot — it's marked stale and the rest still return. The envelope gains partial:true + stale_sections[]; exit code stays 0 (a snapshot was produced). Invalid --deadline-ms → exit 2. Tests: parseDeadlineFlag + withSectionDeadline (hermetic), the usage-error exit, version presence in the PGLite envelope, and the op's version key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): report stall_timeout distinctly + document in-flight limit (#1950) Pre-landing review (codex + adversarial): the stall watchdog aborted opts.signal but the per-iteration abort checks returned partial('timeout'), collapsing a wedge-reap into a user --timeout/SIGINT so JSON consumers couldn't tell them apart. Add a 'stall_timeout' reason (set via a stallAborted flag) on the three import-loop abort sites; deletes/renames-phase and checkpoint sites stay 'timeout'. Sharpen the watchdog comment: the abort is observed BETWEEN files, so a hang inside a single importFile is not interrupted until it returns (TODO: thread a cancellation signal through importFile). * fix(agent): `--` escape suppresses trailing-switch hoisting anywhere (#1738) Pre-landing review: the leading-flag loop breaks at the first positional, so the `escaped` flag only fired for a leading `--`. A `--` placed after a positional left trailing-switch hoisting active, so `agent run note -- body --detach` silently detached and dropped the `--` as junk. Suppress hoisting whenever a literal `--` appears in the prompt. Regression test added. * fix(status): deadline-ms usage-error + scoped stale_sections + cancel losing remote call (#1984) Pre-landing review (codex): (1) bare `--deadline-ms` with no value silently fell through to no-budget/--fast instead of a usage error; (2) thin-client timeout reported both sync+cycle stale even under `--section sync`, naming a section the caller excluded (local path was already correct); (3) the section race abandoned the remote promise locally but didn't cancel the in-flight MCP call — pass the budget as timeoutMs so the losing side actually cancels. Regression test added. * v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984) Bundles the already-reviewed autopilot/supervisor stabilization (#2194 #2227 #1994: cycle split, per-source failure cooldown, fan-out clamp, degraded supervisor retry, DB-lock live-supervisor detection) with four operational fixes: minion timeout attempt-accounting (#1737), agent-run trailing-flag parsing (#1738), honest live-sync sources status + progress-aware stall watchdog (#1950), and status version + --deadline-ms partial result (#1984). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document GBRAIN_SYNC_STALL_ABORT_SECONDS env knob (#1950) Post-ship doc sync (/document-release): add the sync stall watchdog env var to the CLAUDE.md sync-tuning table (Five → Six knobs) + regenerate the llms bundle. * test: quarantine #2249 fanout tests as *.serial (R1 env-isolation) (#2194) The cherry-picked autopilot-fanout-clamp + doctor-autopilot-fanout-concurrency tests mutate process.env.GBRAIN_AUDIT_DIR in beforeEach/afterEach, which the check:test-isolation R1 lint flags (parallel shards load multiple files per process). Rename to *.serial.test.ts (sanctioned quarantine — they run under --max-concurrency=1) instead of restructuring the reviewed test bodies. No logic change; both files stay green (9 tests). Fixes the failing verify CI check. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9bf96db807 |
v0.42.51.0 fix(sync): contention-free clock + checkpoint integrity + honest sync freshness (#2255)
* fix(sync): contention-free page-generation clock — sequence swap The page-generation clock backed the query-cache Layer-1 bookmark via a FOR EACH STATEMENT trigger running `UPDATE page_generation_clock SET value=value+1 WHERE id=1`. That took a transaction-length RowExclusiveLock on one tuple, so every concurrent page writer serialized on the prior writer's COMMIT — sync ran at ~0.8 cores regardless of worker count. Swap to a SEQUENCE bumped by nextval() (a microsecond LWLock, never a row lock). The clock's only contract is monotonic advancement on any page INSERT/UPDATE/DELETE; last_value is non-transactional, so rolled-back or concurrent-uncommitted writers only OVER-invalidate the cache (lose a hit), never serve stale. - migration v118: CREATE SEQUENCE + load-bearing 2-arg setval (is_called= true, floor 1, seeded >= old clock and MAX(generation)) + repoint the trigger function body + DELETE query_cache so no old-clock bookmark survives the swap. v107 left immutable. - query-cache-gate.ts: 3 readers -> SELECT last_value FROM page_generation_clock_seq. - schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the sequence on fresh install; table + trigger names retained. - tests: clockValue reads last_value; mechanism proof (trigger fn uses nextval not the row UPDATE); rollback-advances-clock safety pin; real PGLite sequence round-trip (is_called gotcha); shape test requires _seq. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): op_checkpoints array-shape guard — CHECK + repair + defensive loader completed_keys is JSONB and the checkpoint loader runs jsonb_array_elements_text over it. A non-array (scalar) value makes that throw "cannot extract elements from a scalar", which takes down the whole UNION load — including the valid op_checkpoint_paths child rows — and loses all checkpoint progress for that key. No current writer produces a scalar, but an older binary / external script / future bug could. Make the corruption class structurally impossible and self-healing: - migration v119: LOCK TABLE (so an out-of-band scalar can't land between repair and constrain; no-op on single-connection PGLite), repair any pre-existing scalar to '[]' (op_checkpoint_paths child rows are the append-only source of truth, so the reset loses nothing), then add the named CHECK (jsonb_typeof(completed_keys) = 'array') via a pg_constraint IF NOT EXISTS guard. A DB-enforced always-on guard — the correct pattern vs a migration verify-hook, which never runs on already-stamped brains. - schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the same NAMED inline CHECK so fresh installs match migrated brains and v119 skips the duplicate. - op-checkpoint.ts loader: gate the legacy arm on jsonb_typeof = 'array' so a scalar parent is skipped (children still load) instead of throwing the whole union, and log a specific corruption warning when one is seen. - tests: CHECK rejects a scalar (exactly one constraint, no blob+migration dupe); loader survives a scalar parent and returns the children; v119 repair converts a scalar to '[]'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): report actively-running sync via live lock, not stale freshness A slow source that makes partial progress every cycle but never fully completes used to read as permanently "stale" / "never synced" because last_sync_at only advances on a full successful sync. The naive fix (treat recent checkpoint banking as "in progress") is unsafe: a blocked sync banks the good files then writes no anchor, so banking can't tell in-progress from wedged. Use the only honest signal: a LIVE, non-expired per-source sync lock (inspectLock + syncLockId against gbrain_cycle_locks). Every non-skipLock sync holds it and refreshes it; a blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either correctly falls through to the stale path and is NEVER masked. An actively-syncing source (including a never-synced source doing its first sync) counts as synced_recently, preserving the pinned 3-bucket invariant. The lock lookup reuses doctor's existing dynamic db-lock import and swallows any throw (stub engine, pre-lock-table brain) to false, so it can only ADD an in-progress verdict, never suppress a real stale one. Tests (real PGLiteEngine + real lock rows): stale+no-lock -> fail; stale+live-lock -> ok; never-synced+live-lock -> ok; never-synced+no-lock -> fail; expired-TTL lock -> fail (wedged not masked); blocked source with banked checkpoint rows but no lock -> still fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): honest --force-break-lock diagnostic when no lock is held --force-break-lock used to emit the same terse "Lock ... is not held (nothing to break)" line and exit 0 even when a sync was genuinely wedged, sending the operator down a dead end — the wedge was not a held lock. Keep rc=0 (breaking a non-existent lock is idempotently successful; flipping the exit code would break automation), but under --force say plainly that nothing was broken and point at the real next step (gbrain sync / gbrain doctor) plus a `wedge_hint` field in --json output. The non-force path is byte-for-byte unchanged. runBreakLock is exported for the test. Tests: force+no-lock -> wedge_hint JSON + human hint, rc 0; non-force+no-lock -> unchanged terse line, no hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): surface the in-progress sync holder in the freshness message Plan-completion follow-up to the BUG 4 live-lock signal: when a source is actively syncing, name the holder (pid + host) in the check message instead of silently folding it into synced_recently. The note is appended only when something is in progress, so steady-state messages stay byte-for-byte unchanged (the pinned exact-message + 3-bucket-invariant tests still pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): pre-landing review fixes — monotonic clock seed, scoped CHECK guard Adversarial (codex) review of the implementation diff caught three: - P1 (correctness): the fresh-schema setval was not monotonic. initSchema replays the schema blob, and the unconditional setval(MAX(generation)) could move page_generation_clock_seq.last_value BACKWARD on an already-upgraded brain, letting a stored query_cache bookmark serve stale rows. Seed via GREATEST over the sequence's OWN last_value (+ old table value + MAX(generation)) in all 3 fresh schemas and migration v118, so a replay is idempotent — mirrors the old table's ON CONFLICT DO NOTHING. Pinned by a new monotonic regression test. - P2: v119's CHECK-exists guard keyed on conname only (not globally unique). Scope it to conrelid = 'op_checkpoints'::regclass. - P3: in-progress note ran into the prior sentence in fail/warn doctor messages; separate it with '. '. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make Anthropic/ZE no-key tests hermetic against a dev config key These "no key" tests cleared only ANTHROPIC_API_KEY / ZEROENTROPY_API_KEY from the env, but hasAnthropicKey() and checkZeEmbeddingHealth() also read the key from ~/.gbrain/config.json. On a dev machine whose real config holds a key, the no-key assertions flipped and the tests failed locally (they passed only in key-less CI). Add a shared with-env emptyHome() helper and point GBRAIN_HOME at an empty dir in every no-key path so loadConfig finds nothing — matching the already-hermetic anthropic-key / gateway-probe tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.44.1.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): sync doctor + op-checkpoint entries to v0.44.1.0 truth checkSyncFreshness now reports an actively-running sync via the live per-source lock (names holder pid+host, counts as synced_recently) instead of flagging it stale; loadOpCheckpoint gates the legacy union arm on jsonb_typeof = 'array' so a scalar parent can't take down the whole load, and migration v119's CHECK constraint makes the corruption class structurally impossible. Reference docs describe current behavior only — both entries updated in place, no release-clause appends. Guard + llms freshness test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-version to v0.42.51.0 (natural next-off-master) Maintainer override of the queue allocator's leap to 0.44.1.0 (it jumped past in-flight sibling PR claims at 0.42.50/0.43.0/0.44.0). Take the natural next slot in the 0.42.x line above the immediate sibling claim (0.42.50.0); a merge re-bump resolves any collision if a cathedral PR lands first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): bound + retry the OpenClaw install so a transient npm hang can't burn the Tier 2 budget The Tier 2 (LLM Skills) job failed at 30m16s — the `npm install -g openclaw@2026.4.9` step hung on a transient npm/registry stall (orphan `npm install openclaw` was still running at cancel time) and consumed the entire 30m job budget that v0.42.50.0 (#2254) introduced. The install normally finishes in under a minute (Tier 2 is ~4m end to end on master), so this is flaky-install infra, not a test failure. Wrap the install in `timeout 120` + a 3-attempt retry loop with an 8-minute step backstop: a hung attempt is killed in 2 min and retried instead of eating the whole job. Same bound-the-hang philosophy as #2254's job timeouts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
7968f84077 |
v0.42.49.0 feat(pace): native DB-contention pacing for embed/sync backfills (#2240)
* feat(pace): composable DB-contention pacer primitive createDbPacer/createNoopPacer (concurrency permit + in-band EWMA + jittered cooperative sleep, abort-throws, fail-open) + named pace-mode bundles (env>config>bundle, default off) + shared embed-backfill lock key. * feat(embed): wire DB-pacing into embed paths + single-flight + bounded keyset re-entry embedStaleForSource + CLI embedAllStale/embedAll lower worker count to the resolved cap and observe()/pace() their DB ops; embed job + embed-backfill handler resolve env>config>bundle; CLI --pace flags; --background carries overrides into the job payload; single-flight via shared per-source lock; budget-timer re-arm around paced sleeps; EmbedResult.pacing telemetry. * feat(sync): shared DB-pacer permit across parallel worker engines One pacer spans the per-worker PostgresEngines (the multi-pool permit case); observe() import writes, pace() between files, dispose on all exit paths. * docs(pace): CLAUDE.md Pace Mode section + regenerated llms bundle * fix(pace): pre-landing review fixes (Codex P1/P2) - never unref() the cooperative-sleep timer (could exit mid-sleep) - pace() excludes the wall-clock budget + re-arm after each sleep - pacing only lowers concurrency, never raises above an operator cap - serialized job pace resolves at config tier so GBRAIN_PACE_* still wins - --pace-max-concurrency consumes its value token * chore: bump version and changelog (v0.42.48.0) Native DB-contention pacing for embed/sync backfills. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version to v0.42.49.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
7ea92d602c |
v0.42.48.0 feat(durability): auto-harden brain repos for git durability on PAT+URL (#2241)
* feat(git): divergence-safe pull, push-probe, default-branch detection for brain durability Add GIT_ENV_AUTH + divergenceSafePull (skip-on-dirty, conflict-abort-clean, never-mid-rebase), detectDefaultBranch, pushProbe, and an env-gated GBRAIN_GIT_ALLOW_FILE_TRANSPORT escape hatch. Export GIT_ENV. pullRepo's --ff-only contract is unchanged. * feat(durability): brain-repo hardening core (hook, helper, cron, PAT, AGENTS rules) hardenBrainRepo/unhardenBrainRepo: local untracked post-commit hook + committed brain-commit-push.sh (one shared push-retry template), repo-scoped credential with existing-helper reuse, push-probe verify, active-resolver-file rules with taxonomy from _brain-filing-rules.json, minimal DB-free pull cron. PAT redaction via redactSecretsInText. * feat(sources): harden/pull/unharden commands + auto-harden on add --url sources harden/pull/unharden subcommands; --pat-file/--no-harden on add; auto-harden managed clones on add; unharden-before-remove. cli.ts pre-connect early-exit for DB-free 'sources pull --path' (the cron entry, never opens PGLite). * test(durability): unit + integration coverage for brain-repo durability git helpers, core harden/unharden, hook+helper E2E (real background push), cron generators. 41 tests across 4 files. * chore: bump version and changelog (v0.42.48.0) Brain-repo git durability: auto-harden a brain's working tree (local auto-push hook, committed commit-push helper, always-on agent rules, DB-free pull cron, repo-scoped credential, push-probe verify) the moment gbrain gets a PAT + URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sources): route harden exit code through setCliExitVerdict A raw process.exitCode write is zeroed by the owned-verdict flush-exit (#2084 PGLite-Emscripten pollution defense); cli-exit-verdict-pin guard caught it. Use setCliExitVerdict(3) so 'sources harden' actually reports needs-attention to cron/automation. * docs: document brain-repo durability (KEY_FILES + multi-source guide) KEY_FILES: extend git-remote.ts entry (divergenceSafePull, pushProbe, detectDefaultBranch, GIT_ENV_AUTH, GBRAIN_GIT_ALLOW_FILE_TRANSPORT) + add brain-repo-durability.ts/sources-harden.ts entry. multi-source-brains.md: add a Durability (auto-harden) how-to covering sources harden/pull/unharden, --pat-file, the guarantees, and the security posture. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9d88680a51 |
v0.42.47.0 feat(skillpack,advisor): brain-resident skillpacks + proactive gbrain advisor (#2180) (#2231)
* feat(skillpack): brain_resident manifest fields + init-brain-pack scaffolder + tools version-skew lint Add optional brain_resident/schema_pack to the v1 manifest (additive, forward-compatible). New runInitBrainPack scaffolds a brain-resident pack (brain_resident:true, exact gbrain_min_version, 5-section machine-parseable README) beside brain content; applyWritePlan factored out of init-scaffold. brain-pack-lint validates each skill's declared tools: against the serving op set (E6 version-skew). Wires gbrain skillpack init-brain-pack. * feat(skillpack): Topology A brain-pack discovery on sources add + bounded nag After 'gbrain sources add', if the source ships a brain_resident pack, print an agent-readable advisory (ask the user before scaffolding). nag-state.ts tracks declines per (source-repo brain_id, source, pack) with escalate-then-suppress; declines count only on CLI-interactive displays, never cron/MCP. Fail-open: a malformed/absent pack never breaks sources add. * feat(advisor,skillpack): list_brain_skillpack MCP tool + gbrain advisor Topology B: dedicated source-scoped list_brain_skillpack op + get_skill source_id disambiguation (brain-resident-locate.ts); git scaffold-spec never a server FS path; source-aware schema match. LEARN_INSTRUCTION + serve-http banner. gbrain advisor: read-only ranked actions from brain state (8 resilient collectors, shared renderer, JSONL history, --json severity exit codes, local-only argv --apply dispatcher). Exposed over MCP behind mcp.publish_advisor (default off, read-only on remote; workspace collectors no-op remotely). Generalizes post-install-advisory to a single current-state recommended set (install→scaffold). * feat(skills): bundle gbrain-advisor skill + weekly cron recipe + ranking eval skills/gbrain-advisor teaches a harness to run gbrain advisor on a cadence and ping the user (read-only; ask before fixing). Registered in manifest.json, RESOLVER.md, openclaw.plugin.json. E4 ranking-precision eval on seeded-defect fixtures (100%). * chore: bump version and changelog (v0.42.47.0) Brain-resident skillpacks + gbrain advisor (#2180). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync CLAUDE.md + KEY_FILES for brain-resident skillpacks + advisor (#2180) Skill count 29->30, Skills section gains the brain-resident skillpacks + advisor capability, KEY_FILES gets current-state entries for the new modules. Regenerate llms bundles. * test,fix: align stale assertions with generalized advisory + advisor resolver triggers (#2180) - post-install-advisory.test.ts: install→scaffold wording (the install verb was removed); restore two-column in book-mirror copy; drop the removed skillpack-list line. - RESOLVER.md: gbrain-advisor trigger now fuzzy-matches a declared frontmatter trigger (resolver round-trip D5/C). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: regenerate llms bundle for updated advisor resolver row (#2180) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
c023a6041d |
v0.42.46.0 fix(engine): federated read scope reaches by-slug reads (#2200) (#2239)
* fix(engine): federated sourceIds[] scope on by-slug secondary reads (#2200) getTags/getLinks/getBacklinks/getTimeline (both engines) + TimelineOpts now accept a federated `sourceIds[]` read grant, precedence over scalar sourceId, filtering `source_id = ANY($::text[])` — mirroring getPage from v0.42.37.0. - getTags: `page_id = (subquery)` -> `IN (subquery)` + DISTINCT so a slug present in >1 granted source unions tags instead of throwing on a multi-row subquery. - getLinks/getBacklinks: federated branch scopes ALL THREE page endpoints (from, to, AND the authoring origin) so a cross-source link can't disclose a foreign slug. Scalar/unscoped branches unchanged (trusted internal callers keep the cross-source view). - getTimeline: Postgres 8-branch cartesian tree collapsed to one fragment-composed query; PGLite adds the sourceIds branch to its dynamic WHERE. * fix(ops): route by-slug reads through the federated source scope (#2200) get_page resolves tags against the concrete page's source; get_tags/get_links/ get_backlinks/get_timeline route through sourceScopeOpts(ctx) (replacing the copy-pasted scalar `ctx.sourceId ? {sourceId} : {}`). New linkReadScopeOpts promotes an UNTRUSTED remote scalar scope to sourceIds[] so legacy/pre-federated tokens also get all-endpoint link scoping; trusted local CLI keeps cross-source. * test: federated read scope on by-slug reads + engine parity (#2200) Per-op federated reads, isolation (out-of-grant -> empty), cross-source decoy guard, far-endpoint + origin leak guards (F1), same-slug union (D3A), empty-array contract, scalar-remote promotion (D1), getTimeline date-window after the Postgres fragment refactor (D5A), and engine-parity arms for all four methods. * chore: bump version and changelog (v0.42.46.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): document federated by-slug read scope + linkReadScopeOpts (#2200) The #2200 fix routed get_page tags + get_tags/get_links/get_backlinks/ get_timeline through the federated source scope and added sourceIds[] to the engine read methods + TimelineOpts. Bring KEY_FILES.md to current state: the operations.ts entry's sourceScopeOpts read-op list now includes the by-slug reads and documents the linkReadScopeOpts helper (three-endpoint link scoping + untrusted-remote scalar promotion); the engine.ts entry notes the by-slug read methods + TimelineOpts carry the same sourceIds[] federated axis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
5c49225e4b |
v0.42.45.0 feat(sync): delta-aware cost estimator — stop wedging the daily cron (#2139) (#2224)
* feat(core): shared computeSyncDelta + spend-posture module (#2139) sync-delta.ts: ONE implementation of "what changed since last_commit", consumed by both the sync executor and the inline cost estimator so the gate's dollar figure can't drift from what the sync imports. spend-posture.ts: spend.posture config + parseUsdLimit/formatUsdLimit off-switch parsing (off/unlimited/none → Infinity; undefined at the budget boundary so ledger rows never serialize null). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sync): delta-aware cost estimator + non-TTY auto-defer + per-source failure acks (#2139) The inline-embed cost gate was a ~400x phantom: it priced the entire tree whenever the working tree was dirty (always, on an active brain), then blocked the daily cron with exit 2. Now: - performSyncInner + the estimator both route through computeSyncDelta, so the estimate mirrors execution (fetch-first delta; dirty-but-caught-up tree → $0). - shouldBlockSync is posture-aware; non-TTY above floor AUTO-DEFERS embeds to capped backfill jobs (exit 0) instead of wedging — single shared runInlineCostGate on both --all and single-source paths. - --full prices delta + stale backlog (full sync sweeps it inline). - off/unlimited on the cost knobs; tokenmax bypasses the backfill cap (still ledgered) but never the cooldown. - --skip-failed/--retry-failed scoped per source; the D15 parallel refusal is lifted (the #1939 ledger is per-source + lock-serialized). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(config): register spend-control keys + validate spend.posture (#2139) Adds spend.posture + the five previously --force-only spend knobs to KNOWN_CONFIG_KEYS so `config set` accepts them directly (removes the archaeology the issue complained about), and rejects invalid spend.posture values at set time with a paste-ready hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reindex,enrich,onboard): spend.posture across the remaining cost gates (#2139) reindex-code: tokenmax makes the cost gate informational; --max-cost accepts off/unlimited. enrich + onboard --auto: tokenmax lifts the refuse-without-cap guardrail and runs UNCAPPED (spend still ledgered by BudgetTracker). Explicit --max-usd always wins over posture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cost-gate, delta estimator, spend-posture, off-switch coverage (#2139) New sync-delta + sync-cost-estimate unit suites; rewritten cost-gate serial tests (auto-defer instead of exit 2, posture, off-switch, format split, single-source); parseUsdLimit/posture-aware shouldBlockSync; backfill cap-off + tokenmax-bypass + cooldown-still-refuses; config known-key acceptance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spend-controls): single spend-control surface + ref-map + follow-up TODOs (#2139) New docs/operations/spend-controls.md (every gate, key, default, off switch, posture interaction); CLAUDE.md reference-map row; two P3 follow-up TODOs (measured chunk-count gating, per-source defer granularity). llms bundles regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(spend): SSRF-harden estimator fetch + complete off/uncapped across reindex/enrich/onboard (#2139) Ship-stage codex pre-landing review caught four P1s in the secondary cost gates: - The delta estimator's fetch-first ran `git fetch` through the plain git() helper, bypassing the GIT_SSRF_FLAGS + GIT_TERMINAL_PROMPT=0 hardening that real sync uses. Added `fetchRemote()` to git-remote.ts (same flags as pullRepo) and route the estimator through it — a cost preview / dry-run can no longer hit a remote through a less-protected path. - `reindex --max-cost off`, `enrich --max-usd off`, `onboard --auto --max-usd off` were parsed but didn't actually proceed/uncap. Now: explicit off (and spend.posture=tokenmax) proceed past the confirmation/missing-cap refusal AND run uncapped. enrich threads an Infinity sentinel mapped to "no BudgetTracker ceiling" (never raw Infinity → no null in audit rows); reindex/onboard use their native undefined=uncapped path. Spend still ledgered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.45.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(KEY_FILES): update sync/embedding/git-remote/reindex entries to post-#2139 truth document-release pass: the cost-gate entries described the pre-#2139 behavior (full-tree-ceiling estimator, --skip-failed-rejects-under-parallel, exit-2 confirmation gate). Updated to current truth — delta-aware estimator via the shared computeSyncDelta, per-source failure acks under parallel, non-TTY auto-defer (no exit 2), posture-aware shouldBlockSync. Added entries for the two new core modules (sync-delta.ts, spend-posture.ts) + fetchRemote on git-remote.ts + reindex --max-cost off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a81f7e05e8 |
v0.42.43.0 feat(context): push-based context (#2095) + teardown-exit hardening (#2084) (#2175)
* fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084)
Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering
sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the
bound raced past — keep Bun's event loop alive, so every `gbrain query` paid
a flat 10s tax exiting via the hard-deadline force-exit banner.
Three changes, one contract:
- flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the
command is not a daemon, exit deliberately — after stdout AND stderr drain
(writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a
blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the
regression class; pinned by a 256KB real-pipe subprocess test.
- drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites
(op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation
x3, ze-switch, dream, read-only timeout path). Drains the background-work
registry, then disconnect (best-effort), bounded by the 10s unref'd
hard-deadline — which is now armed around the TEARDOWN window only, not
before the op handler (the old placement would have force-killed any op
slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites
previously skipped the drain entirely and had no hang timer at all.
- Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing
paths (status, friction, claw-test, smoke-test, eval cross-modal /
takes-quality replay / conversation-parser / whoknows-thin, status-thin)
become process.exitCode + return so they flow through the drains and the
flush-exit. Pre-engine usage/parse/refusal exits stay as-is.
BrainRegistry.disconnectAll deliberately unchanged: zero production callers
in src/, per-engine disconnects already bounded, and the kernel reclaims
sockets on exit (src/core/timeout.ts doctrine).
DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084)
Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs
verified only against one production deployment — CI had no transaction-mode
pooler and could never see the class. Now it can:
- docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting
postgres-1, mirroring the production split-pool topology (direct :5432 +
pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext
password in the userlist) + IGNORE_STARTUP_PARAMETERS for the
statement_timeout/idle_in_transaction_session_timeout startup params
gbrain's client sets (the Supabase pooler whitelists the same).
- test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url
into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs),
then spawns the real CLI against the POOLED url and asserts: exit 0,
stdout intact (the #1959 truncation class), and NO
"did not return within 10000ms — force-exiting" banner (pre-#2084 it
printed on 100% of query-shaped ops on this topology). Class bound, not
exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL.
- scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL +
GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases.
Verified live: both tests green against pgbouncer 1.25.2 in transaction mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095)
One row per page the brain volunteers (op / reflex / watch channels).
"Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at
(the existing bumpLastRetrievedAt write-back is the open/cite signal), so
there is no second tracking path. session_id/turn are nullable
caller-supplied attribution; rationale is a deterministic template string,
never raw conversation text.
- Migration v116 (idempotent) + mirrors in src/schema.sql +
src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also
folds in pre-existing comment-only drift from the v114 links edits).
- src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE
multi-row parameterized INSERT — never per-row awaited round-trips) +
purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains).
- Dream cycle purge phase prunes stale events alongside op_checkpoints /
brainstorm checkpoints / batch-retry audit files.
- RLS on Postgres comes from the v35 auto_rls_on_create_table event
trigger (the same mechanism that covered v110 page_aliases and v115
op_checkpoint_paths); the volunteer Postgres e2e pins it.
- No ::jsonb anywhere; no bootstrap probe needed (nothing references the
table pre-creation; writers guard with try/catch).
Tests: v116 shape + columns + indexes + live insert/purge round-trip on
PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095)
- entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing
per-turn extractor across the last N turns (oldest→newest), merges by the
normalizeAlias form with occurrence/newest-turn/user-mention metadata, and
orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES
cap drops stale assistant chatter, not the entity the user just named.
Closes the filed assistant-introduced-entities recall TODO; true pronoun
coreference (never-named antecedents) stays out of scope.
- retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence +
matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6)
lives next to the arm definitions so identity and score can't drift.
Arm-2 provenance is classified in JS (codex D8 — the combined OR can't
report which predicate matched). Federated sourceIds[] scope (alias arm
loops per source; arm 2 uses source_id = ANY — no engine-interface
change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for
windowing): the legacy title-whole-word rule would suppress every entity
merely MENTIONED in a prior window turn, breaking the feature by
construction — slugs only enter context when a pointer/page was actually
surfaced. Default stays 'slug-and-title' for the window=1 legacy path.
- volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF,
unprefixed → one user turn), volunteerContext (zero-LLM: extract →
resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate →
cap 3/5; deterministic rationale strings, never raw conversation text),
and volunteerUsageStats (per-arm/channel precision from the
last_retrieved_at join, labeled approximate — 5-min throttle false
negatives, unrelated-read false positives; codex D9).
Tests: 35 green across volunteer-context (window parsing, pronoun follow-up
via assistant-introduced entity, confidence gating, slug-only suppression,
takes-fence privacy, multi-source scope, caps, stats join math) +
retrieval-reflex back-compat + resolve-ipc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095)
New read-scope op on the contract surface (CLI `gbrain volunteer-context`
with stdin → window, MCP tool for free): takes a rolling conversation
window, returns confidence-gated page pointers with rationales + synopses.
`window` is optional-unless-stats (validated in the handler, codex D9);
`stats: true` returns the volunteered-vs-used precision summary, labeled
APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both
bias the join). Source scope threads through sourceScopeOpts — federated
grants narrow the volunteer to the granted sources.
Event logging is fire-and-forget through a new `volunteer-events`
background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked
dangling promise set + bounded drain + snapshot-drop on timeout so a
long-lived process never accumulates ghosts). ONE batched INSERT per call,
drained on every exit path by the commit-1 drain hoist; failure never
fails the op (pinned by an injected failing-engine test).
cli formatResult renders both shapes (pointer lines with confidence/arm/
rationale; the stats summary with per-arm precision).
Tests: op contract surface, window-required validation, sink round-trip
with session_id/turn attribution, failing-engine fail-open, federated
grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e
proving the op + sink + stats join AND that context_volunteer_events has
RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): reflex consumes the rolling window + ambient-channel logging (#2095)
The default-on retrieval reflex now extracts entities from the last N turns
(retrieval_reflex_window_turns, default 4; env
GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy
current-turn-only behavior exactly). assemble() passes the recent
user/assistant turns (hard cap 12); the reflex slices to the configured
window. Assistant-introduced entities and "what did she invest in?"
follow-ups whose antecedent was NAMED in the window now surface pointers —
the issue's "zero agent-initiated queries" success criterion on the
ambient path.
Under windowing, suppression switches to slug-only (codex D7): the legacy
title-whole-word rule would suppress every entity merely MENTIONED in a
prior window turn, breaking the feature by construction. Slugs only enter
prior context when a pointer/page was actually surfaced, so
already-surfaced pages still suppress. The suppression mode flows through
all three resolver rungs (host opts, serve IPC request, direct Postgres).
Ambient-channel feedback (codex D11): the server-side resolver paths
(serve IPC + direct Postgres) log volunteered pointers with
channel: 'reflex' through the drained volunteer-events sink, so
`gbrain volunteer-context --stats` measures the default-on path where most
volunteering happens. Host-injected resolvers (no gbrain engine) can't
log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the
pointer cap are unchanged.
Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only
vs already-surfaced suppression; throwing resolver stays fail-open
(16 green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): gbrain watch — push transport over stdin (#2095)
The issue's headline: the brain volunteers pages as the conversation flows,
instead of waiting to be asked. `some-transcript-feed | gbrain watch` reads
turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed
lines are user turns), keeps a rolling window (--window-turns, default 4),
and streams confidence-gated pointers with rationales to stdout (--json for
JSONL). Session dedupe rides the core's slug-only suppression — a slug is
volunteered at most once per session. Events log on channel 'watch' with
session_id + turn through the drained sink.
Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an
interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF —
so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1
placeholder): when main() resolves the work is over, the CLI_ONLY finally
drains volunteer events via drainThenDisconnect, and the entrypoint
flush-exit ends the process. Keeping it in the daemon set would have made
the piped EOF path hang on lingering sockets — the exact #2084 class.
SIGINT closes the stream and flows through the same drain path instead of
killing mid-write. Per-turn resolution failures are fail-open (the stream
never dies on a transient DB error).
Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) +
THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP
op) + main --help entry.
Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling
window via assistant-introduced entity, session dedupe, --json shape with
turn attribution, channel-watch event rows, --min-confidence gate, CRLF/
blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch`
on a fresh PGLite brain exits 0 at EOF with no force-exit banner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave
- docs/architecture/KEY_FILES.md (current-state): context entries gain the
window extractor, arm provenance/confidence, suppression modes, volunteer
+ volunteer-events modules; background-work entry now lists FIVE sinks and
the drainThenDisconnect owner-disconnect contract; new entries for
src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
the confidence model, CLI usage, config keys, and the approximate-stats
caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
check, structured messages[] param); the #1981 entity-detection TODO
narrowed (window extraction covered assistant-introduced entities +
named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
#2084 wave.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): truncate context_volunteer_events in setupDB (#2095)
The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows
persisted across e2e runs on a reused database and poisoned count/stats
assertions in volunteer-context-postgres on the second run. No FK to pages
(slug join), so position before pages is for hygiene only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084)
Caught by the full unit suite: `gbrain apply-migrations` on PGLite started
exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM
backend's proc_exit status into process.exitCode (initdb at create-time,
the postmaster at close-time — `exitCode=status` in pglite's dist), and
the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around
create/close (a guarded attempt verified this). The pre-#2084 success path
never read process.exitCode, so the pollution was invisible; the new
deliberate flush-exit propagated it faithfully.
Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in
cli-force-exit.ts — every gbrain-owned exit-code assignment routes through
the setter (still mirrored to process.exitCode for outside readers), and
both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect
hard-deadline backstop) read the getter. Swept all assignment sites:
cli.ts (op error, friction, claw-test, smoke-test, eval runners, status,
import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot.
Also updates the v0.42.20 structural pins to the drainThenDisconnect shape
(ordering invariant asserted INSIDE the helper + >=8 helper call sites,
superseding the two-inline-pairs assertion).
Verified: apply-migrations spawn test green; `init --migrate-only` exits 0;
an errored op still exits 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: re-pin the teardown-arming invariant at its post-#2084 home
Master's v0.42.41.0 triage wave and the #2084 wave fixed the same
pre-armed-timer bug independently; the merge keeps #2084's shape (arming
inside the shared drainThenDisconnect helper, covering all 8 exit paths).
The structural pin now asserts the same invariant — no pre-try arming;
gated, unref'd, before-drain, cleared — at the helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: coverage for ambient reflex-channel logging + watch window/cap flags
Ship coverage audit (85%, gate PASS) named five gaps; the two substantive
cheap ones close here: the codex-D11 logChannel='reflex' path now has a
behavioral pin (events land on channel 'reflex' through the drained sink;
no logChannel → no events), and gbrain watch's --window-turns / --max-pages
flags are exercised (turn-1 attribution under window=1; cap to one page).
Remaining flagged-not-blocking: the wallclock-timeout branch (untestable
without >10s real-clock flake — same rationale as the arming pin),
formatResult's volunteer case (module-private), and the cycle purge wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT
formatResult exported for tests (same import-safety contract as cliAliases);
test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message,
and approximate stats summary. test/watch-command.test.ts gains a real
subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream,
assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under
the actual signal, not just the shared exit path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin
The merged-state suite caught it: doctor --fast --json reported FAIL but
exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes
(doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit
silently zeroes — getCliExitCode() deliberately never reads ambient
process.exitCode (the PGLite-Emscripten pollution defense), so any setter
that bypasses setCliExitCode reports success on failure.
Swept both sites and added the structural class pin: a test greps src/ for
raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that
introduces one fails loudly instead of lying about exit codes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.43.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: quarantine the watch SIGINT subprocess test to the serial lane
The parallel unit shards flake on concurrent CLI subprocess spawns (failed
at 7ms in-suite, green solo) — same isolation rationale as
apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: update project documentation for v0.42.43.0
Post-ship doc verification against the release diff (#2095 push-based
context + #2084 superset hardening), with a cross-model doc review:
- push-context.md: version tag corrected to v0.42.43.0; per-call knobs
now cover prior_context/days and watch's flag surface accurately;
feedback-log writes described as best-effort; synopsis fence-strip
described as unconditional.
- CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context
release reference corrected to v0.42.43.0.
- KEY_FILES.md: ci-local entry rewritten to current topology (4-shard
parallel default, four Postgres services, transaction-mode PgBouncer
+ GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts
dropped from the selector entry.
- TESTING.md: inventory entries for the new #2084 structural pins
(cli-exit-verdict-pin, cli-pipe-truncation), the push-context test
suite (volunteer-context, watch-command, watch-sigint.serial,
cli-format-volunteer), migrate v117 coverage, and the two new E2E
files (pgbouncer-teardown env gating, volunteer-context-postgres RLS
pin); check:all row corrected (not a superset of verify).
- AGENTS.md + RELEASING.md: ci:local descriptions updated to the
sharded + pooler topology.
- CHANGELOG (wording only, entry preserved): "retrieved" instead of
"opened" for the used-signal, pooler scoped to the local CI gate,
feedback log labeled best-effort.
- llms-config.ts: index the new push-context guide; bundles
regenerated (build:llms) and freshness test green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(test): correct the v116 reference — the table shipped as migration v117
* fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095)
Five specialist reviewers (testing/maintainability/security/performance/
data-migration) on the reconciled diff; every finding applied:
Performance: the alias arm now resolves all granted sources CONCURRENTLY
(a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources
cross-region, inside the reflex's 1.5s budget); watch's session dedupe is
O(1) Set membership instead of a monotonically growing priorContext string
(O(T²) over a long-lived session); getWindowTurns iterates from the tail
(per-turn cost no longer grows with session length); the resolver's
provenance maps fold into the existing candidate pass.
Security: volunteer_context clamps caller-supplied attribution at the trust
boundary — session_id capped at 256 chars (a read-scoped token could bank
~1MiB TEXT per request, retained 90 days), turn logged only when a safe
integer (a non-integer threw inside the batched INSERT and silently dropped
the whole batch). The privacy comments now state precisely what rationale
may contain (the matched entity's surface form — which by construction
resolved to an existing alias/title/slug — never free conversation text).
Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from
volunteer.ts and shared by watch/cli (the two surfaces can no longer
drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly
site for all three channels; watch's window default now honors the same
retrieval_reflex_window_turns config knob the reflex reads; the stale
pre-v116 comments swept to pre-v117.
Testing: the two flake-class CRITICALs fixed (pipe test asserts the
backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test
waits on watch's new machine-readable ready line instead of a fixed 15s
sleep — 2.5s and deterministic now); new coverage for the sink's timeout
branch + ghost-reference drop, watch per-turn fail-open, untrusted knob
clamps (min_confidence/max_pages/days), window-cap ordering (newest user
mention survives), serve-IPC suppression passthrough + channel=reflex
logging, windowTurnCount edge semantics, and structural pins for the sink
registration + cycle purge wiring. The exit-verdict pin's grep is now
operator/whitespace-tolerant.
Deferred with TODOs: resolver index shapes for the per-turn query;
batched first-prune after a long dream-cycle gap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp
Four red-team findings on the #2095 push-context surface:
- RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's
cap, so a recurring already-pushed entity burned cap slots every turn and
starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside
the pointer loop BEFORE the confidence gate and the cap.
- RT3 honest stats: reflex-channel event logging moved from inside the
resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires
only after the response write succeeds, and buildReflexAddition logs only
after the per-turn timeout admits the block. A block the client's 250ms
budget abandoned was never injected and no longer counts as volunteered.
(logChannel resolver opt removed; logDeliveredReflexPointers is the seam.)
- RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo
can't reintroduce the re-scan-everything-per-turn cost class.
- RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP,
push-context guide, TODO to route watch via serve IPC); host-resolver
suppression contract at ResolveEntitiesFn (TODO for a capability gate).
Tests: starvation guard (watch + volunteerContext unit), window clamp floor +
ceiling, delivery-side logging (helper writes channel=reflex through the
drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test
rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims
Three CI-only check failures, two root causes:
1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when
loadConfig() returned null (no config file AND no DATABASE_URL — a clean
CI shard with no brain). loadConfig drops its env→config mapping in that
case, so the documented escape hatch silently died and the window fell
back to 4 → windowed extraction widened when the test set window=1 →
prior-turn entity leaked. Fixed: read the env var directly in
windowTurnCount, mirroring reflexEnabled's direct process.env read. This
is a real product bug, not just a test artifact — any config-less host
using the env hatch was affected. Regression test pins it.
2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a
sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway
and never reset it. The legacy-embedding preload only restores the
OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign
config survives into the next file — and a file's beforeAll runs BEFORE
the preload's restoring beforeEach, so federation-health built a
vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new
test files reshuffled the deterministic file→shard assignment, exposing
this latent ordering bug. Hardened both victims to establish the gateway
state they assert (sync-cost-preview resets to the unconfigured fallback;
federation-health pins legacy 1536 before initSchema) so they're
order-independent. Verified against a simulated leaker run before them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(context): use withEnv() in the window env-hatch test (test-isolation guard)
The regression test added in
|
||
|
|
4ee530f3c5 |
v0.42.42.0 fix(cli): bounded teardown + explicit exit — kill the 10s force-exit tax on txn-mode poolers (#2084) (#2141)
* feat(core): finishCliTeardown + flushThenExit — bounded teardown, owned exit verdict (#2084) cli-force-exit.ts becomes the single owner of one-shot CLI exit + teardown: - finishCliTeardown: bounded sink drain -> bounded disconnect under a backstop whose deadline is COMPUTED from the bounds it guards (floor 10s; GBRAIN_TEARDOWN_DEADLINE_MS env override). Arms at teardown start, never before the op handler. - flushThenExit: stdio write-fence (unref'd guard, EPIPE-safe) + REF'D aliveness grace for non-TTY stdio — Bun only delivers queued pipe writes while the process is alive (no flush API reaches the native queue). - setCliExitVerdict/currentExitCode: the exit verdict lives in a gbrain-owned channel, never read back from process.exitCode (PGLite's Emscripten runtime scribbles its own status there mid-run). - background-work.ts exports backgroundWorkSinkCount() for the deadline formula. Unit tests + a spawned-Bun harness proving byte-complete piped output. * fix(cli): route all nine disconnect sites through finishCliTeardown; one exit seam (#2084) Deletes the pre-handler 10s force-exit timer (it measured handler + teardown combined: PgBouncer txn-mode deployments paid a flat 10s banner tax on every query, and any >10s op was killed mid-run with exit 0 and truncated output). Sweeps op-dispatch, CLI_ONLY fall-through, search dashboard, read-only timeout path, dream, doctor x3 (fixing a pre-existing pool leak when DB checks throw), and ze-switch. The ONE process exit lives in main().then/catch via flushThenExit(currentExitCode()), gated by shouldForceExitAfterMain(). Exit-code writers (op-dispatch catch, reindex, transcripts, brainstorm, autopilot, frontmatter) now set the verdict through setCliExitVerdict. * fix(pglite): contain Emscripten's process.exitCode writes at PGlite.create (#2084) PGLite's WASM runtime writes its own status into process.exitCode (99 at create; in-memory brains run initdb whose status lands on a later tick; the exit status at close) — on PGLite every error exit was silently clobbered. preservingProcessExitCode wraps create() to keep the global tidy; db.close() stays unwrapped (its 0-write is baseline behavior test runners depend on). The CLI verdict itself is immune: it lives in the owned channel. * test: e2e + structural pins for the #2084 teardown contract E2E: failed op exits 1; every swept command spawned (brain-copy isolation for mutators, no-network); slow-handler regression via the deadline env knob; piped --json parses complete; teardown banner absent on every happy path; daemon survival untouched. Structural: no bare awaited engine disconnects in cli.ts; DISCONNECT_HARD_DEADLINE_MS gone; >=9 helper call sites; verdict channel + create-wrap pins. * test: fix R1 env-isolation violations in retrieval-reflex tests Pre-existing on master: both files mutated GBRAIN_RETRIEVAL_REFLEX directly, failing scripts/check-test-isolation.sh (bun run verify). Converted to the canonical withEnv() pattern; the reflex describe's beforeEach also never restored the flag, leaking it across the shard. * docs: KEY_FILES entries for the teardown contract; close + file TODOS (#2084) KEY_FILES.md: current-state entry for cli-force-exit.ts (helper + central exit seam pair, verdict channel, cli.ts-scoped claim); background-work.ts and pglite-engine.ts entries updated. TODOS.md: the drain-before-owner-disconnect P3 (filed from #1972) is done by this wave; files the trigger-gated GBRAIN_COMMAND_DEADLINE_MS follow-up (eng-review D2/D14). * fix: pre-landing review fixes (#2084) Review army (testing/maintainability/security/performance, 0 critical): - drain defense-in-depth: a throwing drain warns and still disconnects (cannot escape a caller's finally or skip the engine teardown) - behavioral tests for preservingProcessExitCode (connect pins 0; create-throw restores the pre-call verdict) - D9 widening test (live-registry sink count feeds the deadline formula), env 0/negative boundary cases, verdict mirror-write assertion - stale comments: header diagram backstop line, structural-test 'both lifecycle calls' contradiction, KEY_FILES 10s-force-exit clauses, e2e D11 falsification story corrected - named the formula's pool-end literals * fix: adversarial-review hardening — daemon-safe command resolution, flush knob, ref'd backstop (#2084) Cross-model adversarial review (Claude subagent + Codex, both P1'd it): - shouldForceExitAfterMain now resolves the command through parseGlobalFlags — the old first-non-dash heuristic read `gbrain --timeout 30s serve` as command "30s" and the new exit seam would have killed the daemon ~250ms after boot with exit 0 (unit-pinned) - GBRAIN_FLUSH_GRACE_MS env override for the non-TTY aliveness grace (batch consumers piping large payloads to slow readers can raise it; agent loops can lower it) - backstop timer is now REF'D: a hung teardown on an otherwise-empty event loop previously exited naturally — skipping the flush and surfacing PGLite's scribbled process.exitCode - flushThenExit: real process.exit latched once per process - doctor-site comment corrected; in-command process.exit teardown-bypass class (pre-existing) filed as a P2 TODO * chore: bump version and changelog (v0.42.42.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: move #2084 exitCode-containment lifecycle tests to the serial quarantine (R3) * docs: update project documentation for v0.42.42.0 - docs/TESTING.md: replace the stale 4-file serial-quarantine enumeration with a current-state description (the quarantine is glob-discovered, now several dozen files incl. the #2084 exitCode-containment suite); add unit inventory entries for test/cli-finish-teardown.test.ts and test/flush-then-exit-harness.test.ts. - docs/architecture/KEY_FILES.md: rephrase the pglite-engine exitCode containment note to current-state wording (clears the check-key-files-current-state prose-history warning). llms bundles regenerated (byte-identical: both docs are link-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply cross-model doc-review findings for v0.42.42.0 Codex review of docs-vs-shipped-code found 9 gaps; all verified against the code before fixing: - CHANGELOG.md (0.42.42.0 entry, precision narrowing only — no entries touched): "every CLI exit path" -> "every cli.ts disconnect site"; "on every path" -> "on every routed exit path"; dream/doctor/ze-switch claim scoped to dispatcher teardown (command-internal process.exit sites are tracked in TODOS as the open P2). - docs/architecture/KEY_FILES.md: the teardown backstop is REF'D, not unref'd (matches the F3 adversarial-review decision in the code). - src/core/cli-force-exit.ts: header diagram comment had the same stale unref'd claim + `process.exitCode ?? 0`; now matches the implementation (ref'd timer, `currentExitCode()`). Comment-only change. - docs/TESTING.md: verify is the 30-check parallel battery via run-verify-parallel.sh (was described as 4 checks); CI is 10 weighted LPT shards + dedicated verify/serial/slow jobs (was "4-way FNV on shard 1"); test:serial runs one bun process per file (not --max-concurrency=1); dead "cap: 10" line rewritten as debt guidance; inventory entries added for test/cli-should-force-exit.test.ts and test/e2e/pglite-cli-exit.serial.test.ts. bun run verify green (30/30); #2084 test files green; llms bundles regenerated (byte-identical — reference docs are link-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route v0.42.41.0's raw exitCode writers through the verdict channel; reconcile merged structural pins (#2084) CI fallout from merging the v0.42.41.0 triage wave into the #2084 exit-seam design — both waves fixed the same timer-placement bug independently: - doctor.ts + extract.ts set failure exit codes via raw `process.exitCode =` writes (v0.42.41.0's process.exit -> exitCode conversion); the #2084 exit seam reads only the gbrain-owned verdict channel, so doctor FAILs exited 0 (Tier 1 RLS e2e + half-migrated-Minions tests). Converted to setCliExitVerdict, same as the wallclock-124 site in the merge commit. - cli-force-exit-teardown-arming.test.ts pinned v0.42.41.0's inline finally-armed timer, which the merge replaced with finishCliTeardown; rewritten to pin the merged invariant (no pre-try arming in cli.ts; the backstop arms inside the helper before the drain). - eval-capture drain timing bound 1s -> 2s: flaked at 1023ms under CI shard load after the new test files shifted LPT shard packing (13x budget slack still proves bounded-not-hung). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
+1
|
7c27fa129b |
v0.42.41.0 fix: triage wave — 6 data-loss/availability fixes + 9 community PRs (#2128)
* fix(oauth): default omitted authorize scope to client's full grant When a client omits `scope` on /authorize, the authorize() grant computed `(params.scopes || []).filter(...)` → the empty set. That empty grant was written to oauth_codes and propagated into the access AND refresh tokens, so every request failed `insufficient_scope` even though the client was registered with e.g. `read write`. Because refresh inherits the stored grant, it never self-healed — reconnecting just minted another empty-scoped token. Some MCP connectors (observed with Claude Desktop) omit `scope` on /authorize, so they hit this on every connection. Fix: when no scope is requested, default to the client's full registered scope (RFC 6749 §3.3 permits a server default). This mirrors exchangeClientCredentials, which already does `requestedScope ? ... : allowedScopes`. The result is still clamped to the allowed set, so an explicit over-broad request cannot escalate. Adds test/oauth-authorize-scope-default.test.ts covering: omitted/empty → inherits full grant; explicit subset honored; clamp preserved (over-broad and disallowed-only requests cannot escalate or trigger inheritance). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): skip Python venv/ in the code walker collectSyncableFiles (first-sync walker) and the incremental PRUNE_DIR_NAMES set skipped node_modules but not Python venv/. On a Python repo the walker descended into venv/ (thousands of files); the resulting slug collisions crashed putPage's INSERT ... ON CONFLICT ... RETURNING with "undefined is not an object (evaluating 'row.deleted_at')". Add `venv` alongside node_modules in both the import.ts inline skip and PRUNE_DIR_NAMES. venv is the Python equivalent of node_modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): carry asymmetric input_type across the AI SDK to the wire body (#1400) dimsProviderOptions() threads input_type ('query' | 'document') into providerOptions.openaiCompatible for asymmetric models (ZE zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter validates providerOptions against a fixed schema and silently drops the field before building the HTTP body. Every embedQuery() was therefore encoded document-side: the ZE shim's hard default fired ('document'), Voyage and local openai-compat servers got no input_type at all, and asymmetric retrieval silently collapsed toward surface-token overlap — while the providerOptions-level contract test stayed green. Fix: an AsyncLocalStorage (same pattern as __budgetStore) populated in embedSubBatch() only when providerOptions actually threads an input_type, read at body-rewrite time by the fetch shims: - zeroEntropyCompatFetch: recovers the threaded value; document default preserved for ingest paths. - voyageCompatFetch: opt-in like the dims.ts Voyage branch — inject only when threaded; the field stays off the wire otherwise. - NEW openAICompatAsymmetricFetch: fallthrough default for every other openai-compatible recipe (llama-server, litellm, ollama, ...) — the canonical local/proxy paths for asymmetric models. Strict pass-through when nothing was threaded, so symmetric deployments see zero wire change; recipes with their own compat fetch (azure) keep it via the compat.fetch ?? precedence. KNOBS_HASH_VERSION bumped 10→11: cached query_cache rows were keyed on document-side query vectors; pre-fix rows must not be served to post-fix lookups (same convention as the v=3 embedding-provider bump). One-time global cold-miss on upgrade; refills within cache.ttl_seconds. Tests: test/embed-input-type-wire.test.ts runs the REAL SDK transport with a mocked global fetch and asserts on the outbound body — the only layer where this regression is observable. Covers ZE hosted, llama-server, litellm, ollama (query + document sides) and pins the pass-through for non-asymmetric models and Voyage's opt-in shape. 4 of the original 7 assertions fail on master, proving the pin. One structural pin in test/ai/zeroentropy-compat-fetch.test.ts updated to the new line shape (same semantic); KEY_FILES.md gateway.ts entry updated to the new truth. Supersedes #1400 (closed unmerged) — same ALS mechanism, extended to Voyage + all openai-compatible recipes. Credit to @billy-armstrong for the original diagnosis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sync): honor .gitignore in code walk; prune vendor/dist/build collectSyncableFiles (the full-sync / dry-run enumerator) reimplemented its own directory skip list inline (node_modules || ops), bypassing the canonical pruneDir gate and ignoring .gitignore entirely. On a Laravel/PHP repo this descended into vendor/ (~50k Composer files), storage/, and public/build/, trying to import 52k dependency/build files and flooding the index with library internals (a 35-min sync that never finished, killed by the watchdog at 3%). - collectSyncableFiles now enumerates via `git ls-files --cached --others --exclude-standard` when dir is a git work tree, so the walk honors .gitignore (tracked + untracked-not-ignored). Falls back to the FS walk for non-git dirs. EroLab: 52164 -> 1028 files. - The FS fallback now prunes through the canonical pruneDir() instead of a drifted inline list, so the two skip lists can't diverge again. - PRUNE_DIR_NAMES gains vendor/dist/build (dependency + build-output trees). Addresses #1483 (.gbrainignore), #1159 (--respect-gitignore), and the maintainer's #1942 vendor/dist/build prune. Walker regression suites (sync-walker-symlink, brain-writer-walk-prune, sync, sync-walker-submodule) green: 90 pass. * fix(config): ignore DATABASE_URL auto-loaded from cwd .env (#427) Bun merges .env files from the process cwd into process.env before any user code runs. loadConfig() prefers env DATABASE_URL over ~/.gbrain/config.json, so any gbrain invocation from inside a web-app checkout silently retargets the brain at that app's database — reads go to the wrong DB and apply-migrations can write gbrain's schema into a production app database (#427). effectiveEnvDatabaseUrl() re-parses the .env files Bun auto-loads from cwd and treats a DATABASE_URL whose value matches one of them as file-origin: ignored, with a one-time stderr notice. GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs are honored unchanged, so the operator escape hatch and the e2e suite's env-provided URL keep working. Applied at loadConfig, getDbUrlSource (doctor parity), init --non-interactive, and migrate --to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): arm the disconnect hard-deadline at teardown entry, not before the op body The 10s force-exit timer in the shared-op dispatch was armed BEFORE the try block, so any op whose handler ran past 10s wall-clock was killed mid-flight with process.exit(0) and zero stdout. On a slow Postgres pooler (6-10s per fresh connection) a healthy `gbrain search` was force-exited every time — an empty 'success' indistinguishable from no results. The v0.42.20.0 exitCode honor can't help: a mid-op kill fires before any error path sets exitCode. Move the arming into the finally (teardown entry), matching the fall-through owner-disconnect site later in main(): the timer still bounds a hung drain/disconnect (the C13 contract) but can no longer kill a slow-but-progressing op. Verified on a transaction-pooler Supabase brain: search went from 0 bytes/exit 0 at 10s to real results at ~21s. * fix(import): stamp source_id on extracted call-graph edges importCodeFile built CodeEdgeInput rows without source_id, so every edge landed NULL. getCallersOf/getCalleesOf filter `AND source_id = <scoped>` whenever a worktree pin or --source is in play — NULL never matches, so scoped call-graph queries silently returned 0 rows on multi-source brains even though the edges existed (2,122 edges, 26 targeting the probed symbol, count 0 returned). One-line fix: carry the sourceId already in scope into the edge input. Existing NULL rows backfill with: UPDATE code_edges_symbol e SET source_id = p.source_id FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE c.id = e.from_chunk_id AND e.source_id IS NULL; (same for code_edges_chunk). Verified: code-callers returns 21 callers where it returned 0. * docs(migrations): NULL embeddings BEFORE the column-type alter The Postgres recipe ordered ALTER COLUMN TYPE vector(N) before the UPDATE that clears stale embeddings. pgvector refuses to cast existing vectors across dimensions ('expected 1024 dimensions, not 1536'), so the recipe as written aborts the transaction on any brain that has embeddings — which is every brain doing this migration. Swap the steps: NULLs cast fine. * fix: honor legacy token source grants in oauth * fix(cli): bound read-scope op handlers at 180s wallclock (pre-landing review) With the hard-deadline timer correctly scoped to teardown, a genuinely wedged read handler (hung pooler connection mid-query) would hang the CLI forever — the #1633 zombie class the old pre-try timer accidentally bounded at 10s. Reads now get a generous withTimeout (180s default, far above any healthy slow-pooler run; --timeout=Ns overrides; exit 124 with the teardown finally still draining + disconnecting). Writes/admin stay unbounded: a long import/embed must never be killed by a default. * fix(import): stamp unscoped edges 'default', matching the pages-table default Review catch: 'sourceId ?? null' fixed the scoped path but left the unscoped one (reindex --code without --source, importCodeFile callers without opts.sourceId) stranding edges at NULL while their pages land under the schema default (pages.source_id DEFAULT 'default') — so getCallersOf(sym, { sourceId: 'default' }) missed them. Same bug, other door. Fallback is now 'default'. * fix(core): runtime dim-migration recipe NULLs embeddings before the alter Review catch: the doc fix corrected docs/embedding-migrations.md, but embeddingMismatchMessage still PRINTED the broken order — ALTER before UPDATE ... SET embedding = NULL — and linked to the now-contradicting doc. pgvector refuses to cast existing vectors across dimensions, so the printed recipe aborted on any brain that has embeddings. Swap the steps and say why inline. * feat(migrate): v116 — backfill NULL edge source_id + index from_symbol_qualified 1. Backfill: edges written before the stamping fix sit at source_id=NULL and stay invisible to scoped call-graph queries until repaired. Derive each edge's source from its own from_chunk's page (pages.source_id is NOT NULL DEFAULT 'default'). Same SQL verified live on a 2,122-edge production brain. 2. Indexes: getCalleesOf filters both edge tables on from_symbol_qualified, which had no index — every callee lookup was a seq scan, amplified per-BFS-node by the recursive code walk. With NULL edges repaired, scoped walks actually expand, so the latent cost becomes real. Mirrored into src/schema.sql; schema-embedded.ts regenerated. * docs(migrations): align the rationale list with the corrected recipe order The 'Why we don't do this automatically' list still said alter-then-wipe; reorder to wipe-then-alter and replace the fragile 'step 3' numeric cross-reference with a name-based one. * test: regression coverage for edge source_id stamping, timer placement, recipe order - import-code-edges-source-id: scoped import stamps edges + scoped getCallersOf/getCalleesOf match (verified failing pre-fix), plus the unscoped-import case asserting 'default' stamping. - cli-force-exit-teardown-arming: structural pin — the hard-deadline timer arms inside the finally (teardown entry), never before the op body; daemon guard, unref, clearTimeout intact. - embedding-dim-check: recipe order pinned — UPDATE precedes ALTER so the printed SQL can't drift from docs/embedding-migrations.md again. * fix(cli): hard-exit after teardown on wallclock timeout; bound makeContext too Adversarial review, two findings on the new timeout path: 1. On timeout the finally drained, disconnected, then CLEARED the hard-deadline timer — removing the only backstop while the abandoned handler (withTimeout races, it does not cancel) can hold ref'd sockets/SDK timers that keep Bun's loop alive: 'timed out' printed, process immortal — the zombie class this branch exists to kill, resurrected through its own fix. The finally now exits explicitly after teardown completes on the timeout path. 2. makeContext does DB I/O (resolveSourceId) for EVERY op and sat outside any bound — a pooler wedge at context build hung reads, writes, and admin alike. It now shares the same wallclock bound. * fix(import): normalize edge source once — closes the '' door and the unscoped chunk fan-out Adversarial review: txOpts used truthiness while the edge stamp used nullish — sourceId:'' put pages under 'default' but stamped edges '', FK-violating against sources(id) and silently dropping the file's whole call graph in the best-effort catch. The unscoped getChunks could also fan out to same-slug chunks from another source. One normalized edgeSourceId (sourceId || 'default') now drives both the chunk lookup and the stamp. * fix(engine): default edge source_id to 'default' at the insert layer (both engines) Adversarial review: addCodeEdges still wrote e.source_id ?? null, so any future caller that forgets the field reintroduces invisible NULL edges the day after the v116 backfill runs. A NULL source_id is invisible to every scoped call-graph query; default to the schema-default source the way the pages table does. Applied to both engines (parity). * fix(core): facts alter recipe NULLs embeddings before cross-dimension alters Adversarial review: buildFactsAlterRecipe shipped the same defect class this branch fixes for content_chunks 350 lines up — a cross-dimension ALTER ... USING cast that pgvector refuses while rows hold old-width vectors. Dimension changes now wipe first (the facts pipeline re-embeds on next write); same-dim type swaps (halfvec <-> vector) keep the lossless cast and PRESERVE data. Both behaviors pinned by tests. * v0.42.39.0 chore: version bump + CHANGELOG + TODOS Marks the v0.42.20.0 'decouple the op-dispatch force-exit timer' follow-up complete — this branch ships exactly that decoupling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(postgres-engine): atomic JSONB merge in updateSourceConfig — eliminate lost-update race ## Problem `updateSourceConfig` used a read-then-write pattern: read the current `config` row, normalize it in JavaScript, then write the merged result back with `SET config = <normalized> || <patch>`. Under concurrent callers (two background autopilot/cycle paths patching different keys simultaneously), both callers can read the same stale row. The later `SET config = ...` then clobbers the earlier patch, silently dropping whatever keys the first caller wrote. Reproduced at 21/25 lost-update events under real Postgres with parallel callers. ## Fix Fold the normalization and merge into a single atomic `UPDATE … SET config = CASE … END || patch` statement. Because the `SET` expression evaluates against the row-locked latest version of `config`, there is no snapshot window between the read and the write. Concurrent callers now converge correctly (50/50 clean in reproduction test). The `CASE` also normalizes historical bad JSONB shapes inline: - `object` — used as-is - `string` — double-encoded config; inner text parsed with the SQL `IS JSON` guard (Postgres 16+) so unparseable strings fall back to `{}` instead of raising `invalid input syntax for type json` - `array` — array of patch objects aggregated into a flat object via `jsonb_object_agg` - anything else — falls back to `{}` `pglite-engine.updateSourceConfig` already used an atomic `||` merge; this change brings postgres-engine to parity. ## Test Added two assertions to `test/list-all-sources.test.ts`: 1. JSONB string holding non-JSON text normalizes to `{}` (no cast throw) 2. JSONB string holding double-encoded valid JSON is parsed then merged * fix(doctor): five correctness fixes — stale locks, content sanity, graph coverage, exit code, gateway guard ## 1. Stale lock break hints cover gbrain-cycle: keys The doctor stale-lock report only recognized `gbrain-sync:` lock prefixes; everything else fell back to `gbrain sync --break-lock`, which is wrong for dream/autopilot cycle locks. A `gbrain-cycle:<source>` or `gbrain-cycle` lock now suggests `gbrain dream --break-lock [--source <name>]`, and unknown lock shapes fall back to `gbrain doctor` instead of a misleading sync command. ## 2. content_sanity_audit_recent counts reject and quarantine as hard failures v0.42 renamed the hard disposition path: rejected pages emit a `reject` event and quarantined junk pages emit `quarantine`; `hard_block` is now only the pre-v0.42 legacy alias. The status check only counted `hard_block`, so fresh `reject` / `quarantine` events from the new path cleared as `ok` whenever fewer than 10 events existed. The check now sums all three for the hard count, and `soft_block + flag` for the soft count. ## 3. graph_coverage excludes test fixture entity pages from the denominator Brains seeded with code sources (e.g. a sync of the gbrain repo itself) could accumulate test fixture pages typed as `entity` / `person`. Including these in the entity-count denominator diluted coverage and produced spurious warnings ("Entity link coverage 0%, timeline 0%") on knowledge-only brains with no real entity pages. The check now queries a per-entity stats CTE that excludes `tools/gbrain/test/*` slugs and the `templates/new-person` stub, with an additional guard for the all-fixture case (`eligibleEntityCount = 0`). ## 4. process.exitCode instead of process.exit at doctor main exit point `process.exit(hasFail ? 1 : 0)` was a hard kill that prevented cleanup handlers (Bun unload events, open DB connections) from running. Using `process.exitCode = hasFail ? 1 : 0` defers the actual termination until the end of the event loop, allowing cleanup to complete. ## 5. checkSubagentCapability exported for test seams + gateway loop guard The function was private, making it untestable in isolation. It is now exported. Additionally, users running gbrain with a non-Anthropic chat model via `agent.use_gateway_loop=true` no longer receive a spurious warning that `ANTHROPIC_API_KEY` is missing — subagents route via the gateway loop in that configuration and do not need the key directly. ## Tests Doctor test suite: 77 pass, 0 fail (no regressions). * fix(engine): deleteFactsForPage excludeSourcePrefixes (#1928) + reconnect() parity (#2034) Engine-layer API for two cycle/availability fixes that share these files: - deleteFactsForPage gains optional excludeSourcePrefixes so the fence reconcile can protect non-fence facts (e.g. cli: conversation facts). - reconnect(ctx?) is now a first-class BrainEngine method on both engines (PostgresEngine already had it; PGLite gains config capture + reconnect) so callers stop using disconnect()+bare connect(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cycle): stop extract_facts from wiping conversation facts (#1928) The fence reconcile delete-then-reinsert wiped cli:-origin facts (no fence to recreate them); a failed-sync full walk turned it brain-wide (1829 rows, 0 reinserted, status ok). Now: exclude cli: rows from the wipe, do NOT inherit the failed-sync->full-walk fallback for this destructive phase, and warn on net-negative reconcile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(autopilot,supervisor): reconnect() instead of disconnect()+bare connect() (#2034) The autopilot health-probe recovery called connect() with no args after disconnect(), losing the startup config (database_url undefined -> FATAL restart-loop on every DB blip) and opening a null-pool window. Both call sites now use engine.reconnect(), which restores the captured config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): mirror to the assigned source's local_path, never the global repo (#2018) put_page write-through resolved the disk target from the global sync.repo_path, so a default-source page (local_path NULL) got written into an unrelated federated source's working tree. Now it uses the assigned source's own local_path; NULL local_path skips (no leak); the global path is used only as a sole-source fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite-lock): heartbeat + steal-grace so live holders are never stolen (#2058) A live holder's lock was force-removed after 5min age alone, letting a second process share the single-writer data dir -> WAL corruption. The lock now heartbeats while held; a holder is reaped only when its PID is dead OR its heartbeat went stale past the steal grace. Pairs PID liveness with heartbeat age to also defeat PID reuse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(migrate,doctor): self-heal idx_timeline_dedup drift (#2038) A migration renumbered during a merge (v102) could be recorded-as-applied without its DDL running, leaving the 3-column index so every timeline write failed the 4-column ON CONFLICT. runMigrations now always runs a shape-keyed drift repair (dedupe-then-rebuild) even when no migration is pending, and doctor surfaces the drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(timeline): un-silence the swallowed batch catch; pin Date-batch round-trip (#2057) The meetings extractor's bare catch {} hid a brain-wide timeline-write failure (0 entries, no error). It now counts + surfaces batch errors. Adds a Date-bearing batch regression test proving the #1861 jsonb_to_recordset refactor already fixed the original ::text[] cast failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v0.42.41.0) Triage fix wave: 6 authored critical fixes (#1928 facts wipe, #2018 write-through leak, #2034 reconnect loop, #2058 WAL lock, #2038 timeline migration drift, #2057 timeline silent-empty) + community PRs #2064 #2052 #2020 #2033 #2074 #2075 #2009 #2072 #2073. TODOS: deferred #1994 #1963 #2050. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: address adversarial review findings (#1928, #2058, #2038, #2057) Codex as-built review of the authored fixes surfaced 4 real issues: - #2058: add a pid+acquired_at ownership token. A stale holder reaped + replaced past the grace must NOT let its resumed heartbeat refresh, nor releaseLock remove, the NEW owner's lock (re-opened the concurrent-writer hole). Heartbeat and release now verify the on-disk lock is still ours. + regression test. - #1928: the destructive-full-walk guard keyed off phases.includes('sync'), which wrongly suppressed a legitimate full reconcile when sync was SKIPPED (no engine / no brainDir). Key off a syncAttempted flag set only when sync actually ran. - #2038: dedupe keeps MIN(id) not MIN(ctid) — deterministic and consistent with the existing v-migration lower-id rule. - #2057: the extract CLI caller now surfaces batch_errors (stderr + exit 1) instead of printing a clean success over failed inserts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(key-files): sync reference to v0.42.41.0 triage-wave behavior Update KEY_FILES.md to current-state truth for the shipped fixes (no release-history clauses, per the reference-doc discipline): - write-through.ts (#2018): resolves the disk target from the assigned source's own local_path; sole-source falls back to sync.repo_path, multi-source skips with source_has_no_local_path rather than leak. - engine.ts (#2034): reconnect() is now a REQUIRED lifecycle method on both engines; config-restoring, never disconnect()+bare connect(). - migrate.ts (#2073): document v116 edge source_id backfill + callee index, and the always-run (version-counter-blind) timeline dedup self-heal. - new entry for timeline-dedup-repair.ts (#2038) + the timeline_dedup_index doctor check. - new entry for pglite-lock.ts (#2058): heartbeat + steal-grace (GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS) so a live holder is never stolen. - extract-facts.ts (#1928): cli:-fact protection, no failed-sync full-walk inheritance, net_fact_deletion warn floor. bun run build:llms re-run (KEY_FILES is link-only so bundles unchanged); freshness + current-state guards green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): preserve nested multi-source layout; narrow #2018 leak guard The first #2018 fix skipped any no-local_path source on a multi-source brain, which broke the legitimate nested layout (a source without its own tree nests under the host repo at .sources/<id>/ — pinned by put-page-write-through.test). Narrow the guard: a no-local_path source nests under sync.repo_path as before; only SKIP when sync.repo_path is literally another source's own local_path (the actual leak — writing there pollutes that sibling's repo). Caught by the sharded suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: satisfy test-isolation guard for the new lock/reconnect tests CI `verify` flagged 3 intra-process isolation violations in the tests added this wave (the parallel runner shares one process per shard): - pglite-lock.test.ts: the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS mutation now goes through withEnv() instead of a raw process.env write (R1). - pglite-reconnect: renamed to *.serial.test.ts — it creates per-test engines to exercise the connect/reconnect lifecycle, which doesn't fit the shared beforeAll-engine model (R3/R4). verify is now 30/30; both files green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite): reconnect() is a no-op for in-memory engines (#2034) CI serial-tests + test(5) caught two in-branch regressions from the #2034 PGLite reconnect(): - worker/queue claim-error recovery + their renewLock e2e test assume PGLite reconnect is absent/no-op (queue.ts documents it). Making it a real disconnect+reopen wiped an in-memory engine's state mid-job. reconnect() now no-ops for in-memory (no database_path) — file-backed still re-opens the dir (state persists on disk). Restores the documented worker assumption. - connection-resilience 'Supervisor still has the 3-strikes-then-reconnect path' pinned the removed unsafe-cast text; updated to assert the direct this.engine.reconnect() call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: quarantine embed-input-type-wire to serial lane (CI test(5) leak) #2033's embed-input-type-wire.test.ts configures a 1280-dim embedding gateway; the active dimension survived into engine-find-trajectory when CI's 10-way hash-disjoint sharding co-located them (this branch's added files reshuffled the assignment), failing 7 trajectory tests with 'expected 1280 dimensions, not 1536'. resetGateway() in afterEach clears the gateway but the dimension still leaked. It mutates global gateway/embedding state, so it belongs in the serial lane (own bun process, true isolation) by the repo's own definition. Root-caused by reproducing the exact failing pair locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Austin Arnett <austin@sdsconsultinggroup.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Dave MacDonald <djmacdonald@ucdavis.edu> Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com> Co-authored-by: Alex P. <12667893+aphaiboon@users.noreply.github.com> Co-authored-by: Garry Tan <bo.m.liu@gmail.com> Co-authored-by: jbarol <barol.j@gmail.com> Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> Co-authored-by: PAI <pai@scaffolde.ai> |
||
|
|
ecd6ae8772 |
v0.42.40.0 fix(extract,ingest): well-form lone UTF-16 surrogates before jsonb (#2011) (#2031)
* fix(extract,ingest): well-form lone UTF-16 surrogates before jsonb (#2011) excerpt() in link-extraction.ts sliced the link-context window by raw UTF-16 index, so a boundary landing inside a non-BMP char (emoji, math, CJK) left an unpaired surrogate half in `context`. Serialized to JSONB for the jsonb_to_recordset batch insert, Postgres rejects it at the ::jsonb cast and aborts the whole batch — wedging `extract --stale` because the staleness bookmark only advances on a clean finish. - text-safe.ts: new ensureWellFormed() (Bun isWellFormed/toWellFormed) — one shared surrogate-cleaning primitive. - link-extraction.ts: excerpt() well-forms the slice (root-cause fix). - batch-rows.ts: new sanitizeForJsonb() = ensureWellFormed(stripNul(s)) applied to every free-text body field (link context; timeline summary/detail/source; take claim/source). Identity/security fields stay un-sanitized and fail closed. - postgres-engine.ts + pglite-engine.ts: scalar addLink + addTimelineEntry use sanitizeForJsonb too, matching the batch path on both engines. - brainstorm/orchestrator.ts: consolidate hand-rolled sanitizeUnicode onto ensureWellFormed (also fixes consecutive-lone-surrogate mishandling). Tests: ensureWellFormed unit cases (incl. consecutive lone surrogates), an excerpt window-split regression, PGLite + Postgres-e2e surrogate cases across all free-text fields and scalar paths, and fail-closed identity-field tests proving sanitization was NOT extended to slugs/holders. * v0.42.39.0 chore: bump version and changelog (#2011) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.42.39.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.40.0 chore: re-slot release version (was 0.42.39.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: fix env-mutation isolation violations in retrieval-reflex tests check:test-isolation (rule R1) flags direct process.env mutation in non-serial test files — bun's parallel runner loads multiple files into one process, so a leaked GBRAIN_RETRIEVAL_REFLEX flips reflex behavior in unrelated tests. Both files landed via the #2019 merge; convert the beforeEach/afterEach env juggling to the canonical withEnv() wrapper, which restores the prior value via try/finally even on throw. Fixes the failing `verify` CI check on #2031 (and the `test-status` aggregate that inherits it). All 30 verify checks green locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
8f45624e55 |
v0.42.39.0 feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981) (#2019)
* fix(integrations): parameterize resolver-row fence by recipe id The install fence was hardcoded gbrain:agent-voice:resolver-rows, so any second copy-into-host-repo recipe wrote a mislabeled block (and refresh/ uninstall keyed on recipe id would miss it). Derive it from manifest.recipe. * feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981) Deterministic per-turn pointer layer in the context engine: a zero-LLM, precision-biased scan resolves salient entities (names, @handles) to existing brain pages and injects compact pointers (name → slug → safe synopsis). Detect + point, never auto-dump. Fail-open, capped, suppression on prior context only. Engine-aware resolver ladder (no second DB connection): host ctx.brainQuery → PGLite serve resolve IPC (unix socket) → Postgres cached direct → disabled. Synopsis runs through get_page's privacy strip. Plus the retrieval-reflex recipe + policy skill, the retrieval_reflex_health doctor check, config gate, and the init next-step hint. * chore: bump version and changelog (v0.42.38.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): KEY_FILES entries for Retrieval Reflex surface (#1981) Document the new src/core/context/ modules, the context-engine resolver ladder, the serve resolve IPC, the retrieval_reflex_health doctor check, and the recipe-id-keyed install fence. Current-state only. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
03ffc6ebdb |
v0.42.37.0 fix(jobs): reap stale locks, bound disconnect, complete cooperative-abort (#1972) (#2015)
* fix(jobs): reap stale dead-holder cycle/sync locks (#1972) A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row until something contended for it — reclaim was on-contention only. Add a host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle* namespaces only (never elections/supervisor/reindex), with a snapshot-matched delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse. Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now back inspectLock + listStaleLocks (killed the triplication). * fix(db): bound pool disconnect so teardown can't eat CLI output (#1972) pool.end() against PgBouncer transaction-mode never drained, so disconnect blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write, truncating stdout (e.g. #1959's relational query returned empty). Add a gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against a hard timer, so teardown is bounded regardless of what postgres.js does and is testable. connection-manager ends its direct + read pools concurrently so the per-pool bounds don't stack. PGLite disconnect is unaffected. * fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972) v0.42.29 made only the embed phase honor the abort signal; a 24h pull still showed force-evicts from a long non-embed phase ignoring it. Thread the signal into every cycle-reachable long loop: extract (extractForSlugs + the full-walk extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop + embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket loop. Add a terminal abort check so an aborted cycle never stamps last_full_cycle_at as a completed run (Codex #9). lint now yields + checks abort every 200 pages (it's synchronous; the yield is what lets the signal land). New phase-duration force-evict attribution log names any phase that crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start. * chore: bump version and changelog (v0.42.38.0) #1972 — stale-lock reaper, bounded pool disconnect, and complete cooperative-abort coverage across cycle phases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY), db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/ extract_facts/consolidate/lint + terminal guard) entries to current truth. * test(isolation): fix shard-order flakes exposed by #1972's new test files Adding 3 new test files reshuffled the hash-based shards, exposing two pre-existing test-isolation bugs: - cycle-consolidate.test.ts assumed the global legacy-embedding preload's 1536-d gateway config still held at initSchema, but a co-sharded test that calls resetGateway() in teardown nulls it, so initSchema fell back to the 1280-d default and built a halfvec(1280) facts column its 1536-d fixtures can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern legacy-embedding-preload.ts documents for 1536-d fixture tests). - db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation rule R1. Convert to withEnv(). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
1eb430a2df |
v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts (#1999)
* fix(security): scope cross-source reads to the caller grant; close get_page exact-path leak One shared resolveRequestedScope() routes every source-scoped read op (query, code_callers/callees, search_by_image, code_blast/flow, get_page) through a single fail-closed trust+grant ladder: a remote caller's __all__ collapses to its granted sources (never the whole brain) and an explicit out-of-grant source_id is rejected. get_page's exact-match path now honors a federated grant via getPage(sourceIds[]) in both engines. Legacy bearer tokens carry their stored permissions.source_id grant (bounded, never widened). Also retries getConfig on transient connection loss. Closes #1924, #1371, #1393, #1336, #1603. * fix(ingest): non-string frontmatter no longer aborts lint/sync; embed/hook/catalog papercuts Parser coerces a non-string title to a string and falls back to inference for slug/type (never fabricating a "123" slug), with a lint NON_STRING_FIELD finding surfacing the malformed frontmatter; a defensive guard in content-sanity stops a non-string title from crashing the whole lint/sync run brain-wide. Plus: embed --catch-up no longer arms the overflowed 32-bit budget timer (and surfaces unembeddable chunks); the frontmatter pre-commit hook ships a correct .md/.mdx regex; and the skill catalog parses YAML block-scalar descriptions. Closes #1883, #1658, #1556, #1948, #1946, #1840, #1711. * v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add NON_STRING_FIELD frontmatter validation class to docs for v0.42.37.0 The v0.42.37.0 non-string-frontmatter fix added an eighth validation class (NON_STRING_FIELD / lint code frontmatter-non-string-field). Update the two current-state docs that enumerate the validation classes: - skills/frontmatter-guard/SKILL.md (seven->eight + table row) - docs/integrations/pre-commit.md (seven->eight + table row) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
959af1068d |
v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794) (#1980)
* fix(retry): match EMAXCONNSESSION + SQLSTATE 53300 as retryable conn errors (#1794) * feat(schema): add op_checkpoint_paths append-only delta table (migration v115) (#1794) * refactor(op-checkpoint): append-only deltas via executeRawDirect + withRetry (#1794) * fix(sync): resumable-checkpoint durability + lock-thrash fix (#1794) Durable append-only checkpoint writes (executeRawDirect + retry), fail-loud consecutive-failure abort, first-file/10s flush cadence, race-safe pending-delta under parallel workers, guaranteed final flush on every exit path incl. SIGTERM (no-retry one-shot via registerCleanup), bankedFiles/reason observability, event-loop yield to keep the lock heartbeat alive, and routing the bare (no-source) sync through withRefreshingLock. * fix(db-lock): heartbeat-aware takeover + direct-pool refresh (#1794) * fix(cycle): treat SyncLockBusyError as skip, not a phase failure (#1794) * docs(sync): document the 5 checkpoint/lock env knobs (#1794) * v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): update sync.ts + op-checkpoint.ts entries to resumable-checkpoint current state (#1794) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
612753f318 |
v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970) (#1975)
* v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970) When a source's history is rewritten (force-push, master→main consolidation, squash), the recorded last_commit can fall outside HEAD's history. The old guard sent both "object missing" and "not an ancestor" to performFullSync — a full repo re-walk that never advances the bookmark under a cron timeout on a large cross-region brain, so the source goes silently stale. Fix: only a truly-absent object forces a full reconcile. A present-but-non- ancestor bookmark is diffed tree-to-tree directly (git diff A..B needs no ancestry), importing only the real delta. Adds: oversized-diff fallback to full reconcile (F-B); performFullSync now purges deleted files, gated to file-backed pages by source_path so manual/put_page and metafile pages are spared (F-A); rename-to-unsyncable deletes the stale old page (F-C). 7 new PGLite e2e tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): record #1970 sync bookmark recovery + full-sync delete reconcile in KEY_FILES Update the sync.ts entry to current truth: entry-time bookmark-reachability guard (gc'd anchor → full reconcile; non-ancestor-but-present → direct tree-to-tree diff), oversized-diff fallback, performFullSync now authoritative for deletes (file-backed pages by source_path), and rename-to-unsyncable delete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |