mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
1229bec1eb32761648045ca9369d21dae3daf366
272
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Sanjay SanthanamandGitHub
|
1229bec1eb |
fix(postinstall): cross-platform node shim instead of POSIX shell (#1554)
* fix(postinstall): cross-platform node shim instead of POSIX shell
The postinstall script used POSIX shell syntax ('command -v',
'>/dev/null 2>&1', '1>&2') which Bun's built-in script parser rejects
on Windows. 'bun install' aborted with 'expected a command or
assignment but got: "Redirect"' before gbrain could ever be probed.
Replace with a one-liner 'node -e' shim that:
* uses spawnSync to probe 'gbrain --version' (shell:true on win32 so
the Windows shim/.cmd resolution works)
* on success: runs 'gbrain apply-migrations --yes --non-interactive'
and propagates its exit code
* on failure: writes the same skip message to stderr and exits 0, so
fresh clones (where gbrain isn't on PATH yet) still complete install
POSIX hosts retain the original behavior; Windows hosts now succeed
instead of failing the whole install.
Fixes #1486
* fix(postinstall): move logic to scripts/postinstall.ts to survive Bun Windows script-runner
The `node -e` inline shim still failed on Windows under Bun. Embedding a
program inside the package.json postinstall string lets the lifecycle shell
mangle it: Bun's Windows script-runner expands the `\n` in the hint string
into a REAL newline before node sees it, producing `SyntaxError: Invalid or
unexpected token` and aborting the whole install. `node` is also not
guaranteed present under a Bun install (bun is the guaranteed runtime), and
`shell: win32` re-opened a quoting surface.
Move the logic into a checked-in `scripts/postinstall.ts` run via
`bun run scripts/postinstall.ts`, matching the repo's existing convention of
~19 scripts under scripts/*.ts. This sidesteps all three failure modes:
* `which('gbrain')` from bun does Windows-aware PATH resolution (finds
gbrain / gbrain.exe / gbrain.cmd) with no shell.
* `Bun.spawnSync` with an argv array invokes apply-migrations directly —
no shell, nothing to quote, no `\n` expansion.
* No dependency on `node` being present; bun runs the script.
Behavior is preserved exactly: same `apply-migrations --yes
--non-interactive` command, same issue-218 skip hint, and the same
never-fail-the-install guarantee (every path exits 0). Verified on macOS:
`bun run scripts/postinstall.ts` exits 0 on the skip path (gbrain absent),
on a failing migration, and on a successful migration.
Fixes #1486
|
||
|
|
0cf5596c88 |
v0.42.61.0 chore(release): ten verified community improvements — changelog + version bump (#2890)
Autopilot crash recovery, deterministic atom slugs, takes bootstrap progression, bundled-pack activation, Sonnet 5/Fable 5 pricing, inline citation timelines, pack-driven extraction discovery, book-mirror HTML tables, gateway test-pin, docs sync. Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
a7b0ae80a9 |
v0.42.60.0 chore(release): eleven verified community fixes — changelog + version bump (#2888)
* v0.42.60.0 chore(release): eleven verified community fixes — changelog + version bump Windows full-sync mass-delete fix, gateway tool-loop resume consolidation (fix-wave A), two source-isolation closes, search-cache exclude-policy keying, and six more verified community fixes. Files the take-writes fail-open source fallback and the #2112 doctor hunk as follow-up TODOs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update reference docs for v0.42.60.0 - KEY_FILES.md: unpin stale KNOBS_HASH_VERSION number in the autocut entry (mode.ts is the single source of truth); document the full-sync reconcile path-separator normalization + mass-delete safety valve (planReconcileDeletes, GBRAIN_ALLOW_MASS_RECONCILE); describe the TTY-gated admin bootstrap token banner (--print-admin-token, env-sourced always hidden) - docs/mcp/DEPLOY.md + docs/tutorials/company-brain.md: bootstrap token is now hidden on non-TTY starts; document GBRAIN_ADMIN_BOOTSTRAP_TOKEN and --print-admin-token for headless deploys Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(docs): regenerate llms bundle after KEY_FILES/deploy-doc sync Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
5008b287e4 |
v0.42.59.0 chore(release): five verified community fixes — changelog + version bump (#2797)
Rolls up the five fixes merged as #2735 #2736 #2737 #2738 #2739 (issues #2724 #2677 #2723 #2726, plus the think slice of #2200). Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
a25209bbb2 |
v0.42.58.0 fix(ai): provider-agnostic gateway — env clobber, base-URL /v1, embedding dims (#1249 #1250 #1292 #2271 #2209) (#2627)
* fix(ai): drop empty-string env values before merge so they can't clobber config keys (#1249) Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls; an unconditional process.env spread let that empty string override a valid config.json key, breaking every gateway op with NO_ANTHROPIC_API_KEY. Filter '' / undefined before the merge; '0' and 'false' are preserved. * fix(ai): normalize native provider base URLs + replace embedding guard with a dims-presence check (#1250, #1292) #1250: createAnthropic/createOpenAI were called with no baseURL, so an env-injected bare host (e.g. ANTHROPIC_BASE_URL without /v1) 404'd. Add a shared resolveNativeBaseUrl and pass a normalized baseURL at all anthropic + openai native sites (google deferred until its suffix is verified). #1292/D6: the user_provided_model_unset guard was structurally unreachable as a no-model check (parseModelId throws on a bare provider) and only ever false-positived for litellm:<model>, silently disabling vector search. Replace it with a real dims-presence check for user-provided/zero-default recipes and delete the dead branch in both consumers. Also stop configureGateway from fabricating a default embedding_dimensions, so 'no dims set' stays honest. * fix(ai): trust user-declared embedding dims for local recipes + litellm /v1 hint (#2271, #2209) #2271: a new trust_custom_dims flag adds a passthrough tier so ollama / llama-server / litellm accept a user-supplied --embedding-dimensions instead of being hard-rejected. Fail-closed for fixed-dim providers (openai/voyage/ zeroentropy) and excludes openrouter (declares dims_options). Register modern ollama embed model names. #2209: litellm setup_hint now states the /v1 path convention and the docs pointer is corrected to docs/integrations/embedding-providers.md. * docs+test(ai): KEY_FILES current-state for provider-agnostic gateway + embed-preflight dims-unset test (#1249, #1250, #1292) * fix(ai): point user_provided_dims_unset remediation at 'gbrain init' (config set rejects it) + coverage Pre-landing adversarial review (P1): the new dims-unset guard told users to run 'gbrain config set embedding_dimensions <N>', which config.ts hard-rejects (it's a schema-sizing field). Both consumer messages now point at 'gbrain init --embedding-dimensions'. Adds: pgvector-cap-still-fires regression for the trust_custom_dims passthrough, and a configureGateway backfill-invariant test. * chore: bump version and changelog (v0.42.57.0) Provider-agnostic plumbing wave: #1249 empty-env clobber, #1250 native baseURL normalization, #1292 embedding dims-presence guard, #2271 trust_custom_dims passthrough, #2209 litellm /v1 hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync embedding-providers guide for provider-agnostic gateway wave (v0.42.57.0) Post-ship doc drift fix for the v0.42.57.0 AI-gateway wave: - LiteLLM section now names the /v1 base-URL convention (#2209). - Ollama section lists the newly-registered modern embedders qwen3-embed-8b + snowflake-arctic-embed-l-v2, and notes dims-trust for local recipes (#2271). - llama-server section notes gbrain trusts the user-declared dimension (#2271). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: post-ship doc sweep for v0.42.57.0 provider-agnostic gateway wave - KEY_FILES.md types.ts entry: document EmbeddingTouchpoint.trust_custom_dims (#2271 passthrough tier, runs after dims_options + Matryoshka allowlists) - ENGINES.md: embedding design-choice note now names the provider-agnostic gateway delegation instead of the stale OpenAI-only parenthetical - embedding-providers.md: drop an exact-duplicate doctor-8c paragraph - llms-full.txt regenerated (ENGINES.md is inlined in the bundle) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply codex doc-review findings for v0.42.57.0 (base-URL env note, litellm multimodal) - embedding-providers.md OpenAI section: document OPENAI_BASE_URL / ANTHROPIC_BASE_URL bare-host /v1 normalization (#1250 user-facing surface) - TL;DR table: litellm multimodal is backend-permitting (recipe declares supports_multimodal: true, routed via the openai-compat multimodal path), not "no" Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: pin engine-find-trajectory schema to 1536 + stop gateway-state leaks across shard files CI shard 5 failed 7 findTrajectory tests with 'expected 1280 dimensions, not 1536': engine-find-trajectory hardcodes 1536-d vectors but sizes its schema from AMBIENT gateway state in beforeAll — which runs before the legacy-embedding-preload's per-test 1536 restore. A preceding file that ends with a dimensionless configureGateway (facts-extract-silent-no-op) or a bare resetGateway poisons the next fresh initSchema down to 1280-d columns. The new test files in this PR reshuffled shard bin-packing and exposed the trap. - engine-find-trajectory: pin OpenAI/1536 explicitly before initSchema (the pattern bunfig's preload documents) — deterministic regardless of neighbors - facts-extract-silent-no-op, diagnose-embedding-dims, embed-preflight: restore the legacy 1536 pin in afterAll instead of ending reset/dimensionless Reproduced: synthetic dimensionless-gateway file + old victim = the exact 7 CI failures; with the pin = 0. Verified in-process pair runs both orders. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
058f448b9a |
v0.42.57.0 fix(pglite): incident — never steal a live data-dir lock + corrupted-store recovery hint (#2348) (#2400)
* fix(pglite): never steal the data-dir lock from a live holder (#2348) A busy `gbrain dream`/`embed` holder whose 30s heartbeat lapsed (the JS event loop is blocked during long synchronous WASM imports/CHECKPOINTs) used to get its lock reaped past the steal-grace window. PGLite/WASM is strictly single-writer, so a second OS process then opened the same data dir and corrupted the catalog + pgvector extension state (58P01 / internal_load_library / "type vector does not exist"), recoverable only by wipe+restore. Reap ONLY a dead PID; a live holder is never stolen — a wedged-but-alive or PID-reused holder makes the acquire time out with a message naming the PID. Removes the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS knob (no longer meaningful). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pglite): point a corrupted store at reinit-pglite recovery (#2348) classifyPgliteInitError gains a `corrupt` verdict for the 58P01 / internal_load_library / "vector does not exist" / "content_chunks does not exist" signature (beating the generic wasm-runtime match), so an already-damaged store gets actionable recovery (gbrain reinit-pglite / restore a backup) instead of the wrong macOS-WASM hint. Updates KEY_FILES.md to current lock behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.55.0 fix(pglite): incident — never steal a live lock + corrupted-store recovery hint (#2348) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.56.0 chore: re-bump past #2399 version collision + refresh ownerToken comment #2399 (security wave) claimed 0.42.55.0; take the next slot. Also updates the LockHandle.ownerToken JSDoc to current #2348 behavior (live holders are never reaped, so reap-then-reacquire is dead-holder-only; token guard stays as defense-in-depth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
646179047a |
v0.42.56.0 feat(chronicle): Life Chronicle — temporal timeline + thought diary + bi-temporal per-entity ontology (#2390) (#2533)
* feat(chronicle): register event + diary page types (#2390) Life Chronicle Phase A.1. Adds `event` (timeline atom) and `diary` (first-person interiority) as temporal-primitive page types under life/events/ and life/diary/, extractable:false — registered in ALL_PAGE_TYPES, both base schema packs, and the inferType prefix table, with parity fixtures. Also lands the chronicle read result types (ChronicleTimelineRow/ChronicleTimelineOpts/LastSeenResult) consumed by Phase A.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): event_page_id timeline projection + day/since/last-seen reads (#2390) Life Chronicle Phase A.2. Adds a nullable event_page_id FK to timeline_entries (migration v120; mirrored in schema.sql, pglite-schema, schema-embedded) so a type:event page projects ONE date-index row keyed to its depth page; a partial UNIQUE(event_page_id, date) makes re-extraction with a changed summary an update, not a duplicate. Dual-engine getTimelineForDate / getSince / getLastSeen filter the depth page on deleted_at, hide soft-deleted event projections at READ time (not just doctor), order by event effective_date for intra-day sequence, and honor source scope (sourceIds[] > sourceId). Ops surface as `gbrain day <date> [--week]`, `gbrain since <date> [--kind]`, `gbrain last-seen <entity>`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): auto-emit extractor — backstop + chronicle_extract job (#2390) Life Chronicle Phase A.3. A put_page backstop (gated on status==='imported', the auto-link/timeline trust gate, and the default-OFF auto_chronicle flag; diary + event pages never eligible) enqueues a chronicle_extract minion job. The job runs the extractor off the write path: deterministic when/who, an injectable LLM judge (default = chat gateway; no-op when no gateway), an all-or-nothing parse barrier (a malformed proposal writes NOTHING), then content-addressed event pages + a timeline projection via the new dual-engine upsertEventProjection (idempotent — re-run yields one event + one projection). New: src/core/chronicle/{eligibility,config,extract-events,backstop}.ts, engine.upsertEventProjection (both engines), the jobs.ts handler + a 10-min timeout. 14 unit tests (eligibility, idempotency, parse barrier, backstop gating + enqueue) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): quick-capture for diary + manual events (#2390) Life Chronicle Phase A.5. `gbrain capture` now routes the default slug by type (diary → life/diary/, event → life/events/, else inbox/) and accepts --who/--what/--where/--kind/--depth to assemble the `event:` frontmatter block for `--type event`. User-declared event keys win per-key over the flags. Goes through the existing put_page → write-through → embed path. 6 new unit tests; existing capture tests stay green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): bi-temporal per-entity ontology on the facts table (#2390) Life Chronicle Phase B.10 — the feature's differentiator. Rather than a parallel store, the open-world per-entity ontology EXTENDS the existing `facts` table (eng-review G1): migration v121 adds dimension/value/value_hash /dim_status columns + a deterministic partial-UNIQUE dedup key + an asof read index. facts already supplies bi-temporal validity, supersession (superseded_by), visibility (remote redaction), confidence, provenance, and embedding — all inherited. Dual-engine methods: mergeOntologyFact (deterministic value_hash dedup → idempotent retry; same value corroborates; a different value forward-closes the prior row's valid_until + superseded_by; a BACKDATED conflicting value is recorded WITHOUT rewriting, surfaced by findOntologyConflicts), getOntology with `--asof` valid-time travel (expired_at + status + validity-window in the predicate so quarantined/superseded never leak), discoverOntologyDimensions, findOntologyConflicts. Novel/LLM-proposed dimensions quarantine; a seed lexicon canonicalizes name drift (job_role → role). 9 unit tests cover the full lifecycle; typecheck pins both engines to the interface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): ontology ops — get/add/dimensions/contradictions (#2390) Life Chronicle Phase B.11. Exposes the bi-temporal ontology over CLI + MCP (contract-first, auto-generated): `gbrain ontology <entity> [--asof]`, `gbrain ontology-add <entity> <dim> <value>`, `gbrain ontology-dimensions` (meta-ontology rollup), `gbrain ontology-contradictions`. All reads route through sourceScopeOpts. Privacy: ontology_get redacts diary-sourced observations (source under life/diary/) for untrusted (remote) callers. 3 op tests (incl. the remote-redaction path); 47 op-registry/tool-def/description tests stay green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): agent-context loader — volunteer_chronicle (#2390) Life Chronicle Phase B.12. `loadChronicleContext` hands an agent the recent timeline (last N days) + the validity-resolved current ontology for the entities in play, in one zero-LLM payload, so it orients before acting — the exact gap behind fumbled chronology. Pure composition over getSince + getOntology (no new SQL). Exposed as the `volunteer_chronicle` read op (`gbrain orient [--days] [--entities a,b]`); diary-sourced ontology is redacted for remote callers. 2 loader tests + op-registry green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): backfill op — sweep existing meetings into events (#2390) Life Chronicle Phase A.8. `chronicle_backfill` (local-only admin op; `gbrain chronicle-backfill [--since] [--limit] [--dry-run]`) lists existing meeting/conversation/calendar pages (source-scoped via listPages), filters through the chronicle eligibility predicate, and enqueues one chronicle_extract job per eligible page so existing brains populate the timeline. --dry-run counts only; per-page enqueue failures are surfaced in `errors`, never swallowed. 2 op tests (dry-run count + enqueue). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): delight — on-this-day + narrative rendering (#2390) Life Chronicle Phase A.6 (delight). Dual-engine getOnThisDay (events from the same month-day in prior years; `gbrain on-this-day [--date]`) reusing the chronicle JOIN shape (deleted-event hiding + source scope). A pure renderTimelineNarrative turns timeline rows into prose; `gbrain day --narrative` returns it alongside the events. 5 tests. (Coverage gap-detection ships with the advisor collectors next.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): proactive advisor collector (#2390) Life Chronicle Phase A.7. A brain-state advisor collector surfaces two proactive signals in `gbrain advisor`: unresolved ontology conflicts (warn, → `gbrain ontology-contradictions`) and recent meetings with no timeline coverage (info, → `gbrain chronicle-backfill`). Advisory-only (no dispatch_id); runs over MCP too (not workspace-dependent); tolerant of pre-migration brains. 3 tests + advisor-op-gate green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): doctor chronicle_projection_health probe (#2390) Life Chronicle Phase B.13. An always-run doctor check (keyed off the event_page_id schema, NOT a migration verify-hook) counts timeline projections whose event page is soft-deleted — hidden at read time, surfaced here as a cleanup backlog (`gbrain integrity auto`). Tolerant of pre-migration brains. 1 detection test. (auto_chronicle / chronicle.tz flags already work via getConfig defaults; their docs land with document-release at ship.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): E1 temporal recall — chronicle-type boost on temporal queries (#2390) Life Chronicle Phase A.4 (E1, ambient temporality). Rather than a separate RRF arm (which needs chunk hydration + risks the fusion path), E1 is a bounded post-fusion boost: applyChronicleTypeBoost lifts `event`/`diary` results on temporal queries, wired INSIDE runPostFusionStages' `recency !== 'off'` branch so it fires ONLY on temporal intent — non-temporal search is bit-for-bit unchanged (proven by 110 passing search-path tests). Bounded ([1.0,1.25]) + floor-gated like the other metadata stages; attribution via `chronicle_boost`. 3 unit tests. (Empirical precision/negative measurement lands with the eval.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(chronicle): feature eval — gbrain eval chronicle (PRIMARY proof) (#2390) Life Chronicle Phase A.9, the North-Star proof. A deterministic, CI-safe eval (brings its own in-memory PGLite; no LLM, no gateway) builds a synthetic month corpus with a known gold chronology + a planted ontology supersession + a planted conflict, then scores the chronicle layer on six gold tasks: day reconstruction (intra-day order), last-seen exact date, ontology supersession, --asof valid-time travel, contradiction surfacing, and source isolation. `gbrain eval chronicle [--json]` exits 0 iff all pass — currently 6/6. The OFF baseline (raw meeting pages) structurally can't order intra-day events or time-travel ontology; the ON path does. (The live-LLM OFF-vs-ON agent arm + LongMemEval temporal slice are a follow-up; this deterministic bar gates CI.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chronicle): pre-landing review — conflict validity, parse-barrier date, remote conflict redaction (#2390) Three bugs caught by the codex pre-landing review on the diff: 1. findOntologyConflicts ignored valid_until, so a normal forward supersession (founder→advisor from two sources) falsely reported as a live conflict. Now restricted to currently-open rows (valid_until IS NULL) in both engines. 2. The extractor parse barrier accepted any when-string >= 4 chars; a non-date value slipped past, wrote the event page, then threw on the projection's ::date cast (partial write). isValidProposal now requires a real parseable date. 3. The ontology_conflicts op had no remote diary redaction (ontology_get did); remote callers now get diary-sourced values filtered, and conflicts that lose their disagreement after redaction are dropped. Three regression tests added; 29 chronicle tests + eval 6/6 green; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.56.0 feat(chronicle): Life Chronicle — temporal timeline + diary + bi-temporal ontology (#2390) Bump VERSION + package.json to 0.42.56.0 and add the CHANGELOG entry for the Life Chronicle feature (#2390, closes duplicate #2388). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chronicle): register #2390 surfaces with the five CI guard suites (#2390) CI caught five guard tests that pin registration invariants my diff tripped: - schema-bootstrap-coverage: the four v122 facts ontology columns join COLUMN_EXEMPTIONS (facts is migration-created; the partial indexes live inside v122; every reader filters dimension IS NOT NULL — same precedent as facts.claim_metric et al). - no-valid-until-write (R8): both engines' mergeOntologyFact forward- supersession is a deliberate, documented valid_until write authority (engine-layer, dimension IS NOT NULL only; the contradiction probe still never mutates). - doctor-categories: chronicle_projection_health registered under BRAIN_CHECK_NAMES (same class as child_table_orphans). - checkTypeProliferation: the test is now threshold-relative (computes declared from the active pack, seeds declared+6) so base-pack growth can't silently move the fixed threshold again. - schema-cli: gbrain-base page-type count 25 → 27 (event + diary), with assertions on both new types. All 41 guard tests green; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: file Life Chronicle follow-up TODOs (v0.42.56.0, #2390) Eight deferred items from the CEO/eng review decisions (auto-emit default-flip fast-follow, live eval arm + LongMemEval slice, passive diary consent, interval-splitting, federation, place-as-entity, meta-ontology dashboard, materialized daily pages), each with decision provenance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): KEY_FILES entry for the Life Chronicle module (#2390) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
dde1132a29 |
v0.42.55.0 fix(security): dotfile/skills/slug confinement + DCR consent default + schema-lint migration (#418 #419 #245 #1353 #1647 #171 #1385) (#2399)
* fix(security): confine routing dotfiles, skills dir, slugs, and transcription exec Shared src/core/path-confine.ts consolidates the realpath-containment idiom (moved from sources-ops.ts) and adds isTrustedDotfile + isWriteTargetContained. - .gbrain-source (source-resolver) and .gbrain-mount (brain-resolver) walk-up dotfiles are now lstat trust-gated: a symlink, foreign-owned, or world-writable file is refused on multi-user hosts (#418), fail-closed on stat error. - resolveWorkspaceSkillsDir + every skills-dir tier (env, cwd_walk_up, repo_root, cwd_skills, install_path) route through realpath containment so a symlinked workspace/skills can't escape the declared workspace (#419). - resolveSourceId/resolveBrainId realpath both sides of the registered local_path / mount prefix match so a symlinked cwd can't misattribute source/brain. - validateSlug rejects NUL/control, bidi/RTL overrides, backslashes, and URL-encoded path separators at the shared putPage/updateSlug chokepoint; write-through confirms the file path stays within the source tree. - transcribeLargeFile uses execFileSync arg-arrays + fs.rmSync (no shell), so a path with shell metacharacters is never parsed by a shell (#245). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): default dynamic-registration clients to authorization_code Self-registered DCR clients (the unauthenticated network registration path) previously defaulted to the client_credentials grant, which bypasses the /authorize consent screen. They now default to authorization_code; an explicit client_credentials request is rejected with invalid_client_metadata unless the operator opts in with the new --enable-dcr-insecure flag. A loud stderr WARNING prints at startup whenever DCR is enabled (#1353). Manual CLI/admin client registration is unchanged (operator-trusted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): schema-lint hardening migration (search_path + view security_invoker) Migration v120 brings existing brains to the same posture as fresh installs: - ALTER VIEW page_links SET (security_invoker = on) on Postgres so the view honors the caller's RLS instead of the owner's (the view-through-RLS bypass). - ALTER FUNCTION ... SET search_path on the gbrain-owned trigger/event functions (both engines, IF EXISTS so engine-only functions are skipped; body untouched, so the load-bearing auto_enable_rls event trigger is unchanged). Closes #171. - Broaden the BYPASSRLS preflight in the historical RLS migration gates to honor superuser and inherited-role BYPASSRLS, so a superuser-connected fresh install no longer aborts (#1385). Fresh-install function definitions in schema.sql / pglite-schema.ts are born-correct (regenerated schema-embedded.ts). scripts/check-search-path.sh is a new CI guard (wired into verify) that fails if a trigger function in the schema base files is added without SET search_path. Postgres-only assertions live in the bootstrap E2E; the PGLite path is covered by test/migration-v120.test.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.55.0 fix(security): dotfile/skills/slug confinement, DCR consent default, schema-lint migration Bump VERSION + package.json to 0.42.55.0 and add the CHANGELOG entry for the security-hardening wave (#418 #419 #245 #1353 #1647 #171 #1385). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(security): note the DCR consent default in SECURITY.md (#1353) The "disable client_credentials, only allow authorization_code" guidance is now the built-in DCR default; document the new --enable-dcr-insecure escape hatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(todos): add takes_search + code_def to the federated by-slug P1 (#2200) The v0.42.55.0 eng-review codex pass flagged takes_search (holder-allowlist only) and code_def (brain-wide raw SQL over content_chunks) as remaining same-class surfaces. Noted on the existing #2200 P1 follow-up, with the caveat that the #2399 close-list deliberately keeps #1371/#2200 open until this lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): correct plpgsql alias collision in #1385 BYPASSRLS gate (real-PG) The broadened BYPASSRLS preflight aliased `pg_roles r`, but several RLS DO-blocks already declare `r record` for their backfill FOR loop, so plpgsql resolved `r.oid`/`r.rolbypassrls` to the unassigned record variable → "record \"r\" is not assigned yet" on real Postgres (PGLite tolerated it; the DATABASE_URL-gated e2e jobs are the backstop). Renamed the subquery alias to `pr` at all 10 migrate.ts sites; also broadened the schema.sql base RLS gate the same way (with the `pr` alias) for #1385 consistency on superuser fresh installs, and regenerated schema-embedded.ts. Also fixes a PRE-EXISTING engine-parity bug (confirmed failing on clean origin/master): the relationalFanout shape compared `canonical_chunk_id`, a serial id that diverges between a fresh PGLite engine and a shared Postgres DB (setupDB TRUNCATEs without RESTART IDENTITY). Compare its presence, not the exact value. Validated on real Postgres (pgvector/pg16): migration v120 applies, the v35 RLS backfill runs, and engine-parity + postgres-bootstrap + jsonb-parity are green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
814258dda6 |
v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard (#2375)
* fix(sync): op_checkpoints pin write double-encodes jsonb — every sync aborts (#2339) recordCompleted bound JSON.stringify(array) to a $3::jsonb param via postgres.js .unsafe(), double-encoding it into a jsonb string scalar that violates the v119 op_checkpoints_completed_keys_array CHECK — aborting every multi-source sync on real Postgres at the first checkpoint write. PGLite parses the string silently, which is why unit tests stayed green and it shipped. Cast through $3::text::jsonb so the text->jsonb cast parses a genuine array. Adds a DATABASE_URL-gated parity test + a dedicated Postgres CI job so the guard can never silently skip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db): sweep positional jsonb double-encode sites + AST CI guard (#2324) Every executeRaw/.unsafe site that bound JSON.stringify(x) to a bare positional jsonb cast double-encodes on real Postgres (same class as #2339). Sweep them all to the text::jsonb form across query-cache, sources-ops, llm-base, calibration-profile, impact-capture, subagent, receipt-write, traversal-cache, symbol-resolver, and the agent/sources commands. Adds scripts/check-jsonb-params.mjs (AST-lite scanner for the positional form the legacy template grep misses, incl. generic-typed calls), wired into check-jsonb-pattern.sh, with a self-test. PGLite's native db.query is not scanned — it parses text to jsonb natively, so the bug can't occur there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(search,eval): alias-hop injected results carry page_id (contradiction-probe crash) applyAliasHop injected synthetic SearchResults without page_id (the `as SearchResult` cast hid the missing field), so listActiveTakesForPages bound undefined/NaN into ANY($1::int[]) and crashed the whole contradiction probe on real Postgres. Stamp page_id=page.id at the injection site and add a finite-id filter in generateIntraPagePairs as a defensive backstop (mirrors hybrid.ts:63). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(engines): positional jsonb binding rule (text::jsonb vs the double-encode trap) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard Bumps VERSION + package.json to 0.42.53.0, adds the CHANGELOG entry, and regenerates llms-full.txt. Ships the #2339 sync-abort hotfix, the repo-wide positional jsonb double-encode sweep, the alias-hop contradiction-probe crash fix, and the new positional-form CI guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: post-ship sync — jsonb invariant now covers the positional form + new guard CLAUDE.md JSONB invariant + KEY_FILES (sql-query, check-jsonb-pattern, op-checkpoint) now describe the #2339 positional double-encode class, the $N::text::jsonb fix, and the new check-jsonb-params.mjs guard. Regenerates llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bb2e88c42a |
v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984) (#2287)
* test(supervisor): pin LOCK_HELD fence-exit is never counted as a crash (#2227) A duplicate supervisor loses the queue-scoped DB singleton lock (#1849) and exits LOCK_HELD before spawning a worker or emitting 'started'. summarizeCrashes counts only worker_exited, so the fence path is structurally uncountable. Pin it so a future refactor that logs worker_exited on the fence path fails here instead of silently re-introducing the crash-budget breaker-trip loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(autopilot): per-source cycle binds FS phases to source.local_path, not global repo (#2194 #2227) A per-source autopilot-cycle inherited the global sync.repo_path as brainDir while stamping DB freshness for source_id — mixed scope. FS phases (sync/lint/extract) ran against the wrong tree, so the failure-cooldown and freshness gates would attribute work to the wrong source. Resolve the source's local_path in the handler (reuse the archive-recheck SELECT) and bind brainDir to it; a pure-DB source gets null (FS phases skip) instead of falling through to the global checkout. Legacy no-source dispatch keeps the global repoPath. Prerequisite for the cooldown/split commits (codex outside-voice #8). Resolves TODOS:634. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(supervisor): detect a live supervisor via the DB lock under split $HOME (#2227) jobs supervisor status + doctor read the HOME-derived pidfile, so a supervisor started under a different $HOME (keeper=/root vs ops=/data) read as 'not running' while healthy — the false signal that drives an operator to spawn a duplicate. Both surfaces now fall back to the queue-scoped DB singleton lock (#1849), the HOME-independent authority, when the pidfile shows nothing. New isLockHolderLive keys on lock freshness (ttl + heartbeat steal-grace), never process.kill, so PID reuse can't false-positive (pid-liveness-alone-pid-reuse). Status surfaces the holder host/pid + recorded concurrency/max-rss from the latest started event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(supervisor): degraded retry instead of permanent give-up on crash storm (#1994 #2227) max_crashes_exceeded gave up forever, so a transient DB-pooler blip that tripped the soft budget wedged the queue until a human restart (#2227's breaker-trips tail). Crossing the soft budget now enters degraded mode: keep respawning with capped exponential backoff (60s cap — a paced retry, not a hot loop) and emit a loud crash_budget_degraded health_warn. The existing stable-run reset clears the count once a respawn survives >5min, so a recovered DB self-heals. Permanent give-up fires only at a much-higher hard ceiling (maxCrashes × 10), tunable/disablable via GBRAIN_SUPERVISOR_HARD_STOP_CRASHES (0 = never). Resolves TODOS:92. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): clamp fan-out to worker concurrency + doctor warning (#2194) Fan-out resolved to 4 (Postgres) regardless of worker --concurrency, so surplus cycles queued behind the worker and raced the stalled-sweeper. Two fixes for the same mismatch: - resolveEffectiveFanoutMax clamps to max(1, concurrency-1) (reserve a slot), gated on a LIVE DB-lock holder so a stale started-audit row can't shrink throughput (codex #9/D5); no live holder → unknown → unclamped base. Escape hatch autopilot.fanout_clamp_to_concurrency. - doctor's autopilot_fanout_concurrency check warns when fan-out exceeds effective slots — the misconfig was silent before. Advisory (started-event concurrency), wired into both doctor surfaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): per-source failure cooldown — break the dead-job storm (#2194) Only SUCCESS gated dispatch, so a source whose cycle kept failing/timing-out re-fanned-out every 5-min tick forever (200+ dead jobs/24h). Now a failed source backs off with bounded exponential cooldown (10→120min). Read at DISPATCH from minion_jobs dead/failed rows (timeouts/RSS-kills dead-letter via SQL and never run handler code, so a write-only hook would miss them) AND re-checked at CLAIM time in the handler (codex #5: already-queued/retrying jobs). A success clears it (codex #7); null-source rows excluded (codex #6); engine-parity via executeRaw. Disable with autopilot.failure_cooldown_min=0. Fail-open if config/history reads error. Surfaced via fanout_cooldown_skipped + the fanout summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): split the cycle — per-source phases + one global-maintenance job (#2194 #2227) N per-source cycles each ran the brain-wide global phases (embed-all/orphans/ purge/…) concurrently, thrashing the same rows and taking the worker 4→10GB in <60s → RSS-kill → orphaned stalls. Split them: per-source jobs now run only source-scoped (+ mixed) phases and stamp last_source_cycle_at; a new autopilot-global-maintenance job runs the global phases ONCE per window (idempotency_key + maxWaiting:1 = structural single-flight) and stamps autopilot.last_global_at. This is the codex-endorsed design that replaced the rejected skip-and-stamp-fresh approach (codex #1/#2): no freshness poisoning, no starvation — global work always runs as its own job, never marked done when it wasn't. PHASE_SCOPE is now a runtime partition (GLOBAL ∪ NON_GLOBAL == ALL). last_full_cycle_at still written for doctor/legacy (no longer a global gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): guard nullable engine in supervisor DB-lock fallback (#2227) Follow-up to the supervisor-visibility commit: doctor's engine binding is BrainEngine | null, so the inspectLock fallback must guard on a non-null engine (tsc TS2345). No behavior change — a null engine simply skips the DB-lock probe and falls back to the pidfile reading, as before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): categorize autopilot_fanout_concurrency check as ops (#2194) Follow-up to the fan-out/concurrency commit: the doctor-categories drift guard requires every check name in doctor.ts to belong to exactly one category set. Add the new autopilot_fanout_concurrency check to OPS_CHECK_NAMES (infrastructure liveness, alongside wedged_queue/supervisor). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update KEY_FILES for the autopilot cycle split + supervisor degraded-retry (#2194 #2227) Post-ship document-release: refresh the KEY_FILES current-state entries that drifted — cycle.ts (GLOBAL/NON_GLOBAL phase split + last_source_cycle_at / autopilot.last_global_at), jobs.ts (per-source local_path brainDir, claim-time cooldown, autopilot-global-maintenance handler), supervisor.ts + child-worker (degraded retry instead of permanent give-up; hard ceiling), db-lock.ts (isLockHolderLive), handler-timeouts (new handler). Regenerated llms bundle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minions): handleTimeouts counts the timed-out run as a spent attempt (#1737) The per-job timeout_at dead-letter (handleTimeouts) set status='dead' without incrementing attempts_made, unlike the wall-clock and stall dead-letter siblings. It is the FIRST killer to fire for the long-lane handlers (subagent / embed-backfill / autopilot-cycle) because timeout_ms is stamped at submit, so a timed-out long job reported `attempts: 0/N (started: N)`. Mirror the siblings with attempts_made + 1 (terminal, no retry). Safe against double-count: the worker sweep runs handleStalled -> handleTimeouts -> handleWallClockTimeouts sequentially and awaited, each guarded on status='active', so the first to dead-letter excludes the row from the rest. Regression assertions added (test/minions.test.ts + e2e/minions-resilience.test.ts) so the increment can't be silently dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(agent): recognize trailing switches in `agent run`, keep prompts freeform (#1738) parseRunFlags() broke flag parsing at the first positional token, so any flag after the prompt (`gbrain agent run "do X" --detach`) was swallowed into the prompt string and silently ignored. Now the no-value switches --detach/--follow/ --no-follow are hoisted when they trail the prompt, while everything else stays verbatim: an unknown --word is treated as prompt text (no "unknown flag" throw), a --switch mid-prompt is preserved, and `--` suppresses hoisting entirely for a literal escape. Value-flags now reject a missing or flag-shaped value (and --max-turns/--timeout-ms a non-number) instead of capturing undefined/NaN. Contract change: a prompt that starts with or trails an unguarded --word no longer errors; a literal trailing --detach needs `--`. Help text updated; tests revised + extended (test/agent-cli.test.ts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): honest live-sync status + progress-aware stall-abort (#1950) Finishes the #2255 honest-freshness story for two gaps it left. (a) `gbrain sources status` printed "idle" while a sync proc held the per-source lock (the reported bug). New shared liveSyncStatus() helper in db-lock.ts reads the SAME live-lock signal `gbrain doctor` uses; runStatus now shows "running" (BACKFILL column + a sync_running field in --json) and suppresses the misleading "never synced" warning while a sync is live. One helper, so the surfaces can't drift (doctor/status retrofit tracked as a follow-up). (b) A sync wedged-but-alive kept refreshing its lock heartbeat (it fires on its own timer) and hadn't hit the wall-clock deadline, so only a manual pkill freed it. New in-band stall watchdog keys off FORWARD IMPORT PROGRESS (progress.tick), not the heartbeat: if no file completes for GBRAIN_SYNC_STALL_ABORT_SECONDS (default 900s), it aborts via a controller composed into opts.signal, so the drain returns partial() (last_commit unchanged, next run resumes from the checkpoint) and withRefreshingLock releases the lock. Limits, documented in code: a single file slower than the window trips it; a fully starved event loop won't fire the timer (the wall-clock hard deadline is that backstop). Tests: liveSyncStatus (live/expired/none/per-source) in db-lock-inspect; the resolveStallAbortSeconds env matrix in sync-hard-deadline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(status): version field + per-section --deadline-ms budget (#1984) `gbrain status` had no version in its JSON envelope and could hang on a slow connection with no way to get a partial answer. Two additions: - version: the StatusReport JSON now carries the local gbrain CLI version so a poller can pin behavior to a build. Thin-client also surfaces remote_version (the brain server's version), and the get_status_snapshot MCP op reports its version for that parity. - --deadline-ms=N / --fast: a shared wall-clock budget. Each section is raced against the REMAINING budget via Promise.race (NOT process-watchdog, which SIGKILLs and can't return partial output), so one slow/hung section can't strand the snapshot — it's marked stale and the rest still return. The envelope gains partial:true + stale_sections[]; exit code stays 0 (a snapshot was produced). Invalid --deadline-ms → exit 2. Tests: parseDeadlineFlag + withSectionDeadline (hermetic), the usage-error exit, version presence in the PGLite envelope, and the op's version key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): report stall_timeout distinctly + document in-flight limit (#1950) Pre-landing review (codex + adversarial): the stall watchdog aborted opts.signal but the per-iteration abort checks returned partial('timeout'), collapsing a wedge-reap into a user --timeout/SIGINT so JSON consumers couldn't tell them apart. Add a 'stall_timeout' reason (set via a stallAborted flag) on the three import-loop abort sites; deletes/renames-phase and checkpoint sites stay 'timeout'. Sharpen the watchdog comment: the abort is observed BETWEEN files, so a hang inside a single importFile is not interrupted until it returns (TODO: thread a cancellation signal through importFile). * fix(agent): `--` escape suppresses trailing-switch hoisting anywhere (#1738) Pre-landing review: the leading-flag loop breaks at the first positional, so the `escaped` flag only fired for a leading `--`. A `--` placed after a positional left trailing-switch hoisting active, so `agent run note -- body --detach` silently detached and dropped the `--` as junk. Suppress hoisting whenever a literal `--` appears in the prompt. Regression test added. * fix(status): deadline-ms usage-error + scoped stale_sections + cancel losing remote call (#1984) Pre-landing review (codex): (1) bare `--deadline-ms` with no value silently fell through to no-budget/--fast instead of a usage error; (2) thin-client timeout reported both sync+cycle stale even under `--section sync`, naming a section the caller excluded (local path was already correct); (3) the section race abandoned the remote promise locally but didn't cancel the in-flight MCP call — pass the budget as timeoutMs so the losing side actually cancels. Regression test added. * v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984) Bundles the already-reviewed autopilot/supervisor stabilization (#2194 #2227 #1994: cycle split, per-source failure cooldown, fan-out clamp, degraded supervisor retry, DB-lock live-supervisor detection) with four operational fixes: minion timeout attempt-accounting (#1737), agent-run trailing-flag parsing (#1738), honest live-sync sources status + progress-aware stall watchdog (#1950), and status version + --deadline-ms partial result (#1984). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document GBRAIN_SYNC_STALL_ABORT_SECONDS env knob (#1950) Post-ship doc sync (/document-release): add the sync stall watchdog env var to the CLAUDE.md sync-tuning table (Five → Six knobs) + regenerate the llms bundle. * test: quarantine #2249 fanout tests as *.serial (R1 env-isolation) (#2194) The cherry-picked autopilot-fanout-clamp + doctor-autopilot-fanout-concurrency tests mutate process.env.GBRAIN_AUDIT_DIR in beforeEach/afterEach, which the check:test-isolation R1 lint flags (parallel shards load multiple files per process). Rename to *.serial.test.ts (sanctioned quarantine — they run under --max-concurrency=1) instead of restructuring the reviewed test bodies. No logic change; both files stay green (9 tests). Fixes the failing verify CI check. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9bf96db807 |
v0.42.51.0 fix(sync): contention-free clock + checkpoint integrity + honest sync freshness (#2255)
* fix(sync): contention-free page-generation clock — sequence swap The page-generation clock backed the query-cache Layer-1 bookmark via a FOR EACH STATEMENT trigger running `UPDATE page_generation_clock SET value=value+1 WHERE id=1`. That took a transaction-length RowExclusiveLock on one tuple, so every concurrent page writer serialized on the prior writer's COMMIT — sync ran at ~0.8 cores regardless of worker count. Swap to a SEQUENCE bumped by nextval() (a microsecond LWLock, never a row lock). The clock's only contract is monotonic advancement on any page INSERT/UPDATE/DELETE; last_value is non-transactional, so rolled-back or concurrent-uncommitted writers only OVER-invalidate the cache (lose a hit), never serve stale. - migration v118: CREATE SEQUENCE + load-bearing 2-arg setval (is_called= true, floor 1, seeded >= old clock and MAX(generation)) + repoint the trigger function body + DELETE query_cache so no old-clock bookmark survives the swap. v107 left immutable. - query-cache-gate.ts: 3 readers -> SELECT last_value FROM page_generation_clock_seq. - schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the sequence on fresh install; table + trigger names retained. - tests: clockValue reads last_value; mechanism proof (trigger fn uses nextval not the row UPDATE); rollback-advances-clock safety pin; real PGLite sequence round-trip (is_called gotcha); shape test requires _seq. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): op_checkpoints array-shape guard — CHECK + repair + defensive loader completed_keys is JSONB and the checkpoint loader runs jsonb_array_elements_text over it. A non-array (scalar) value makes that throw "cannot extract elements from a scalar", which takes down the whole UNION load — including the valid op_checkpoint_paths child rows — and loses all checkpoint progress for that key. No current writer produces a scalar, but an older binary / external script / future bug could. Make the corruption class structurally impossible and self-healing: - migration v119: LOCK TABLE (so an out-of-band scalar can't land between repair and constrain; no-op on single-connection PGLite), repair any pre-existing scalar to '[]' (op_checkpoint_paths child rows are the append-only source of truth, so the reset loses nothing), then add the named CHECK (jsonb_typeof(completed_keys) = 'array') via a pg_constraint IF NOT EXISTS guard. A DB-enforced always-on guard — the correct pattern vs a migration verify-hook, which never runs on already-stamped brains. - schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the same NAMED inline CHECK so fresh installs match migrated brains and v119 skips the duplicate. - op-checkpoint.ts loader: gate the legacy arm on jsonb_typeof = 'array' so a scalar parent is skipped (children still load) instead of throwing the whole union, and log a specific corruption warning when one is seen. - tests: CHECK rejects a scalar (exactly one constraint, no blob+migration dupe); loader survives a scalar parent and returns the children; v119 repair converts a scalar to '[]'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): report actively-running sync via live lock, not stale freshness A slow source that makes partial progress every cycle but never fully completes used to read as permanently "stale" / "never synced" because last_sync_at only advances on a full successful sync. The naive fix (treat recent checkpoint banking as "in progress") is unsafe: a blocked sync banks the good files then writes no anchor, so banking can't tell in-progress from wedged. Use the only honest signal: a LIVE, non-expired per-source sync lock (inspectLock + syncLockId against gbrain_cycle_locks). Every non-skipLock sync holds it and refreshes it; a blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either correctly falls through to the stale path and is NEVER masked. An actively-syncing source (including a never-synced source doing its first sync) counts as synced_recently, preserving the pinned 3-bucket invariant. The lock lookup reuses doctor's existing dynamic db-lock import and swallows any throw (stub engine, pre-lock-table brain) to false, so it can only ADD an in-progress verdict, never suppress a real stale one. Tests (real PGLiteEngine + real lock rows): stale+no-lock -> fail; stale+live-lock -> ok; never-synced+live-lock -> ok; never-synced+no-lock -> fail; expired-TTL lock -> fail (wedged not masked); blocked source with banked checkpoint rows but no lock -> still fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): honest --force-break-lock diagnostic when no lock is held --force-break-lock used to emit the same terse "Lock ... is not held (nothing to break)" line and exit 0 even when a sync was genuinely wedged, sending the operator down a dead end — the wedge was not a held lock. Keep rc=0 (breaking a non-existent lock is idempotently successful; flipping the exit code would break automation), but under --force say plainly that nothing was broken and point at the real next step (gbrain sync / gbrain doctor) plus a `wedge_hint` field in --json output. The non-force path is byte-for-byte unchanged. runBreakLock is exported for the test. Tests: force+no-lock -> wedge_hint JSON + human hint, rc 0; non-force+no-lock -> unchanged terse line, no hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): surface the in-progress sync holder in the freshness message Plan-completion follow-up to the BUG 4 live-lock signal: when a source is actively syncing, name the holder (pid + host) in the check message instead of silently folding it into synced_recently. The note is appended only when something is in progress, so steady-state messages stay byte-for-byte unchanged (the pinned exact-message + 3-bucket-invariant tests still pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): pre-landing review fixes — monotonic clock seed, scoped CHECK guard Adversarial (codex) review of the implementation diff caught three: - P1 (correctness): the fresh-schema setval was not monotonic. initSchema replays the schema blob, and the unconditional setval(MAX(generation)) could move page_generation_clock_seq.last_value BACKWARD on an already-upgraded brain, letting a stored query_cache bookmark serve stale rows. Seed via GREATEST over the sequence's OWN last_value (+ old table value + MAX(generation)) in all 3 fresh schemas and migration v118, so a replay is idempotent — mirrors the old table's ON CONFLICT DO NOTHING. Pinned by a new monotonic regression test. - P2: v119's CHECK-exists guard keyed on conname only (not globally unique). Scope it to conrelid = 'op_checkpoints'::regclass. - P3: in-progress note ran into the prior sentence in fail/warn doctor messages; separate it with '. '. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make Anthropic/ZE no-key tests hermetic against a dev config key These "no key" tests cleared only ANTHROPIC_API_KEY / ZEROENTROPY_API_KEY from the env, but hasAnthropicKey() and checkZeEmbeddingHealth() also read the key from ~/.gbrain/config.json. On a dev machine whose real config holds a key, the no-key assertions flipped and the tests failed locally (they passed only in key-less CI). Add a shared with-env emptyHome() helper and point GBRAIN_HOME at an empty dir in every no-key path so loadConfig finds nothing — matching the already-hermetic anthropic-key / gateway-probe tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.44.1.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): sync doctor + op-checkpoint entries to v0.44.1.0 truth checkSyncFreshness now reports an actively-running sync via the live per-source lock (names holder pid+host, counts as synced_recently) instead of flagging it stale; loadOpCheckpoint gates the legacy union arm on jsonb_typeof = 'array' so a scalar parent can't take down the whole load, and migration v119's CHECK constraint makes the corruption class structurally impossible. Reference docs describe current behavior only — both entries updated in place, no release-clause appends. Guard + llms freshness test green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-version to v0.42.51.0 (natural next-off-master) Maintainer override of the queue allocator's leap to 0.44.1.0 (it jumped past in-flight sibling PR claims at 0.42.50/0.43.0/0.44.0). Take the natural next slot in the 0.42.x line above the immediate sibling claim (0.42.50.0); a merge re-bump resolves any collision if a cathedral PR lands first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): bound + retry the OpenClaw install so a transient npm hang can't burn the Tier 2 budget The Tier 2 (LLM Skills) job failed at 30m16s — the `npm install -g openclaw@2026.4.9` step hung on a transient npm/registry stall (orphan `npm install openclaw` was still running at cancel time) and consumed the entire 30m job budget that v0.42.50.0 (#2254) introduced. The install normally finishes in under a minute (Tier 2 is ~4m end to end on master), so this is flaky-install infra, not a test failure. Wrap the install in `timeout 120` + a 3-attempt retry loop with an 8-minute step backstop: a hung attempt is killed in 2 min and retried instead of eating the whole job. Same bound-the-hang philosophy as #2254's job timeouts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
70d5f36db6 |
v0.42.50.0 ci: reliability hardening — cancel-superseded + per-job timeouts + actionlint + hermetic E2E env (#2254)
* ci: cancel superseded runs + per-job timeouts on test.yml & e2e.yml Ports the GH-Actions hygiene gbrain already uses in heavy-tests.yml to the two hot-path workflows. concurrency cancels a superseded run (keyed on PR number for pull_request events — fork-safe — with github.ref fallback for push/scheduled); frees runners and stops a stale-SHA run reporting a flaky failure on an obsolete commit. Per-job timeout-minutes (test matrix 15, verify 12, serial 15, slow-* 12, e2e tier1 20 / tier2 30, trivial jobs 5-10) convert a wedged job from a 6-hour zombie (GitHub's default) into a fast legible fail. fail-fast:false already set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: add actionlint workflow (rhysd/actionlint v1.7.11, SHA-pinned) Lints workflow YAML on .github/workflows/** changes so a malformed workflow / bad action ref / missing-permission bug is caught before it ships a broken pipeline. gbrain edits these workflows often; cheap preventive guard. Mirrors GStack's actionlint job, SHA-pinned to gbrain's convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(e2e): scrub operator/agent env before E2E (hermetic runner) A dev or Conductor shell exports CONDUCTOR_*/MCP_*/OPENCLAW_*/GBRAIN_* overrides that silently change test behavior, making hermetic E2E non-hermetic and its failures unreproducible across machines. run-e2e.sh drops those prefixes before bun starts (denylist — PATH/HOME/TMPDIR/DATABASE_URL survive; GBRAIN_HOME kept for the existing HOME isolation). Adapts GStack's buildHermeticEnv to gbrain's shell runner. Verified: a 78-test e2e file passes with DATABASE_URL surviving + a planted GBRAIN_BRAIN_ID scrubbed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.50.0 ci: reliability hardening — cancel-superseded + per-job timeouts + actionlint + hermetic E2E env Ports GStack's CI-reliability hygiene to gbrain's hot-path workflows: concurrency cancel-in-progress (PR-number keyed), per-job timeout-minutes (no more 6-hour zombie jobs), an actionlint workflow, and an operator-env scrub in run-e2e.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
7968f84077 |
v0.42.49.0 feat(pace): native DB-contention pacing for embed/sync backfills (#2240)
* feat(pace): composable DB-contention pacer primitive createDbPacer/createNoopPacer (concurrency permit + in-band EWMA + jittered cooperative sleep, abort-throws, fail-open) + named pace-mode bundles (env>config>bundle, default off) + shared embed-backfill lock key. * feat(embed): wire DB-pacing into embed paths + single-flight + bounded keyset re-entry embedStaleForSource + CLI embedAllStale/embedAll lower worker count to the resolved cap and observe()/pace() their DB ops; embed job + embed-backfill handler resolve env>config>bundle; CLI --pace flags; --background carries overrides into the job payload; single-flight via shared per-source lock; budget-timer re-arm around paced sleeps; EmbedResult.pacing telemetry. * feat(sync): shared DB-pacer permit across parallel worker engines One pacer spans the per-worker PostgresEngines (the multi-pool permit case); observe() import writes, pace() between files, dispose on all exit paths. * docs(pace): CLAUDE.md Pace Mode section + regenerated llms bundle * fix(pace): pre-landing review fixes (Codex P1/P2) - never unref() the cooperative-sleep timer (could exit mid-sleep) - pace() excludes the wall-clock budget + re-arm after each sleep - pacing only lowers concurrency, never raises above an operator cap - serialized job pace resolves at config tier so GBRAIN_PACE_* still wins - --pace-max-concurrency consumes its value token * chore: bump version and changelog (v0.42.48.0) Native DB-contention pacing for embed/sync backfills. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version to v0.42.49.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
7ea92d602c |
v0.42.48.0 feat(durability): auto-harden brain repos for git durability on PAT+URL (#2241)
* feat(git): divergence-safe pull, push-probe, default-branch detection for brain durability Add GIT_ENV_AUTH + divergenceSafePull (skip-on-dirty, conflict-abort-clean, never-mid-rebase), detectDefaultBranch, pushProbe, and an env-gated GBRAIN_GIT_ALLOW_FILE_TRANSPORT escape hatch. Export GIT_ENV. pullRepo's --ff-only contract is unchanged. * feat(durability): brain-repo hardening core (hook, helper, cron, PAT, AGENTS rules) hardenBrainRepo/unhardenBrainRepo: local untracked post-commit hook + committed brain-commit-push.sh (one shared push-retry template), repo-scoped credential with existing-helper reuse, push-probe verify, active-resolver-file rules with taxonomy from _brain-filing-rules.json, minimal DB-free pull cron. PAT redaction via redactSecretsInText. * feat(sources): harden/pull/unharden commands + auto-harden on add --url sources harden/pull/unharden subcommands; --pat-file/--no-harden on add; auto-harden managed clones on add; unharden-before-remove. cli.ts pre-connect early-exit for DB-free 'sources pull --path' (the cron entry, never opens PGLite). * test(durability): unit + integration coverage for brain-repo durability git helpers, core harden/unharden, hook+helper E2E (real background push), cron generators. 41 tests across 4 files. * chore: bump version and changelog (v0.42.48.0) Brain-repo git durability: auto-harden a brain's working tree (local auto-push hook, committed commit-push helper, always-on agent rules, DB-free pull cron, repo-scoped credential, push-probe verify) the moment gbrain gets a PAT + URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sources): route harden exit code through setCliExitVerdict A raw process.exitCode write is zeroed by the owned-verdict flush-exit (#2084 PGLite-Emscripten pollution defense); cli-exit-verdict-pin guard caught it. Use setCliExitVerdict(3) so 'sources harden' actually reports needs-attention to cron/automation. * docs: document brain-repo durability (KEY_FILES + multi-source guide) KEY_FILES: extend git-remote.ts entry (divergenceSafePull, pushProbe, detectDefaultBranch, GIT_ENV_AUTH, GBRAIN_GIT_ALLOW_FILE_TRANSPORT) + add brain-repo-durability.ts/sources-harden.ts entry. multi-source-brains.md: add a Durability (auto-harden) how-to covering sources harden/pull/unharden, --pat-file, the guarantees, and the security posture. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9d88680a51 |
v0.42.47.0 feat(skillpack,advisor): brain-resident skillpacks + proactive gbrain advisor (#2180) (#2231)
* feat(skillpack): brain_resident manifest fields + init-brain-pack scaffolder + tools version-skew lint Add optional brain_resident/schema_pack to the v1 manifest (additive, forward-compatible). New runInitBrainPack scaffolds a brain-resident pack (brain_resident:true, exact gbrain_min_version, 5-section machine-parseable README) beside brain content; applyWritePlan factored out of init-scaffold. brain-pack-lint validates each skill's declared tools: against the serving op set (E6 version-skew). Wires gbrain skillpack init-brain-pack. * feat(skillpack): Topology A brain-pack discovery on sources add + bounded nag After 'gbrain sources add', if the source ships a brain_resident pack, print an agent-readable advisory (ask the user before scaffolding). nag-state.ts tracks declines per (source-repo brain_id, source, pack) with escalate-then-suppress; declines count only on CLI-interactive displays, never cron/MCP. Fail-open: a malformed/absent pack never breaks sources add. * feat(advisor,skillpack): list_brain_skillpack MCP tool + gbrain advisor Topology B: dedicated source-scoped list_brain_skillpack op + get_skill source_id disambiguation (brain-resident-locate.ts); git scaffold-spec never a server FS path; source-aware schema match. LEARN_INSTRUCTION + serve-http banner. gbrain advisor: read-only ranked actions from brain state (8 resilient collectors, shared renderer, JSONL history, --json severity exit codes, local-only argv --apply dispatcher). Exposed over MCP behind mcp.publish_advisor (default off, read-only on remote; workspace collectors no-op remotely). Generalizes post-install-advisory to a single current-state recommended set (install→scaffold). * feat(skills): bundle gbrain-advisor skill + weekly cron recipe + ranking eval skills/gbrain-advisor teaches a harness to run gbrain advisor on a cadence and ping the user (read-only; ask before fixing). Registered in manifest.json, RESOLVER.md, openclaw.plugin.json. E4 ranking-precision eval on seeded-defect fixtures (100%). * chore: bump version and changelog (v0.42.47.0) Brain-resident skillpacks + gbrain advisor (#2180). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync CLAUDE.md + KEY_FILES for brain-resident skillpacks + advisor (#2180) Skill count 29->30, Skills section gains the brain-resident skillpacks + advisor capability, KEY_FILES gets current-state entries for the new modules. Regenerate llms bundles. * test,fix: align stale assertions with generalized advisory + advisor resolver triggers (#2180) - post-install-advisory.test.ts: install→scaffold wording (the install verb was removed); restore two-column in book-mirror copy; drop the removed skillpack-list line. - RESOLVER.md: gbrain-advisor trigger now fuzzy-matches a declared frontmatter trigger (resolver round-trip D5/C). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: regenerate llms bundle for updated advisor resolver row (#2180) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
c023a6041d |
v0.42.46.0 fix(engine): federated read scope reaches by-slug reads (#2200) (#2239)
* fix(engine): federated sourceIds[] scope on by-slug secondary reads (#2200) getTags/getLinks/getBacklinks/getTimeline (both engines) + TimelineOpts now accept a federated `sourceIds[]` read grant, precedence over scalar sourceId, filtering `source_id = ANY($::text[])` — mirroring getPage from v0.42.37.0. - getTags: `page_id = (subquery)` -> `IN (subquery)` + DISTINCT so a slug present in >1 granted source unions tags instead of throwing on a multi-row subquery. - getLinks/getBacklinks: federated branch scopes ALL THREE page endpoints (from, to, AND the authoring origin) so a cross-source link can't disclose a foreign slug. Scalar/unscoped branches unchanged (trusted internal callers keep the cross-source view). - getTimeline: Postgres 8-branch cartesian tree collapsed to one fragment-composed query; PGLite adds the sourceIds branch to its dynamic WHERE. * fix(ops): route by-slug reads through the federated source scope (#2200) get_page resolves tags against the concrete page's source; get_tags/get_links/ get_backlinks/get_timeline route through sourceScopeOpts(ctx) (replacing the copy-pasted scalar `ctx.sourceId ? {sourceId} : {}`). New linkReadScopeOpts promotes an UNTRUSTED remote scalar scope to sourceIds[] so legacy/pre-federated tokens also get all-endpoint link scoping; trusted local CLI keeps cross-source. * test: federated read scope on by-slug reads + engine parity (#2200) Per-op federated reads, isolation (out-of-grant -> empty), cross-source decoy guard, far-endpoint + origin leak guards (F1), same-slug union (D3A), empty-array contract, scalar-remote promotion (D1), getTimeline date-window after the Postgres fragment refactor (D5A), and engine-parity arms for all four methods. * chore: bump version and changelog (v0.42.46.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): document federated by-slug read scope + linkReadScopeOpts (#2200) The #2200 fix routed get_page tags + get_tags/get_links/get_backlinks/ get_timeline through the federated source scope and added sourceIds[] to the engine read methods + TimelineOpts. Bring KEY_FILES.md to current state: the operations.ts entry's sourceScopeOpts read-op list now includes the by-slug reads and documents the linkReadScopeOpts helper (three-endpoint link scoping + untrusted-remote scalar promotion); the engine.ts entry notes the by-slug read methods + TimelineOpts carry the same sourceIds[] federated axis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
5c49225e4b |
v0.42.45.0 feat(sync): delta-aware cost estimator — stop wedging the daily cron (#2139) (#2224)
* feat(core): shared computeSyncDelta + spend-posture module (#2139) sync-delta.ts: ONE implementation of "what changed since last_commit", consumed by both the sync executor and the inline cost estimator so the gate's dollar figure can't drift from what the sync imports. spend-posture.ts: spend.posture config + parseUsdLimit/formatUsdLimit off-switch parsing (off/unlimited/none → Infinity; undefined at the budget boundary so ledger rows never serialize null). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sync): delta-aware cost estimator + non-TTY auto-defer + per-source failure acks (#2139) The inline-embed cost gate was a ~400x phantom: it priced the entire tree whenever the working tree was dirty (always, on an active brain), then blocked the daily cron with exit 2. Now: - performSyncInner + the estimator both route through computeSyncDelta, so the estimate mirrors execution (fetch-first delta; dirty-but-caught-up tree → $0). - shouldBlockSync is posture-aware; non-TTY above floor AUTO-DEFERS embeds to capped backfill jobs (exit 0) instead of wedging — single shared runInlineCostGate on both --all and single-source paths. - --full prices delta + stale backlog (full sync sweeps it inline). - off/unlimited on the cost knobs; tokenmax bypasses the backfill cap (still ledgered) but never the cooldown. - --skip-failed/--retry-failed scoped per source; the D15 parallel refusal is lifted (the #1939 ledger is per-source + lock-serialized). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(config): register spend-control keys + validate spend.posture (#2139) Adds spend.posture + the five previously --force-only spend knobs to KNOWN_CONFIG_KEYS so `config set` accepts them directly (removes the archaeology the issue complained about), and rejects invalid spend.posture values at set time with a paste-ready hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reindex,enrich,onboard): spend.posture across the remaining cost gates (#2139) reindex-code: tokenmax makes the cost gate informational; --max-cost accepts off/unlimited. enrich + onboard --auto: tokenmax lifts the refuse-without-cap guardrail and runs UNCAPPED (spend still ledgered by BudgetTracker). Explicit --max-usd always wins over posture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cost-gate, delta estimator, spend-posture, off-switch coverage (#2139) New sync-delta + sync-cost-estimate unit suites; rewritten cost-gate serial tests (auto-defer instead of exit 2, posture, off-switch, format split, single-source); parseUsdLimit/posture-aware shouldBlockSync; backfill cap-off + tokenmax-bypass + cooldown-still-refuses; config known-key acceptance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spend-controls): single spend-control surface + ref-map + follow-up TODOs (#2139) New docs/operations/spend-controls.md (every gate, key, default, off switch, posture interaction); CLAUDE.md reference-map row; two P3 follow-up TODOs (measured chunk-count gating, per-source defer granularity). llms bundles regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(spend): SSRF-harden estimator fetch + complete off/uncapped across reindex/enrich/onboard (#2139) Ship-stage codex pre-landing review caught four P1s in the secondary cost gates: - The delta estimator's fetch-first ran `git fetch` through the plain git() helper, bypassing the GIT_SSRF_FLAGS + GIT_TERMINAL_PROMPT=0 hardening that real sync uses. Added `fetchRemote()` to git-remote.ts (same flags as pullRepo) and route the estimator through it — a cost preview / dry-run can no longer hit a remote through a less-protected path. - `reindex --max-cost off`, `enrich --max-usd off`, `onboard --auto --max-usd off` were parsed but didn't actually proceed/uncap. Now: explicit off (and spend.posture=tokenmax) proceed past the confirmation/missing-cap refusal AND run uncapped. enrich threads an Infinity sentinel mapped to "no BudgetTracker ceiling" (never raw Infinity → no null in audit rows); reindex/onboard use their native undefined=uncapped path. Spend still ledgered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.45.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(KEY_FILES): update sync/embedding/git-remote/reindex entries to post-#2139 truth document-release pass: the cost-gate entries described the pre-#2139 behavior (full-tree-ceiling estimator, --skip-failed-rejects-under-parallel, exit-2 confirmation gate). Updated to current truth — delta-aware estimator via the shared computeSyncDelta, per-source failure acks under parallel, non-TTY auto-defer (no exit 2), posture-aware shouldBlockSync. Added entries for the two new core modules (sync-delta.ts, spend-posture.ts) + fetchRemote on git-remote.ts + reindex --max-cost off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
090bb53203 |
v0.42.44.0 docs(tutorial): point AlphaClaw deploy link at the official site (#2165) (#2171)
* docs(tutorial): point AlphaClaw deploy link at the official site (#2165) Step 4 of the personal-brain tutorial linked to the wrong top-level domain for AlphaClaw. Corrected so the deploy step works as written. * chore: bump version and changelog (v0.42.44.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
a81f7e05e8 |
v0.42.43.0 feat(context): push-based context (#2095) + teardown-exit hardening (#2084) (#2175)
* fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084)
Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering
sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the
bound raced past — keep Bun's event loop alive, so every `gbrain query` paid
a flat 10s tax exiting via the hard-deadline force-exit banner.
Three changes, one contract:
- flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the
command is not a daemon, exit deliberately — after stdout AND stderr drain
(writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a
blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the
regression class; pinned by a 256KB real-pipe subprocess test.
- drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites
(op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation
x3, ze-switch, dream, read-only timeout path). Drains the background-work
registry, then disconnect (best-effort), bounded by the 10s unref'd
hard-deadline — which is now armed around the TEARDOWN window only, not
before the op handler (the old placement would have force-killed any op
slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites
previously skipped the drain entirely and had no hang timer at all.
- Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing
paths (status, friction, claw-test, smoke-test, eval cross-modal /
takes-quality replay / conversation-parser / whoknows-thin, status-thin)
become process.exitCode + return so they flow through the drains and the
flush-exit. Pre-engine usage/parse/refusal exits stay as-is.
BrainRegistry.disconnectAll deliberately unchanged: zero production callers
in src/, per-engine disconnects already bounded, and the kernel reclaims
sockets on exit (src/core/timeout.ts doctrine).
DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084)
Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs
verified only against one production deployment — CI had no transaction-mode
pooler and could never see the class. Now it can:
- docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting
postgres-1, mirroring the production split-pool topology (direct :5432 +
pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext
password in the userlist) + IGNORE_STARTUP_PARAMETERS for the
statement_timeout/idle_in_transaction_session_timeout startup params
gbrain's client sets (the Supabase pooler whitelists the same).
- test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url
into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs),
then spawns the real CLI against the POOLED url and asserts: exit 0,
stdout intact (the #1959 truncation class), and NO
"did not return within 10000ms — force-exiting" banner (pre-#2084 it
printed on 100% of query-shaped ops on this topology). Class bound, not
exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL.
- scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL +
GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases.
Verified live: both tests green against pgbouncer 1.25.2 in transaction mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095)
One row per page the brain volunteers (op / reflex / watch channels).
"Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at
(the existing bumpLastRetrievedAt write-back is the open/cite signal), so
there is no second tracking path. session_id/turn are nullable
caller-supplied attribution; rationale is a deterministic template string,
never raw conversation text.
- Migration v116 (idempotent) + mirrors in src/schema.sql +
src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also
folds in pre-existing comment-only drift from the v114 links edits).
- src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE
multi-row parameterized INSERT — never per-row awaited round-trips) +
purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains).
- Dream cycle purge phase prunes stale events alongside op_checkpoints /
brainstorm checkpoints / batch-retry audit files.
- RLS on Postgres comes from the v35 auto_rls_on_create_table event
trigger (the same mechanism that covered v110 page_aliases and v115
op_checkpoint_paths); the volunteer Postgres e2e pins it.
- No ::jsonb anywhere; no bootstrap probe needed (nothing references the
table pre-creation; writers guard with try/catch).
Tests: v116 shape + columns + indexes + live insert/purge round-trip on
PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095)
- entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing
per-turn extractor across the last N turns (oldest→newest), merges by the
normalizeAlias form with occurrence/newest-turn/user-mention metadata, and
orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES
cap drops stale assistant chatter, not the entity the user just named.
Closes the filed assistant-introduced-entities recall TODO; true pronoun
coreference (never-named antecedents) stays out of scope.
- retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence +
matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6)
lives next to the arm definitions so identity and score can't drift.
Arm-2 provenance is classified in JS (codex D8 — the combined OR can't
report which predicate matched). Federated sourceIds[] scope (alias arm
loops per source; arm 2 uses source_id = ANY — no engine-interface
change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for
windowing): the legacy title-whole-word rule would suppress every entity
merely MENTIONED in a prior window turn, breaking the feature by
construction — slugs only enter context when a pointer/page was actually
surfaced. Default stays 'slug-and-title' for the window=1 legacy path.
- volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF,
unprefixed → one user turn), volunteerContext (zero-LLM: extract →
resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate →
cap 3/5; deterministic rationale strings, never raw conversation text),
and volunteerUsageStats (per-arm/channel precision from the
last_retrieved_at join, labeled approximate — 5-min throttle false
negatives, unrelated-read false positives; codex D9).
Tests: 35 green across volunteer-context (window parsing, pronoun follow-up
via assistant-introduced entity, confidence gating, slug-only suppression,
takes-fence privacy, multi-source scope, caps, stats join math) +
retrieval-reflex back-compat + resolve-ipc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095)
New read-scope op on the contract surface (CLI `gbrain volunteer-context`
with stdin → window, MCP tool for free): takes a rolling conversation
window, returns confidence-gated page pointers with rationales + synopses.
`window` is optional-unless-stats (validated in the handler, codex D9);
`stats: true` returns the volunteered-vs-used precision summary, labeled
APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both
bias the join). Source scope threads through sourceScopeOpts — federated
grants narrow the volunteer to the granted sources.
Event logging is fire-and-forget through a new `volunteer-events`
background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked
dangling promise set + bounded drain + snapshot-drop on timeout so a
long-lived process never accumulates ghosts). ONE batched INSERT per call,
drained on every exit path by the commit-1 drain hoist; failure never
fails the op (pinned by an injected failing-engine test).
cli formatResult renders both shapes (pointer lines with confidence/arm/
rationale; the stats summary with per-arm precision).
Tests: op contract surface, window-required validation, sink round-trip
with session_id/turn attribution, failing-engine fail-open, federated
grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e
proving the op + sink + stats join AND that context_volunteer_events has
RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): reflex consumes the rolling window + ambient-channel logging (#2095)
The default-on retrieval reflex now extracts entities from the last N turns
(retrieval_reflex_window_turns, default 4; env
GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy
current-turn-only behavior exactly). assemble() passes the recent
user/assistant turns (hard cap 12); the reflex slices to the configured
window. Assistant-introduced entities and "what did she invest in?"
follow-ups whose antecedent was NAMED in the window now surface pointers —
the issue's "zero agent-initiated queries" success criterion on the
ambient path.
Under windowing, suppression switches to slug-only (codex D7): the legacy
title-whole-word rule would suppress every entity merely MENTIONED in a
prior window turn, breaking the feature by construction. Slugs only enter
prior context when a pointer/page was actually surfaced, so
already-surfaced pages still suppress. The suppression mode flows through
all three resolver rungs (host opts, serve IPC request, direct Postgres).
Ambient-channel feedback (codex D11): the server-side resolver paths
(serve IPC + direct Postgres) log volunteered pointers with
channel: 'reflex' through the drained volunteer-events sink, so
`gbrain volunteer-context --stats` measures the default-on path where most
volunteering happens. Host-injected resolvers (no gbrain engine) can't
log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the
pointer cap are unchanged.
Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only
vs already-surfaced suppression; throwing resolver stays fail-open
(16 green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): gbrain watch — push transport over stdin (#2095)
The issue's headline: the brain volunteers pages as the conversation flows,
instead of waiting to be asked. `some-transcript-feed | gbrain watch` reads
turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed
lines are user turns), keeps a rolling window (--window-turns, default 4),
and streams confidence-gated pointers with rationales to stdout (--json for
JSONL). Session dedupe rides the core's slug-only suppression — a slug is
volunteered at most once per session. Events log on channel 'watch' with
session_id + turn through the drained sink.
Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an
interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF —
so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1
placeholder): when main() resolves the work is over, the CLI_ONLY finally
drains volunteer events via drainThenDisconnect, and the entrypoint
flush-exit ends the process. Keeping it in the daemon set would have made
the piped EOF path hang on lingering sockets — the exact #2084 class.
SIGINT closes the stream and flows through the same drain path instead of
killing mid-write. Per-turn resolution failures are fail-open (the stream
never dies on a transient DB error).
Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) +
THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP
op) + main --help entry.
Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling
window via assistant-introduced entity, session dedupe, --json shape with
turn attribution, channel-watch event rows, --min-confidence gate, CRLF/
blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch`
on a fresh PGLite brain exits 0 at EOF with no force-exit banner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave
- docs/architecture/KEY_FILES.md (current-state): context entries gain the
window extractor, arm provenance/confidence, suppression modes, volunteer
+ volunteer-events modules; background-work entry now lists FIVE sinks and
the drainThenDisconnect owner-disconnect contract; new entries for
src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
the confidence model, CLI usage, config keys, and the approximate-stats
caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
check, structured messages[] param); the #1981 entity-detection TODO
narrowed (window extraction covered assistant-introduced entities +
named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
#2084 wave.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): truncate context_volunteer_events in setupDB (#2095)
The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows
persisted across e2e runs on a reused database and poisoned count/stats
assertions in volunteer-context-postgres on the second run. No FK to pages
(slug join), so position before pages is for hygiene only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084)
Caught by the full unit suite: `gbrain apply-migrations` on PGLite started
exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM
backend's proc_exit status into process.exitCode (initdb at create-time,
the postmaster at close-time — `exitCode=status` in pglite's dist), and
the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around
create/close (a guarded attempt verified this). The pre-#2084 success path
never read process.exitCode, so the pollution was invisible; the new
deliberate flush-exit propagated it faithfully.
Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in
cli-force-exit.ts — every gbrain-owned exit-code assignment routes through
the setter (still mirrored to process.exitCode for outside readers), and
both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect
hard-deadline backstop) read the getter. Swept all assignment sites:
cli.ts (op error, friction, claw-test, smoke-test, eval runners, status,
import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot.
Also updates the v0.42.20 structural pins to the drainThenDisconnect shape
(ordering invariant asserted INSIDE the helper + >=8 helper call sites,
superseding the two-inline-pairs assertion).
Verified: apply-migrations spawn test green; `init --migrate-only` exits 0;
an errored op still exits 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: re-pin the teardown-arming invariant at its post-#2084 home
Master's v0.42.41.0 triage wave and the #2084 wave fixed the same
pre-armed-timer bug independently; the merge keeps #2084's shape (arming
inside the shared drainThenDisconnect helper, covering all 8 exit paths).
The structural pin now asserts the same invariant — no pre-try arming;
gated, unref'd, before-drain, cleared — at the helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: coverage for ambient reflex-channel logging + watch window/cap flags
Ship coverage audit (85%, gate PASS) named five gaps; the two substantive
cheap ones close here: the codex-D11 logChannel='reflex' path now has a
behavioral pin (events land on channel 'reflex' through the drained sink;
no logChannel → no events), and gbrain watch's --window-turns / --max-pages
flags are exercised (turn-1 attribution under window=1; cap to one page).
Remaining flagged-not-blocking: the wallclock-timeout branch (untestable
without >10s real-clock flake — same rationale as the arming pin),
formatResult's volunteer case (module-private), and the cycle purge wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT
formatResult exported for tests (same import-safety contract as cliAliases);
test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message,
and approximate stats summary. test/watch-command.test.ts gains a real
subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream,
assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under
the actual signal, not just the shared exit path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin
The merged-state suite caught it: doctor --fast --json reported FAIL but
exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes
(doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit
silently zeroes — getCliExitCode() deliberately never reads ambient
process.exitCode (the PGLite-Emscripten pollution defense), so any setter
that bypasses setCliExitCode reports success on failure.
Swept both sites and added the structural class pin: a test greps src/ for
raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that
introduces one fails loudly instead of lying about exit codes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.43.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: quarantine the watch SIGINT subprocess test to the serial lane
The parallel unit shards flake on concurrent CLI subprocess spawns (failed
at 7ms in-suite, green solo) — same isolation rationale as
apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: update project documentation for v0.42.43.0
Post-ship doc verification against the release diff (#2095 push-based
context + #2084 superset hardening), with a cross-model doc review:
- push-context.md: version tag corrected to v0.42.43.0; per-call knobs
now cover prior_context/days and watch's flag surface accurately;
feedback-log writes described as best-effort; synopsis fence-strip
described as unconditional.
- CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context
release reference corrected to v0.42.43.0.
- KEY_FILES.md: ci-local entry rewritten to current topology (4-shard
parallel default, four Postgres services, transaction-mode PgBouncer
+ GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts
dropped from the selector entry.
- TESTING.md: inventory entries for the new #2084 structural pins
(cli-exit-verdict-pin, cli-pipe-truncation), the push-context test
suite (volunteer-context, watch-command, watch-sigint.serial,
cli-format-volunteer), migrate v117 coverage, and the two new E2E
files (pgbouncer-teardown env gating, volunteer-context-postgres RLS
pin); check:all row corrected (not a superset of verify).
- AGENTS.md + RELEASING.md: ci:local descriptions updated to the
sharded + pooler topology.
- CHANGELOG (wording only, entry preserved): "retrieved" instead of
"opened" for the used-signal, pooler scoped to the local CI gate,
feedback log labeled best-effort.
- llms-config.ts: index the new push-context guide; bundles
regenerated (build:llms) and freshness test green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(test): correct the v116 reference — the table shipped as migration v117
* fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095)
Five specialist reviewers (testing/maintainability/security/performance/
data-migration) on the reconciled diff; every finding applied:
Performance: the alias arm now resolves all granted sources CONCURRENTLY
(a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources
cross-region, inside the reflex's 1.5s budget); watch's session dedupe is
O(1) Set membership instead of a monotonically growing priorContext string
(O(T²) over a long-lived session); getWindowTurns iterates from the tail
(per-turn cost no longer grows with session length); the resolver's
provenance maps fold into the existing candidate pass.
Security: volunteer_context clamps caller-supplied attribution at the trust
boundary — session_id capped at 256 chars (a read-scoped token could bank
~1MiB TEXT per request, retained 90 days), turn logged only when a safe
integer (a non-integer threw inside the batched INSERT and silently dropped
the whole batch). The privacy comments now state precisely what rationale
may contain (the matched entity's surface form — which by construction
resolved to an existing alias/title/slug — never free conversation text).
Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from
volunteer.ts and shared by watch/cli (the two surfaces can no longer
drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly
site for all three channels; watch's window default now honors the same
retrieval_reflex_window_turns config knob the reflex reads; the stale
pre-v116 comments swept to pre-v117.
Testing: the two flake-class CRITICALs fixed (pipe test asserts the
backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test
waits on watch's new machine-readable ready line instead of a fixed 15s
sleep — 2.5s and deterministic now); new coverage for the sink's timeout
branch + ghost-reference drop, watch per-turn fail-open, untrusted knob
clamps (min_confidence/max_pages/days), window-cap ordering (newest user
mention survives), serve-IPC suppression passthrough + channel=reflex
logging, windowTurnCount edge semantics, and structural pins for the sink
registration + cycle purge wiring. The exit-verdict pin's grep is now
operator/whitespace-tolerant.
Deferred with TODOs: resolver index shapes for the per-turn query;
batched first-prune after a long dream-cycle gap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp
Four red-team findings on the #2095 push-context surface:
- RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's
cap, so a recurring already-pushed entity burned cap slots every turn and
starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside
the pointer loop BEFORE the confidence gate and the cap.
- RT3 honest stats: reflex-channel event logging moved from inside the
resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires
only after the response write succeeds, and buildReflexAddition logs only
after the per-turn timeout admits the block. A block the client's 250ms
budget abandoned was never injected and no longer counts as volunteered.
(logChannel resolver opt removed; logDeliveredReflexPointers is the seam.)
- RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo
can't reintroduce the re-scan-everything-per-turn cost class.
- RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP,
push-context guide, TODO to route watch via serve IPC); host-resolver
suppression contract at ResolveEntitiesFn (TODO for a capability gate).
Tests: starvation guard (watch + volunteerContext unit), window clamp floor +
ceiling, delivery-side logging (helper writes channel=reflex through the
drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test
rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims
Three CI-only check failures, two root causes:
1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when
loadConfig() returned null (no config file AND no DATABASE_URL — a clean
CI shard with no brain). loadConfig drops its env→config mapping in that
case, so the documented escape hatch silently died and the window fell
back to 4 → windowed extraction widened when the test set window=1 →
prior-turn entity leaked. Fixed: read the env var directly in
windowTurnCount, mirroring reflexEnabled's direct process.env read. This
is a real product bug, not just a test artifact — any config-less host
using the env hatch was affected. Regression test pins it.
2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a
sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway
and never reset it. The legacy-embedding preload only restores the
OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign
config survives into the next file — and a file's beforeAll runs BEFORE
the preload's restoring beforeEach, so federation-health built a
vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new
test files reshuffled the deterministic file→shard assignment, exposing
this latent ordering bug. Hardened both victims to establish the gateway
state they assert (sync-cost-preview resets to the unconfigured fallback;
federation-health pins legacy 1536 before initSchema) so they're
order-independent. Verified against a simulated leaker run before them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(context): use withEnv() in the window env-hatch test (test-isolation guard)
The regression test added in
|
||
|
|
4ee530f3c5 |
v0.42.42.0 fix(cli): bounded teardown + explicit exit — kill the 10s force-exit tax on txn-mode poolers (#2084) (#2141)
* feat(core): finishCliTeardown + flushThenExit — bounded teardown, owned exit verdict (#2084) cli-force-exit.ts becomes the single owner of one-shot CLI exit + teardown: - finishCliTeardown: bounded sink drain -> bounded disconnect under a backstop whose deadline is COMPUTED from the bounds it guards (floor 10s; GBRAIN_TEARDOWN_DEADLINE_MS env override). Arms at teardown start, never before the op handler. - flushThenExit: stdio write-fence (unref'd guard, EPIPE-safe) + REF'D aliveness grace for non-TTY stdio — Bun only delivers queued pipe writes while the process is alive (no flush API reaches the native queue). - setCliExitVerdict/currentExitCode: the exit verdict lives in a gbrain-owned channel, never read back from process.exitCode (PGLite's Emscripten runtime scribbles its own status there mid-run). - background-work.ts exports backgroundWorkSinkCount() for the deadline formula. Unit tests + a spawned-Bun harness proving byte-complete piped output. * fix(cli): route all nine disconnect sites through finishCliTeardown; one exit seam (#2084) Deletes the pre-handler 10s force-exit timer (it measured handler + teardown combined: PgBouncer txn-mode deployments paid a flat 10s banner tax on every query, and any >10s op was killed mid-run with exit 0 and truncated output). Sweeps op-dispatch, CLI_ONLY fall-through, search dashboard, read-only timeout path, dream, doctor x3 (fixing a pre-existing pool leak when DB checks throw), and ze-switch. The ONE process exit lives in main().then/catch via flushThenExit(currentExitCode()), gated by shouldForceExitAfterMain(). Exit-code writers (op-dispatch catch, reindex, transcripts, brainstorm, autopilot, frontmatter) now set the verdict through setCliExitVerdict. * fix(pglite): contain Emscripten's process.exitCode writes at PGlite.create (#2084) PGLite's WASM runtime writes its own status into process.exitCode (99 at create; in-memory brains run initdb whose status lands on a later tick; the exit status at close) — on PGLite every error exit was silently clobbered. preservingProcessExitCode wraps create() to keep the global tidy; db.close() stays unwrapped (its 0-write is baseline behavior test runners depend on). The CLI verdict itself is immune: it lives in the owned channel. * test: e2e + structural pins for the #2084 teardown contract E2E: failed op exits 1; every swept command spawned (brain-copy isolation for mutators, no-network); slow-handler regression via the deadline env knob; piped --json parses complete; teardown banner absent on every happy path; daemon survival untouched. Structural: no bare awaited engine disconnects in cli.ts; DISCONNECT_HARD_DEADLINE_MS gone; >=9 helper call sites; verdict channel + create-wrap pins. * test: fix R1 env-isolation violations in retrieval-reflex tests Pre-existing on master: both files mutated GBRAIN_RETRIEVAL_REFLEX directly, failing scripts/check-test-isolation.sh (bun run verify). Converted to the canonical withEnv() pattern; the reflex describe's beforeEach also never restored the flag, leaking it across the shard. * docs: KEY_FILES entries for the teardown contract; close + file TODOS (#2084) KEY_FILES.md: current-state entry for cli-force-exit.ts (helper + central exit seam pair, verdict channel, cli.ts-scoped claim); background-work.ts and pglite-engine.ts entries updated. TODOS.md: the drain-before-owner-disconnect P3 (filed from #1972) is done by this wave; files the trigger-gated GBRAIN_COMMAND_DEADLINE_MS follow-up (eng-review D2/D14). * fix: pre-landing review fixes (#2084) Review army (testing/maintainability/security/performance, 0 critical): - drain defense-in-depth: a throwing drain warns and still disconnects (cannot escape a caller's finally or skip the engine teardown) - behavioral tests for preservingProcessExitCode (connect pins 0; create-throw restores the pre-call verdict) - D9 widening test (live-registry sink count feeds the deadline formula), env 0/negative boundary cases, verdict mirror-write assertion - stale comments: header diagram backstop line, structural-test 'both lifecycle calls' contradiction, KEY_FILES 10s-force-exit clauses, e2e D11 falsification story corrected - named the formula's pool-end literals * fix: adversarial-review hardening — daemon-safe command resolution, flush knob, ref'd backstop (#2084) Cross-model adversarial review (Claude subagent + Codex, both P1'd it): - shouldForceExitAfterMain now resolves the command through parseGlobalFlags — the old first-non-dash heuristic read `gbrain --timeout 30s serve` as command "30s" and the new exit seam would have killed the daemon ~250ms after boot with exit 0 (unit-pinned) - GBRAIN_FLUSH_GRACE_MS env override for the non-TTY aliveness grace (batch consumers piping large payloads to slow readers can raise it; agent loops can lower it) - backstop timer is now REF'D: a hung teardown on an otherwise-empty event loop previously exited naturally — skipping the flush and surfacing PGLite's scribbled process.exitCode - flushThenExit: real process.exit latched once per process - doctor-site comment corrected; in-command process.exit teardown-bypass class (pre-existing) filed as a P2 TODO * chore: bump version and changelog (v0.42.42.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: move #2084 exitCode-containment lifecycle tests to the serial quarantine (R3) * docs: update project documentation for v0.42.42.0 - docs/TESTING.md: replace the stale 4-file serial-quarantine enumeration with a current-state description (the quarantine is glob-discovered, now several dozen files incl. the #2084 exitCode-containment suite); add unit inventory entries for test/cli-finish-teardown.test.ts and test/flush-then-exit-harness.test.ts. - docs/architecture/KEY_FILES.md: rephrase the pglite-engine exitCode containment note to current-state wording (clears the check-key-files-current-state prose-history warning). llms bundles regenerated (byte-identical: both docs are link-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply cross-model doc-review findings for v0.42.42.0 Codex review of docs-vs-shipped-code found 9 gaps; all verified against the code before fixing: - CHANGELOG.md (0.42.42.0 entry, precision narrowing only — no entries touched): "every CLI exit path" -> "every cli.ts disconnect site"; "on every path" -> "on every routed exit path"; dream/doctor/ze-switch claim scoped to dispatcher teardown (command-internal process.exit sites are tracked in TODOS as the open P2). - docs/architecture/KEY_FILES.md: the teardown backstop is REF'D, not unref'd (matches the F3 adversarial-review decision in the code). - src/core/cli-force-exit.ts: header diagram comment had the same stale unref'd claim + `process.exitCode ?? 0`; now matches the implementation (ref'd timer, `currentExitCode()`). Comment-only change. - docs/TESTING.md: verify is the 30-check parallel battery via run-verify-parallel.sh (was described as 4 checks); CI is 10 weighted LPT shards + dedicated verify/serial/slow jobs (was "4-way FNV on shard 1"); test:serial runs one bun process per file (not --max-concurrency=1); dead "cap: 10" line rewritten as debt guidance; inventory entries added for test/cli-should-force-exit.test.ts and test/e2e/pglite-cli-exit.serial.test.ts. bun run verify green (30/30); #2084 test files green; llms bundles regenerated (byte-identical — reference docs are link-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route v0.42.41.0's raw exitCode writers through the verdict channel; reconcile merged structural pins (#2084) CI fallout from merging the v0.42.41.0 triage wave into the #2084 exit-seam design — both waves fixed the same timer-placement bug independently: - doctor.ts + extract.ts set failure exit codes via raw `process.exitCode =` writes (v0.42.41.0's process.exit -> exitCode conversion); the #2084 exit seam reads only the gbrain-owned verdict channel, so doctor FAILs exited 0 (Tier 1 RLS e2e + half-migrated-Minions tests). Converted to setCliExitVerdict, same as the wallclock-124 site in the merge commit. - cli-force-exit-teardown-arming.test.ts pinned v0.42.41.0's inline finally-armed timer, which the merge replaced with finishCliTeardown; rewritten to pin the merged invariant (no pre-try arming in cli.ts; the backstop arms inside the helper before the drain). - eval-capture drain timing bound 1s -> 2s: flaked at 1023ms under CI shard load after the new test files shifted LPT shard packing (13x budget slack still proves bounded-not-hung). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
+1
|
7c27fa129b |
v0.42.41.0 fix: triage wave — 6 data-loss/availability fixes + 9 community PRs (#2128)
* fix(oauth): default omitted authorize scope to client's full grant When a client omits `scope` on /authorize, the authorize() grant computed `(params.scopes || []).filter(...)` → the empty set. That empty grant was written to oauth_codes and propagated into the access AND refresh tokens, so every request failed `insufficient_scope` even though the client was registered with e.g. `read write`. Because refresh inherits the stored grant, it never self-healed — reconnecting just minted another empty-scoped token. Some MCP connectors (observed with Claude Desktop) omit `scope` on /authorize, so they hit this on every connection. Fix: when no scope is requested, default to the client's full registered scope (RFC 6749 §3.3 permits a server default). This mirrors exchangeClientCredentials, which already does `requestedScope ? ... : allowedScopes`. The result is still clamped to the allowed set, so an explicit over-broad request cannot escalate. Adds test/oauth-authorize-scope-default.test.ts covering: omitted/empty → inherits full grant; explicit subset honored; clamp preserved (over-broad and disallowed-only requests cannot escalate or trigger inheritance). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): skip Python venv/ in the code walker collectSyncableFiles (first-sync walker) and the incremental PRUNE_DIR_NAMES set skipped node_modules but not Python venv/. On a Python repo the walker descended into venv/ (thousands of files); the resulting slug collisions crashed putPage's INSERT ... ON CONFLICT ... RETURNING with "undefined is not an object (evaluating 'row.deleted_at')". Add `venv` alongside node_modules in both the import.ts inline skip and PRUNE_DIR_NAMES. venv is the Python equivalent of node_modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): carry asymmetric input_type across the AI SDK to the wire body (#1400) dimsProviderOptions() threads input_type ('query' | 'document') into providerOptions.openaiCompatible for asymmetric models (ZE zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter validates providerOptions against a fixed schema and silently drops the field before building the HTTP body. Every embedQuery() was therefore encoded document-side: the ZE shim's hard default fired ('document'), Voyage and local openai-compat servers got no input_type at all, and asymmetric retrieval silently collapsed toward surface-token overlap — while the providerOptions-level contract test stayed green. Fix: an AsyncLocalStorage (same pattern as __budgetStore) populated in embedSubBatch() only when providerOptions actually threads an input_type, read at body-rewrite time by the fetch shims: - zeroEntropyCompatFetch: recovers the threaded value; document default preserved for ingest paths. - voyageCompatFetch: opt-in like the dims.ts Voyage branch — inject only when threaded; the field stays off the wire otherwise. - NEW openAICompatAsymmetricFetch: fallthrough default for every other openai-compatible recipe (llama-server, litellm, ollama, ...) — the canonical local/proxy paths for asymmetric models. Strict pass-through when nothing was threaded, so symmetric deployments see zero wire change; recipes with their own compat fetch (azure) keep it via the compat.fetch ?? precedence. KNOBS_HASH_VERSION bumped 10→11: cached query_cache rows were keyed on document-side query vectors; pre-fix rows must not be served to post-fix lookups (same convention as the v=3 embedding-provider bump). One-time global cold-miss on upgrade; refills within cache.ttl_seconds. Tests: test/embed-input-type-wire.test.ts runs the REAL SDK transport with a mocked global fetch and asserts on the outbound body — the only layer where this regression is observable. Covers ZE hosted, llama-server, litellm, ollama (query + document sides) and pins the pass-through for non-asymmetric models and Voyage's opt-in shape. 4 of the original 7 assertions fail on master, proving the pin. One structural pin in test/ai/zeroentropy-compat-fetch.test.ts updated to the new line shape (same semantic); KEY_FILES.md gateway.ts entry updated to the new truth. Supersedes #1400 (closed unmerged) — same ALS mechanism, extended to Voyage + all openai-compatible recipes. Credit to @billy-armstrong for the original diagnosis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sync): honor .gitignore in code walk; prune vendor/dist/build collectSyncableFiles (the full-sync / dry-run enumerator) reimplemented its own directory skip list inline (node_modules || ops), bypassing the canonical pruneDir gate and ignoring .gitignore entirely. On a Laravel/PHP repo this descended into vendor/ (~50k Composer files), storage/, and public/build/, trying to import 52k dependency/build files and flooding the index with library internals (a 35-min sync that never finished, killed by the watchdog at 3%). - collectSyncableFiles now enumerates via `git ls-files --cached --others --exclude-standard` when dir is a git work tree, so the walk honors .gitignore (tracked + untracked-not-ignored). Falls back to the FS walk for non-git dirs. EroLab: 52164 -> 1028 files. - The FS fallback now prunes through the canonical pruneDir() instead of a drifted inline list, so the two skip lists can't diverge again. - PRUNE_DIR_NAMES gains vendor/dist/build (dependency + build-output trees). Addresses #1483 (.gbrainignore), #1159 (--respect-gitignore), and the maintainer's #1942 vendor/dist/build prune. Walker regression suites (sync-walker-symlink, brain-writer-walk-prune, sync, sync-walker-submodule) green: 90 pass. * fix(config): ignore DATABASE_URL auto-loaded from cwd .env (#427) Bun merges .env files from the process cwd into process.env before any user code runs. loadConfig() prefers env DATABASE_URL over ~/.gbrain/config.json, so any gbrain invocation from inside a web-app checkout silently retargets the brain at that app's database — reads go to the wrong DB and apply-migrations can write gbrain's schema into a production app database (#427). effectiveEnvDatabaseUrl() re-parses the .env files Bun auto-loads from cwd and treats a DATABASE_URL whose value matches one of them as file-origin: ignored, with a one-time stderr notice. GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs are honored unchanged, so the operator escape hatch and the e2e suite's env-provided URL keep working. Applied at loadConfig, getDbUrlSource (doctor parity), init --non-interactive, and migrate --to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): arm the disconnect hard-deadline at teardown entry, not before the op body The 10s force-exit timer in the shared-op dispatch was armed BEFORE the try block, so any op whose handler ran past 10s wall-clock was killed mid-flight with process.exit(0) and zero stdout. On a slow Postgres pooler (6-10s per fresh connection) a healthy `gbrain search` was force-exited every time — an empty 'success' indistinguishable from no results. The v0.42.20.0 exitCode honor can't help: a mid-op kill fires before any error path sets exitCode. Move the arming into the finally (teardown entry), matching the fall-through owner-disconnect site later in main(): the timer still bounds a hung drain/disconnect (the C13 contract) but can no longer kill a slow-but-progressing op. Verified on a transaction-pooler Supabase brain: search went from 0 bytes/exit 0 at 10s to real results at ~21s. * fix(import): stamp source_id on extracted call-graph edges importCodeFile built CodeEdgeInput rows without source_id, so every edge landed NULL. getCallersOf/getCalleesOf filter `AND source_id = <scoped>` whenever a worktree pin or --source is in play — NULL never matches, so scoped call-graph queries silently returned 0 rows on multi-source brains even though the edges existed (2,122 edges, 26 targeting the probed symbol, count 0 returned). One-line fix: carry the sourceId already in scope into the edge input. Existing NULL rows backfill with: UPDATE code_edges_symbol e SET source_id = p.source_id FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE c.id = e.from_chunk_id AND e.source_id IS NULL; (same for code_edges_chunk). Verified: code-callers returns 21 callers where it returned 0. * docs(migrations): NULL embeddings BEFORE the column-type alter The Postgres recipe ordered ALTER COLUMN TYPE vector(N) before the UPDATE that clears stale embeddings. pgvector refuses to cast existing vectors across dimensions ('expected 1024 dimensions, not 1536'), so the recipe as written aborts the transaction on any brain that has embeddings — which is every brain doing this migration. Swap the steps: NULLs cast fine. * fix: honor legacy token source grants in oauth * fix(cli): bound read-scope op handlers at 180s wallclock (pre-landing review) With the hard-deadline timer correctly scoped to teardown, a genuinely wedged read handler (hung pooler connection mid-query) would hang the CLI forever — the #1633 zombie class the old pre-try timer accidentally bounded at 10s. Reads now get a generous withTimeout (180s default, far above any healthy slow-pooler run; --timeout=Ns overrides; exit 124 with the teardown finally still draining + disconnecting). Writes/admin stay unbounded: a long import/embed must never be killed by a default. * fix(import): stamp unscoped edges 'default', matching the pages-table default Review catch: 'sourceId ?? null' fixed the scoped path but left the unscoped one (reindex --code without --source, importCodeFile callers without opts.sourceId) stranding edges at NULL while their pages land under the schema default (pages.source_id DEFAULT 'default') — so getCallersOf(sym, { sourceId: 'default' }) missed them. Same bug, other door. Fallback is now 'default'. * fix(core): runtime dim-migration recipe NULLs embeddings before the alter Review catch: the doc fix corrected docs/embedding-migrations.md, but embeddingMismatchMessage still PRINTED the broken order — ALTER before UPDATE ... SET embedding = NULL — and linked to the now-contradicting doc. pgvector refuses to cast existing vectors across dimensions, so the printed recipe aborted on any brain that has embeddings. Swap the steps and say why inline. * feat(migrate): v116 — backfill NULL edge source_id + index from_symbol_qualified 1. Backfill: edges written before the stamping fix sit at source_id=NULL and stay invisible to scoped call-graph queries until repaired. Derive each edge's source from its own from_chunk's page (pages.source_id is NOT NULL DEFAULT 'default'). Same SQL verified live on a 2,122-edge production brain. 2. Indexes: getCalleesOf filters both edge tables on from_symbol_qualified, which had no index — every callee lookup was a seq scan, amplified per-BFS-node by the recursive code walk. With NULL edges repaired, scoped walks actually expand, so the latent cost becomes real. Mirrored into src/schema.sql; schema-embedded.ts regenerated. * docs(migrations): align the rationale list with the corrected recipe order The 'Why we don't do this automatically' list still said alter-then-wipe; reorder to wipe-then-alter and replace the fragile 'step 3' numeric cross-reference with a name-based one. * test: regression coverage for edge source_id stamping, timer placement, recipe order - import-code-edges-source-id: scoped import stamps edges + scoped getCallersOf/getCalleesOf match (verified failing pre-fix), plus the unscoped-import case asserting 'default' stamping. - cli-force-exit-teardown-arming: structural pin — the hard-deadline timer arms inside the finally (teardown entry), never before the op body; daemon guard, unref, clearTimeout intact. - embedding-dim-check: recipe order pinned — UPDATE precedes ALTER so the printed SQL can't drift from docs/embedding-migrations.md again. * fix(cli): hard-exit after teardown on wallclock timeout; bound makeContext too Adversarial review, two findings on the new timeout path: 1. On timeout the finally drained, disconnected, then CLEARED the hard-deadline timer — removing the only backstop while the abandoned handler (withTimeout races, it does not cancel) can hold ref'd sockets/SDK timers that keep Bun's loop alive: 'timed out' printed, process immortal — the zombie class this branch exists to kill, resurrected through its own fix. The finally now exits explicitly after teardown completes on the timeout path. 2. makeContext does DB I/O (resolveSourceId) for EVERY op and sat outside any bound — a pooler wedge at context build hung reads, writes, and admin alike. It now shares the same wallclock bound. * fix(import): normalize edge source once — closes the '' door and the unscoped chunk fan-out Adversarial review: txOpts used truthiness while the edge stamp used nullish — sourceId:'' put pages under 'default' but stamped edges '', FK-violating against sources(id) and silently dropping the file's whole call graph in the best-effort catch. The unscoped getChunks could also fan out to same-slug chunks from another source. One normalized edgeSourceId (sourceId || 'default') now drives both the chunk lookup and the stamp. * fix(engine): default edge source_id to 'default' at the insert layer (both engines) Adversarial review: addCodeEdges still wrote e.source_id ?? null, so any future caller that forgets the field reintroduces invisible NULL edges the day after the v116 backfill runs. A NULL source_id is invisible to every scoped call-graph query; default to the schema-default source the way the pages table does. Applied to both engines (parity). * fix(core): facts alter recipe NULLs embeddings before cross-dimension alters Adversarial review: buildFactsAlterRecipe shipped the same defect class this branch fixes for content_chunks 350 lines up — a cross-dimension ALTER ... USING cast that pgvector refuses while rows hold old-width vectors. Dimension changes now wipe first (the facts pipeline re-embeds on next write); same-dim type swaps (halfvec <-> vector) keep the lossless cast and PRESERVE data. Both behaviors pinned by tests. * v0.42.39.0 chore: version bump + CHANGELOG + TODOS Marks the v0.42.20.0 'decouple the op-dispatch force-exit timer' follow-up complete — this branch ships exactly that decoupling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(postgres-engine): atomic JSONB merge in updateSourceConfig — eliminate lost-update race ## Problem `updateSourceConfig` used a read-then-write pattern: read the current `config` row, normalize it in JavaScript, then write the merged result back with `SET config = <normalized> || <patch>`. Under concurrent callers (two background autopilot/cycle paths patching different keys simultaneously), both callers can read the same stale row. The later `SET config = ...` then clobbers the earlier patch, silently dropping whatever keys the first caller wrote. Reproduced at 21/25 lost-update events under real Postgres with parallel callers. ## Fix Fold the normalization and merge into a single atomic `UPDATE … SET config = CASE … END || patch` statement. Because the `SET` expression evaluates against the row-locked latest version of `config`, there is no snapshot window between the read and the write. Concurrent callers now converge correctly (50/50 clean in reproduction test). The `CASE` also normalizes historical bad JSONB shapes inline: - `object` — used as-is - `string` — double-encoded config; inner text parsed with the SQL `IS JSON` guard (Postgres 16+) so unparseable strings fall back to `{}` instead of raising `invalid input syntax for type json` - `array` — array of patch objects aggregated into a flat object via `jsonb_object_agg` - anything else — falls back to `{}` `pglite-engine.updateSourceConfig` already used an atomic `||` merge; this change brings postgres-engine to parity. ## Test Added two assertions to `test/list-all-sources.test.ts`: 1. JSONB string holding non-JSON text normalizes to `{}` (no cast throw) 2. JSONB string holding double-encoded valid JSON is parsed then merged * fix(doctor): five correctness fixes — stale locks, content sanity, graph coverage, exit code, gateway guard ## 1. Stale lock break hints cover gbrain-cycle: keys The doctor stale-lock report only recognized `gbrain-sync:` lock prefixes; everything else fell back to `gbrain sync --break-lock`, which is wrong for dream/autopilot cycle locks. A `gbrain-cycle:<source>` or `gbrain-cycle` lock now suggests `gbrain dream --break-lock [--source <name>]`, and unknown lock shapes fall back to `gbrain doctor` instead of a misleading sync command. ## 2. content_sanity_audit_recent counts reject and quarantine as hard failures v0.42 renamed the hard disposition path: rejected pages emit a `reject` event and quarantined junk pages emit `quarantine`; `hard_block` is now only the pre-v0.42 legacy alias. The status check only counted `hard_block`, so fresh `reject` / `quarantine` events from the new path cleared as `ok` whenever fewer than 10 events existed. The check now sums all three for the hard count, and `soft_block + flag` for the soft count. ## 3. graph_coverage excludes test fixture entity pages from the denominator Brains seeded with code sources (e.g. a sync of the gbrain repo itself) could accumulate test fixture pages typed as `entity` / `person`. Including these in the entity-count denominator diluted coverage and produced spurious warnings ("Entity link coverage 0%, timeline 0%") on knowledge-only brains with no real entity pages. The check now queries a per-entity stats CTE that excludes `tools/gbrain/test/*` slugs and the `templates/new-person` stub, with an additional guard for the all-fixture case (`eligibleEntityCount = 0`). ## 4. process.exitCode instead of process.exit at doctor main exit point `process.exit(hasFail ? 1 : 0)` was a hard kill that prevented cleanup handlers (Bun unload events, open DB connections) from running. Using `process.exitCode = hasFail ? 1 : 0` defers the actual termination until the end of the event loop, allowing cleanup to complete. ## 5. checkSubagentCapability exported for test seams + gateway loop guard The function was private, making it untestable in isolation. It is now exported. Additionally, users running gbrain with a non-Anthropic chat model via `agent.use_gateway_loop=true` no longer receive a spurious warning that `ANTHROPIC_API_KEY` is missing — subagents route via the gateway loop in that configuration and do not need the key directly. ## Tests Doctor test suite: 77 pass, 0 fail (no regressions). * fix(engine): deleteFactsForPage excludeSourcePrefixes (#1928) + reconnect() parity (#2034) Engine-layer API for two cycle/availability fixes that share these files: - deleteFactsForPage gains optional excludeSourcePrefixes so the fence reconcile can protect non-fence facts (e.g. cli: conversation facts). - reconnect(ctx?) is now a first-class BrainEngine method on both engines (PostgresEngine already had it; PGLite gains config capture + reconnect) so callers stop using disconnect()+bare connect(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cycle): stop extract_facts from wiping conversation facts (#1928) The fence reconcile delete-then-reinsert wiped cli:-origin facts (no fence to recreate them); a failed-sync full walk turned it brain-wide (1829 rows, 0 reinserted, status ok). Now: exclude cli: rows from the wipe, do NOT inherit the failed-sync->full-walk fallback for this destructive phase, and warn on net-negative reconcile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(autopilot,supervisor): reconnect() instead of disconnect()+bare connect() (#2034) The autopilot health-probe recovery called connect() with no args after disconnect(), losing the startup config (database_url undefined -> FATAL restart-loop on every DB blip) and opening a null-pool window. Both call sites now use engine.reconnect(), which restores the captured config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): mirror to the assigned source's local_path, never the global repo (#2018) put_page write-through resolved the disk target from the global sync.repo_path, so a default-source page (local_path NULL) got written into an unrelated federated source's working tree. Now it uses the assigned source's own local_path; NULL local_path skips (no leak); the global path is used only as a sole-source fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite-lock): heartbeat + steal-grace so live holders are never stolen (#2058) A live holder's lock was force-removed after 5min age alone, letting a second process share the single-writer data dir -> WAL corruption. The lock now heartbeats while held; a holder is reaped only when its PID is dead OR its heartbeat went stale past the steal grace. Pairs PID liveness with heartbeat age to also defeat PID reuse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(migrate,doctor): self-heal idx_timeline_dedup drift (#2038) A migration renumbered during a merge (v102) could be recorded-as-applied without its DDL running, leaving the 3-column index so every timeline write failed the 4-column ON CONFLICT. runMigrations now always runs a shape-keyed drift repair (dedupe-then-rebuild) even when no migration is pending, and doctor surfaces the drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(timeline): un-silence the swallowed batch catch; pin Date-batch round-trip (#2057) The meetings extractor's bare catch {} hid a brain-wide timeline-write failure (0 entries, no error). It now counts + surfaces batch errors. Adds a Date-bearing batch regression test proving the #1861 jsonb_to_recordset refactor already fixed the original ::text[] cast failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v0.42.41.0) Triage fix wave: 6 authored critical fixes (#1928 facts wipe, #2018 write-through leak, #2034 reconnect loop, #2058 WAL lock, #2038 timeline migration drift, #2057 timeline silent-empty) + community PRs #2064 #2052 #2020 #2033 #2074 #2075 #2009 #2072 #2073. TODOS: deferred #1994 #1963 #2050. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: address adversarial review findings (#1928, #2058, #2038, #2057) Codex as-built review of the authored fixes surfaced 4 real issues: - #2058: add a pid+acquired_at ownership token. A stale holder reaped + replaced past the grace must NOT let its resumed heartbeat refresh, nor releaseLock remove, the NEW owner's lock (re-opened the concurrent-writer hole). Heartbeat and release now verify the on-disk lock is still ours. + regression test. - #1928: the destructive-full-walk guard keyed off phases.includes('sync'), which wrongly suppressed a legitimate full reconcile when sync was SKIPPED (no engine / no brainDir). Key off a syncAttempted flag set only when sync actually ran. - #2038: dedupe keeps MIN(id) not MIN(ctid) — deterministic and consistent with the existing v-migration lower-id rule. - #2057: the extract CLI caller now surfaces batch_errors (stderr + exit 1) instead of printing a clean success over failed inserts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(key-files): sync reference to v0.42.41.0 triage-wave behavior Update KEY_FILES.md to current-state truth for the shipped fixes (no release-history clauses, per the reference-doc discipline): - write-through.ts (#2018): resolves the disk target from the assigned source's own local_path; sole-source falls back to sync.repo_path, multi-source skips with source_has_no_local_path rather than leak. - engine.ts (#2034): reconnect() is now a REQUIRED lifecycle method on both engines; config-restoring, never disconnect()+bare connect(). - migrate.ts (#2073): document v116 edge source_id backfill + callee index, and the always-run (version-counter-blind) timeline dedup self-heal. - new entry for timeline-dedup-repair.ts (#2038) + the timeline_dedup_index doctor check. - new entry for pglite-lock.ts (#2058): heartbeat + steal-grace (GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS) so a live holder is never stolen. - extract-facts.ts (#1928): cli:-fact protection, no failed-sync full-walk inheritance, net_fact_deletion warn floor. bun run build:llms re-run (KEY_FILES is link-only so bundles unchanged); freshness + current-state guards green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): preserve nested multi-source layout; narrow #2018 leak guard The first #2018 fix skipped any no-local_path source on a multi-source brain, which broke the legitimate nested layout (a source without its own tree nests under the host repo at .sources/<id>/ — pinned by put-page-write-through.test). Narrow the guard: a no-local_path source nests under sync.repo_path as before; only SKIP when sync.repo_path is literally another source's own local_path (the actual leak — writing there pollutes that sibling's repo). Caught by the sharded suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: satisfy test-isolation guard for the new lock/reconnect tests CI `verify` flagged 3 intra-process isolation violations in the tests added this wave (the parallel runner shares one process per shard): - pglite-lock.test.ts: the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS mutation now goes through withEnv() instead of a raw process.env write (R1). - pglite-reconnect: renamed to *.serial.test.ts — it creates per-test engines to exercise the connect/reconnect lifecycle, which doesn't fit the shared beforeAll-engine model (R3/R4). verify is now 30/30; both files green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite): reconnect() is a no-op for in-memory engines (#2034) CI serial-tests + test(5) caught two in-branch regressions from the #2034 PGLite reconnect(): - worker/queue claim-error recovery + their renewLock e2e test assume PGLite reconnect is absent/no-op (queue.ts documents it). Making it a real disconnect+reopen wiped an in-memory engine's state mid-job. reconnect() now no-ops for in-memory (no database_path) — file-backed still re-opens the dir (state persists on disk). Restores the documented worker assumption. - connection-resilience 'Supervisor still has the 3-strikes-then-reconnect path' pinned the removed unsafe-cast text; updated to assert the direct this.engine.reconnect() call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: quarantine embed-input-type-wire to serial lane (CI test(5) leak) #2033's embed-input-type-wire.test.ts configures a 1280-dim embedding gateway; the active dimension survived into engine-find-trajectory when CI's 10-way hash-disjoint sharding co-located them (this branch's added files reshuffled the assignment), failing 7 trajectory tests with 'expected 1280 dimensions, not 1536'. resetGateway() in afterEach clears the gateway but the dimension still leaked. It mutates global gateway/embedding state, so it belongs in the serial lane (own bun process, true isolation) by the repo's own definition. Root-caused by reproducing the exact failing pair locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Austin Arnett <austin@sdsconsultinggroup.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Dave MacDonald <djmacdonald@ucdavis.edu> Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com> Co-authored-by: Alex P. <12667893+aphaiboon@users.noreply.github.com> Co-authored-by: Garry Tan <bo.m.liu@gmail.com> Co-authored-by: jbarol <barol.j@gmail.com> Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> Co-authored-by: PAI <pai@scaffolde.ai> |
||
|
|
ecd6ae8772 |
v0.42.40.0 fix(extract,ingest): well-form lone UTF-16 surrogates before jsonb (#2011) (#2031)
* fix(extract,ingest): well-form lone UTF-16 surrogates before jsonb (#2011) excerpt() in link-extraction.ts sliced the link-context window by raw UTF-16 index, so a boundary landing inside a non-BMP char (emoji, math, CJK) left an unpaired surrogate half in `context`. Serialized to JSONB for the jsonb_to_recordset batch insert, Postgres rejects it at the ::jsonb cast and aborts the whole batch — wedging `extract --stale` because the staleness bookmark only advances on a clean finish. - text-safe.ts: new ensureWellFormed() (Bun isWellFormed/toWellFormed) — one shared surrogate-cleaning primitive. - link-extraction.ts: excerpt() well-forms the slice (root-cause fix). - batch-rows.ts: new sanitizeForJsonb() = ensureWellFormed(stripNul(s)) applied to every free-text body field (link context; timeline summary/detail/source; take claim/source). Identity/security fields stay un-sanitized and fail closed. - postgres-engine.ts + pglite-engine.ts: scalar addLink + addTimelineEntry use sanitizeForJsonb too, matching the batch path on both engines. - brainstorm/orchestrator.ts: consolidate hand-rolled sanitizeUnicode onto ensureWellFormed (also fixes consecutive-lone-surrogate mishandling). Tests: ensureWellFormed unit cases (incl. consecutive lone surrogates), an excerpt window-split regression, PGLite + Postgres-e2e surrogate cases across all free-text fields and scalar paths, and fail-closed identity-field tests proving sanitization was NOT extended to slugs/holders. * v0.42.39.0 chore: bump version and changelog (#2011) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.42.39.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.40.0 chore: re-slot release version (was 0.42.39.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: fix env-mutation isolation violations in retrieval-reflex tests check:test-isolation (rule R1) flags direct process.env mutation in non-serial test files — bun's parallel runner loads multiple files into one process, so a leaked GBRAIN_RETRIEVAL_REFLEX flips reflex behavior in unrelated tests. Both files landed via the #2019 merge; convert the beforeEach/afterEach env juggling to the canonical withEnv() wrapper, which restores the prior value via try/finally even on throw. Fixes the failing `verify` CI check on #2031 (and the `test-status` aggregate that inherits it). All 30 verify checks green locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
8f45624e55 |
v0.42.39.0 feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981) (#2019)
* fix(integrations): parameterize resolver-row fence by recipe id The install fence was hardcoded gbrain:agent-voice:resolver-rows, so any second copy-into-host-repo recipe wrote a mislabeled block (and refresh/ uninstall keyed on recipe id would miss it). Derive it from manifest.recipe. * feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981) Deterministic per-turn pointer layer in the context engine: a zero-LLM, precision-biased scan resolves salient entities (names, @handles) to existing brain pages and injects compact pointers (name → slug → safe synopsis). Detect + point, never auto-dump. Fail-open, capped, suppression on prior context only. Engine-aware resolver ladder (no second DB connection): host ctx.brainQuery → PGLite serve resolve IPC (unix socket) → Postgres cached direct → disabled. Synopsis runs through get_page's privacy strip. Plus the retrieval-reflex recipe + policy skill, the retrieval_reflex_health doctor check, config gate, and the init next-step hint. * chore: bump version and changelog (v0.42.38.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): KEY_FILES entries for Retrieval Reflex surface (#1981) Document the new src/core/context/ modules, the context-engine resolver ladder, the serve resolve IPC, the retrieval_reflex_health doctor check, and the recipe-id-keyed install fence. Current-state only. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
03ffc6ebdb |
v0.42.37.0 fix(jobs): reap stale locks, bound disconnect, complete cooperative-abort (#1972) (#2015)
* fix(jobs): reap stale dead-holder cycle/sync locks (#1972) A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row until something contended for it — reclaim was on-contention only. Add a host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle* namespaces only (never elections/supervisor/reindex), with a snapshot-matched delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse. Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now back inspectLock + listStaleLocks (killed the triplication). * fix(db): bound pool disconnect so teardown can't eat CLI output (#1972) pool.end() against PgBouncer transaction-mode never drained, so disconnect blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write, truncating stdout (e.g. #1959's relational query returned empty). Add a gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against a hard timer, so teardown is bounded regardless of what postgres.js does and is testable. connection-manager ends its direct + read pools concurrently so the per-pool bounds don't stack. PGLite disconnect is unaffected. * fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972) v0.42.29 made only the embed phase honor the abort signal; a 24h pull still showed force-evicts from a long non-embed phase ignoring it. Thread the signal into every cycle-reachable long loop: extract (extractForSlugs + the full-walk extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop + embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket loop. Add a terminal abort check so an aborted cycle never stamps last_full_cycle_at as a completed run (Codex #9). lint now yields + checks abort every 200 pages (it's synchronous; the yield is what lets the signal land). New phase-duration force-evict attribution log names any phase that crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start. * chore: bump version and changelog (v0.42.38.0) #1972 — stale-lock reaper, bounded pool disconnect, and complete cooperative-abort coverage across cycle phases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY), db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/ extract_facts/consolidate/lint + terminal guard) entries to current truth. * test(isolation): fix shard-order flakes exposed by #1972's new test files Adding 3 new test files reshuffled the hash-based shards, exposing two pre-existing test-isolation bugs: - cycle-consolidate.test.ts assumed the global legacy-embedding preload's 1536-d gateway config still held at initSchema, but a co-sharded test that calls resetGateway() in teardown nulls it, so initSchema fell back to the 1280-d default and built a halfvec(1280) facts column its 1536-d fixtures can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern legacy-embedding-preload.ts documents for 1536-d fixture tests). - db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation rule R1. Convert to withEnv(). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
1eb430a2df |
v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts (#1999)
* fix(security): scope cross-source reads to the caller grant; close get_page exact-path leak One shared resolveRequestedScope() routes every source-scoped read op (query, code_callers/callees, search_by_image, code_blast/flow, get_page) through a single fail-closed trust+grant ladder: a remote caller's __all__ collapses to its granted sources (never the whole brain) and an explicit out-of-grant source_id is rejected. get_page's exact-match path now honors a federated grant via getPage(sourceIds[]) in both engines. Legacy bearer tokens carry their stored permissions.source_id grant (bounded, never widened). Also retries getConfig on transient connection loss. Closes #1924, #1371, #1393, #1336, #1603. * fix(ingest): non-string frontmatter no longer aborts lint/sync; embed/hook/catalog papercuts Parser coerces a non-string title to a string and falls back to inference for slug/type (never fabricating a "123" slug), with a lint NON_STRING_FIELD finding surfacing the malformed frontmatter; a defensive guard in content-sanity stops a non-string title from crashing the whole lint/sync run brain-wide. Plus: embed --catch-up no longer arms the overflowed 32-bit budget timer (and surfaces unembeddable chunks); the frontmatter pre-commit hook ships a correct .md/.mdx regex; and the skill catalog parses YAML block-scalar descriptions. Closes #1883, #1658, #1556, #1948, #1946, #1840, #1711. * v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add NON_STRING_FIELD frontmatter validation class to docs for v0.42.37.0 The v0.42.37.0 non-string-frontmatter fix added an eighth validation class (NON_STRING_FIELD / lint code frontmatter-non-string-field). Update the two current-state docs that enumerate the validation classes: - skills/frontmatter-guard/SKILL.md (seven->eight + table row) - docs/integrations/pre-commit.md (seven->eight + table row) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
959af1068d |
v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794) (#1980)
* fix(retry): match EMAXCONNSESSION + SQLSTATE 53300 as retryable conn errors (#1794) * feat(schema): add op_checkpoint_paths append-only delta table (migration v115) (#1794) * refactor(op-checkpoint): append-only deltas via executeRawDirect + withRetry (#1794) * fix(sync): resumable-checkpoint durability + lock-thrash fix (#1794) Durable append-only checkpoint writes (executeRawDirect + retry), fail-loud consecutive-failure abort, first-file/10s flush cadence, race-safe pending-delta under parallel workers, guaranteed final flush on every exit path incl. SIGTERM (no-retry one-shot via registerCleanup), bankedFiles/reason observability, event-loop yield to keep the lock heartbeat alive, and routing the bare (no-source) sync through withRefreshingLock. * fix(db-lock): heartbeat-aware takeover + direct-pool refresh (#1794) * fix(cycle): treat SyncLockBusyError as skip, not a phase failure (#1794) * docs(sync): document the 5 checkpoint/lock env knobs (#1794) * v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): update sync.ts + op-checkpoint.ts entries to resumable-checkpoint current state (#1794) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
612753f318 |
v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970) (#1975)
* v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970) When a source's history is rewritten (force-push, master→main consolidation, squash), the recorded last_commit can fall outside HEAD's history. The old guard sent both "object missing" and "not an ancestor" to performFullSync — a full repo re-walk that never advances the bookmark under a cron timeout on a large cross-region brain, so the source goes silently stale. Fix: only a truly-absent object forces a full reconcile. A present-but-non- ancestor bookmark is diffed tree-to-tree directly (git diff A..B needs no ancestry), importing only the real delta. Adds: oversized-diff fallback to full reconcile (F-B); performFullSync now purges deleted files, gated to file-backed pages by source_path so manual/put_page and metafile pages are spared (F-A); rename-to-unsyncable deletes the stale old page (F-C). 7 new PGLite e2e tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): record #1970 sync bookmark recovery + full-sync delete reconcile in KEY_FILES Update the sync.ts entry to current truth: entry-time bookmark-reachability guard (gc'd anchor → full reconcile; non-ancestor-but-present → direct tree-to-tree diff), oversized-diff fallback, performFullSync now authoritative for deletes (file-backed pages by source_path), and rename-to-unsyncable delete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
099d9a8f55 |
v0.42.34.0 feat(search): typed-edge relational retrieval — relationship questions get relationship answers (#1959)
* feat(search): deterministic relational-query parser
Pure, ReDoS-bounded parser that detects relationship queries ("who invested
in X", "who at X works on Y", "who introduced me to X", "what connects A and
B") and maps them to typed edges. Schema-pack-extensible vocab with subset
validation against the link types ingest produces, so query-side and
ingest-side relation vocabularies can't drift. No-match / pronoun-seed /
adjacency guards keep it precision-first (a candidate only; the arm fires
only when a real seed also resolves).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(engine): relationalFanout typed-edge fan-out (both engines)
Generalizes traversePaths to a SEED ARRAY, aggregating to ranked NODES
(shortest hop, edge-richness count, via-link-types, shortest connecting
path, canonical chunk id) instead of edges. Within-source traversal (never
crosses a source boundary even across a federated scope), link_source=
'mentions' excluded by default, deleted_at filtered at seed + every neighbor
+ every node, bounded depth (<=3) + candidate cap. Adds RelationalFanoutRow
/ RelationalFanoutOpts + the relational SearchResult/SearchOpts fields to
types.
Lands in lockstep in postgres + pglite engines, pinned by a DATABASE_URL-
gated parity block in engine-parity.test.ts; a PGLite unit test exercises
the SQL (typed-edge filter, mentions exclusion, deleted_at, canonical chunk,
multi-seed connects, determinism) in default CI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(search): relational recall arm + federation key hardening
Wires edge-derived candidates into bare hybridSearch as a FOURTH RRF arm
(relational-recall.ts): parse the original query -> scope-aware,
confidence-gated seed resolution (drops fallback_slugify; never traverses
from a guess) -> relationalFanout -> batch-hydrate, reinforcing each page's
REAL canonical chunk (page-level key for chunkless entity pages) ->
--explain attribution + fail-open audit row. Text-only (no-op in image
mode); pure no-op for non-relational queries; rides every downstream stage
(cosine, post-fusion boosts, dedup, reranker, autocut, token budget).
Mode wiring: relationalRetrieval + relational_retrieval_depth knobs
(conservative off; balanced/tokenmax on; depth 2), per-call thread-through
in both bare + cached paths, KNOBS_HASH_VERSION 9->10 (rel=/reld=), config
keys, modes-dashboard descriptions, and a `relational` param on the query op.
Federation hardening (structural, engine-wide): the RRF/dedup key now
carries source_id via a shared rrfKey() (fixes a latent cross-source
collapse where same-slug pages in different sources merged), and the query
cache scopes by a canonical source-set key (cacheScopeKey) so a federated
read can't be served a single-source row.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(eval): relational benchmark + recall@k harness metrics
NamedThingBench harness gains recall@k / recall@10 (the relational headline
metric) on QuestionResult + FamilyReport, plus typed seed/linkTypes/kind on
NamedThingQuestion so the graph-relationship family is machine-checkable.
Adds the relational benchmark corpus (test/fixtures/retrieval-quality/
relational/): a small entity graph whose answers are LEXICALLY UNRECOVERABLE
— every page body is generic and never names the entity it relates to, so
only the typed edge connects query to answer. corpus.ts is the canonical
source for both the seed loader and the 38-question gold set; relational.jsonl
is generated from it (a drift test pins them equal).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(eval): relational A/B proof + arm fires on all retrieval paths
Fixes the integration bug the eval caught: the relational arm was only
injected on the main RRF path, so it silently did nothing whenever vector
was unavailable — no embedding provider configured (the default in many
deployments) OR embed failure. The arm is now built once and fused via RRF
on ALL THREE hybridSearch return paths (no-embedding-provider, embed-failed
keyword fallback, main path). Without this it would have been dead in
exactly the setups that most need it.
Adds `gbrain eval retrieval-quality --ab-relational`: runs the gold set
twice (arm off vs on) in a fixed mode and reports the graph-relationship
recall@10 lift + Hit@3 + latency add. The CI A/B test pins the headline
result — recall@10 jumps from <25% (lexically unrecoverable) to >75% with
the arm on — and a non-relational query returns identical results arm-on vs
off (the no-op / no-regression gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.34.0)
Relational retrieval feature: typed-edge recall arm + federation key
hardening. Also updates the KNOBS_HASH_VERSION 9→10 assertions across the
remaining search test files (the bump invalidates relational-off cache rows).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document typed-edge relational retrieval (v0.42.34.0)
CLAUDE.md Search Mode: add relationalRetrieval to the knob table, the
knobs_hash v=9→10 note, and a relational-retrieval summary. RETRIEVAL.md:
add the relational recall arm to the pipeline diagram. Regenerate llms
bundles (build:llms).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(test): size relational fixtures to the actual embedding-column dim
CI shard runs with the ZeroEntropy gateway default (1280-d), but the
relational test fixtures hardcoded 1536-d embeddings, so chunk inserts were
rejected with "expected 1280 dimensions, not 1536" (CheckExpectedDim) — the
`test (6)` shard + `test-status` failures. The column width tracks the
configured gateway default and can shift with shard order, so fixtures now
probe `content_chunks.embedding`'s actual `atttypmod` after initSchema and
size embeddings to it (the pglite-engine.test.ts pattern), via a shared
`probeEmbeddingDim` helper. Verified passing at a forced 1280-d column.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
b31de6613e |
v0.42.33.0 fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (#1881) (#1960)
* fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (#1881) recloneIfMissing deleted local_path whenever a source had a remote_url and a non-healthy on-disk state, with no check that gbrain actually created the clone. A source whose local_path was a user's live working tree (remote_url set, no gbrain-created clone) could have its directory removed and re-cloned over. - isOwnedClone(): ownership, not path-containment. True only for a config .managed_clone marker (written by addSource --url) or exact normalized-path equality with defaultCloneDir(id) (back-compat for pre-marker default clones). - recloneIfMissing: ownership guard aborts before ANY filesystem op; EXDEV-safe sibling-temp clone + atomic swap (old aside -> new in -> drop old) with best-effort restore + a message naming where the original is preserved; symlink-leaf reject before the destructive rename. - sync.ts validate_repo_state guards reclone on isOwnedClone (no per-sync warn). - sources restore degrades to a warning for an unowned source instead of the misleading "missing clone, try sync" hint. Tests: #1881 regression (tree survives), isOwnedClone matrix, symlink reject, EXDEV swap residue-free, --clone-dir owned-via-marker, restore CV3, unownedHint healthy/degraded, sync-level refusal. * chore: bump version and changelog (v0.42.33.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document sources-ops reclone-ownership invariant for v0.42.33.0 (#1881) Add the missing src/core/sources-ops.ts entry to KEY_FILES.md capturing the must-never-violate reclone-ownership guarantee: gbrain only deletes/re-clones a clone it created (isOwnedClone), never a user working tree. Covers managed_clone marker, defaultCloneDir back-compat, EXDEV-safe swap, TOCTOU + symlink-leaf guards, unmanaged_path SourceOpError, and the read-only sources restore path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
5a06af5a57 |
v0.42.32.0 fix(sync): coerce non-string frontmatter titles + bounded auto-skip failure ledger (#1939) (#1956)
* fix(import): coerce non-string frontmatter title/slug/type (#1939) YAML `title: 2024-06-01` parses to a Date and `title: 1458` to a number; the old `(frontmatter.X as string)` cast was a compile-time lie, so downstream `.toLowerCase()` threw and (via the importer failure gate) could wedge sync indefinitely. parseMarkdown now coerces via coerceFrontmatterString (Date -> UTC ISO date, deterministic), and the pure assessContentSanity self-protects against a non-string title. * feat(sync): bounded auto-skip failure ledger; poison file can't wedge indexing (#1939) New src/core/sync-failure-ledger.ts owns the failure store + a crash-safe, multi-source, concurrent bounded auto-skip valve. A file that fails N consecutive syncs (GBRAIN_SYNC_AUTOSKIP_AFTER, default 3) auto-skips so it can't freeze all indexing forever, while fresh failures still fail-closed and a `<head>` history-rewrite sentinel hard-blocks even with --skip-failed. - (source_id, path) keying — failures never merge across sources - success clears a path so attempts are truly consecutive - advance-before-ack ordering (a crash can't mark a file skipped while wedged) - shared applySyncFailureGate used by BOTH the incremental and full-sync gates - legacy-row normalization + duplicate collapse on load - cross-process lock + atomic temp-rename, age-based stale-lock break sync.ts re-exports the ledger for existing callers; import.ts records source-scoped and defers the bookmark to the gate under managedBookmark. * fix(doctor): sync_failures severity via one shared decision on both surfaces (#1939) Local buildChecks and remote doctorReportRemote now both route through decideSyncFailureSeverity, so a stuck bookmark escalates WARN -> FAIL consistently (oldest-open age > fail cadence, or large unresolved count), auto-skipped pages stay visible (WARN, not hidden), and the acknowledged/acknowledged_at field-split that caused drift is gone. The remote surface stays subprocess-free (file read + Date.parse only). * chore(test): add trailing newline to e5-lease-cap-ab baseline fixture * fix(sync): address adversarial review findings on the failure ledger (#1939) - #1: a parse-failed file that is later deleted/renamed-away no longer leaves a permanent open ledger row. Removed paths (filtered.deleted, renamed-from, and the "gone from disk" forward-delete skip branch) are treated as resolved so the ledger self-heals instead of aging doctor to a stuck FAIL. - #3: decideSyncFailureSeverity escalates to FAIL on OPEN (blocking) failures only — auto_skipped rows already advanced the bookmark, so they stay WARN-visible regardless of count, matching the state-machine contract. * chore: bump version and changelog (v0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document sync-failure ledger + auto-skip valve for v0.42.30.0 KEY_FILES.md: new src/core/sync-failure-ledger.ts entry (bounded auto-skip state machine, decideGateAction/decideSyncFailureSeverity/applySyncFailureGate, GBRAIN_SYNC_AUTOSKIP_AFTER); update sync.ts (failure store moved to ledger, re-exported), doctor.ts (sync_failures severity via shared rule on both surfaces), markdown.ts (coerceFrontmatterString), import.ts (managedBookmark). live-sync.md: poison-file auto-skip tricky-spot. Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.31.0 (queue collision on 0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.32.0 (queue collision) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f401d7407e |
v0.42.31.0 feat(links): open link_source provenance + link-add/link-rm/link-sources (#1941) (#1957)
* feat(links): relax link_source CHECK to kebab-case provenance + migration v114 Open link_source from a closed allowlist to a kebab-case format gate (^[a-z][a-z0-9]*(-[a-z0-9]+)*$, char_length<=64) so external derivers stamp their own provenance (e.g. citation-graph) without a per-deriver migration. Migration v114: Postgres NOT VALID + VALIDATE (lock-friendly, transaction:false); PGLite plain DROP+ADD. Updates the schema.sql + engine provenance contract comments. (#1941) * feat(links): expose link provenance on link ops + link-add/link-rm/link-sources add_link/remove_link now accept --link-source/--link-type; add_link guards the reconciliation-managed built-ins (markdown/frontmatter/mentions/ wikilink-resolved) and defaults omitted provenance to 'manual' (was the misleading engine default 'markdown'). New cliHints.aliases mechanism with a startup collision guard registers link-add/link-rm; printOpHelp shows the invoked alias name. New list_link_sources read op + listLinkSources engine method (both engines, {sourceId?,sourceIds?}, deterministic order) powers `gbrain link-sources`, added to the minion read allowlist. (#1941) * test(links): kebab provenance, op guard, link-sources, aliases + parity Covers the v114 regex/length boundaries, upgrade-path constraint swap on existing data, the managed-built-in op guard + manual default, remove_link type/source filters, list_link_sources scoping (scalar + federated) and PG/PGLite parity, and alias resolution/collision/help. Fixes the prior 'inferred'-rejection assertion (now valid kebab) in the mentions test. (#1941) * chore: bump version and changelog (v0.42.31.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update KEY_FILES for v0.42.31.0 link provenance surface KEY_FILES.md current-state updates for #1941: link_source now an open kebab-case provenance (migration v114), the add_link/remove_link guard + defaults, list_link_sources + listLinkSources, and cliHints.aliases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minions): supervisor queue-singleton keying + pidfile cleanup (follow-up #1849) Two correctness bugs in the v0.42.29.0 supervisor-singleton work, caught by adversarial review: - supervisorLockId mixed a config-derived DB identity into the key, but the lock row already lives inside the target database. Two supervisors on the same physical DB via different-but-equivalent URLs (pooler vs direct port, host alias, trailing params) computed different ids and BOTH acquired the "singleton" lock. Key on the queue alone; the database half of the mutex is physical. Removes the now-dead currentDbIdentity() from worker-registry. - The pidfile-cleanup process.on('exit') listener was installed AFTER the DB-lock acquire, so the LOCK_HELD early-exit stranded the pidfile this process had just created. Install the listener first. Regression test pins the listener-before-lock ordering; updates the lockId test to the queue-only invariant; KEY_FILES updated to current state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
613da94093 |
v0.42.29.0 fix(minions): long-job abort-honoring + attempt accounting + supervisor singleton; topic-aware voice (#1737, #1849, #1851) (#1943)
* fix(minions): honest attempt accounting + cooperative abort-honoring + per-handler timeouts (#1737) - Wall-clock and stall dead-letter paths now increment attempts_made (terminal, no retry — wall-clock fires at 2x cumulative timeout; retrying non-idempotent embed/subagent work would duplicate side effects). Surface stalled_counter in jobs get so 'started 3 / stalled 2 / attempts 0' reads true instead of looking like broken accounting. - Thread AbortSignal through embed-backfill/autopilot-cycle -> runPhaseEmbed -> runEmbedCore -> embedAll(Stale)/embedPage, checking it on BOTH --stale and --all paths and between embed batches. A timed-out embed phase now bails within a batch, so the cycle finally releases gbrain_cycle_locks instead of running the full 10-15 min after the job was killed (the daily cycle-wedge). New shared src/core/abort-check.ts (isAborted/throwIfAborted/anySignal). - Per-handler default wall-clock budget (handler-timeouts.ts) stamped at submit for long handlers without an explicit timeout_ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minions): queue-scoped DB supervisor singleton + canonical pidfile + doctor max-rss check (#1849) - Acquire a queue-scoped DB lock (tryAcquireDbLock, keyed on the raw DB identity + queue) on supervisor.start(): a second supervisor on the same (db, queue) fails fast with exit 2 regardless of $HOME/--pid-file. Refresh on a dedicated timer; on refresh failure past the threshold, fail SAFE (exit non-zero) before the TTL could lapse and let a second supervisor take over. Release on shutdown. - Canonical default pidfile keyed on brain id (currentBrainId, config-only, no DB connect) so two brains under one HOME no longer share supervisor.pid. - doctor: new supervisor_singleton check surfaces the effective --max-rss (from the started audit event) and warns when the lock holder's (host,pid) differs from the local pidfile — comparing host+pid, not bare pid. Registered in doctor-categories. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(agent-voice): topic-aware persona context via server-resolved topicId (#1851) Summon Mars/Venus into a specific conversation topic so they boot already knowing the recent thread. A per-topic call link carries only topicId (+ optional display topicName); the server resolves the recent-conversation context from the brain (topics/<topicId>.md) — topic CONTENT is never accepted over the wire (that would be prompt injection + a leak into URLs/referrers/logs). topicId is a strict slug with a path-traversal guard. New '# Topic Context' prompt slot injected after the persona body so identity-first ordering still wins; persona identity unchanged. No topicId -> generic behavior. Contract doc + persona skill docs updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: file #1737 slot-reservation fair-scheduling follow-up TODO (F7) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.29.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync KEY_FILES.md for v0.42.29.0 minions + abort wave (#1737, #1849) Fold the #1737/#1849 behavior into the existing per-file entries and add the two new core files, keeping the doc at current-state truth: - queue.ts: honest attempt accounting on wall-clock + stall dead-letter paths; defaultTimeoutMsFor stamping at submit. - supervisor.ts: queue-scoped DB singleton lock (supervisorLockId, classifySupervisorSingleton, LOCK_LOST, refresh-fail-safe, brain-id pidfile, max_rss_mb audit). - worker-registry.ts: currentDbIdentity(). - New entries: src/core/abort-check.ts, src/core/minions/handler-timeouts.ts. - New doctor.ts extension: supervisor_singleton check. - cycle.ts / embed.ts extensions: AbortSignal threading note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f7f8512b14 |
v0.42.28.0 fix(engine): batch inserts use jsonb_to_recordset, not text[] array literals (#1861) (#1927)
* fix(engine): batch inserts use jsonb_to_recordset, not text[] array literals (#1861) addLinksBatch/addTimelineEntriesBatch/addTakesBatch passed free text through unnest(${arr}::text[]); postgres.js serialized it to a Postgres text[] literal that array_in rejected ("malformed array literal") on calendar/Zoom context, aborting the whole `extract links --stale` sweep. Bind the batch as one JSONB doc via jsonb_to_recordset(($1::jsonb)->'rows') through the audited executeRawJsonb contract instead. Shared row builders (src/core/batch-rows.ts) keep both engines byte-identical; NUL is stripped only from free-text body fields (context/summary/detail/claim), while identity/security fields (slugs/source_ids/holder/kind/dates) still reject NUL. addTakesBatch is now batchRetry-wrapped ('addTakesBatch' audit site) and its BrainEngine signature takes BatchOpts. Scalar addLink context is NUL-stripped too. Regression tests on both engines: PGLite always-on poison/NUL/parity suite + DATABASE_URL-gated Postgres lane (the engine that actually crashed). * test: make "no Anthropic key" tests hermetic via withoutAnthropicKey hasAnthropicKey() reads both ANTHROPIC_API_KEY and ~/.gbrain config; tests that only deleted the env var fired a real LLM call on configured machines (warning flipped NO_ANTHROPIC_API_KEY -> LLM_OUTPUT_NOT_JSON). New test/helpers/no-anthropic-key.ts neutralizes both sources (env + GBRAIN_HOME temp dir) for the duration of the call. Refactors the five no-key tests in think-pipeline + takes-mcp-allowlist to use it, including two that previously passed only by luck of the live LLM output. * chore: docs + version bump (v0.42.28.0) KEY_FILES.md/RETRIEVAL.md describe the jsonb_to_recordset batch path; TODOS.md files the #1861 follow-ups (element-isolation, remaining ::text[] sites, shared SQL-string hoist, batch-insert edge-case tests). CHANGELOG + VERSION + package.json to 0.42.28.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync TESTING.md batch-insert references for v0.42.28.0 The #1861 fix migrated links/timeline/takes batch inserts from unnest(::text[]) to jsonb_to_recordset. Update the stale "postgres-js unnest() binding" note and add the two new poison-regression test files (test/links-timeline-jsonb-poison.test.ts PGLite half, test/e2e/jsonb-batch-poison-postgres.test.ts Postgres lane) to the inventory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sql-query): reject top-level array jsonb params in executeRawJsonb (#1861 P2a) The "no top-level array" rule was only a comment. A bare JS array bound to a $N::jsonb position can serialize as a Postgres array literal (not jsonb) through postgres.js, silently re-entering the "malformed array literal" class #1861 just escaped. executeRawJsonb now throws a clear error steering callers to the { rows: [...] } object wrapper. Verified breaks zero call sites (all pass objects or null). Codex adversarial P2a; batch-size enforcement (P2b) filed as a TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
805814451e |
v0.42.26.0 docs(supabase): update connection-string setup to new UI + Transaction pooler (#1848) (#1875)
* docs(supabase): update connection-string setup to new UI + Transaction pooler Supabase moved the connection string under "Connect" in the top nav and now shows three options (Direct, Transaction pooler, Session pooler). Update the tutorial, gbrain init prompts, the setup skill, the verify runbook, and the live-sync guide to recommend the Transaction pooler (port 6543) — which gbrain is tuned for (prepared statements disabled, DDL/locks routed to a derived direct connection). Document the IPv4 footgun: the derived direct connection is IPv6-only, so on IPv4-only hosts reads work but sync silently skips pages. Tutorial 7c now leads with the free fix (GBRAIN_DIRECT_DATABASE_URL -> Session pooler, port 5432) and keeps the IPv4 add-on as the paid alternative. Removes stale "transaction mode breaks sync (.begin() is not a function)" warnings and the port-6543 "Session pooler" mislabels. Extends PR #1848 by @FilipHarald. * chore: bump version and changelog (v0.42.26.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9a0bae8d62 |
v0.42.25.0 fix(pricing): unify chat-model pricing into one canonical source; add Opus 4.8 (#1819) (#1827)
* fix(pricing): unify chat-model pricing into one canonical source; add Opus 4.8 (#1819) Single canonical CANONICAL_PRICING table (src/core/model-pricing.ts) with canonicalLookup; ANTHROPIC_PRICING and takes-quality MODEL_PRICING become derived views. cost-tracker, cross-modal runner, skillopt preflight, brainstorm orchestrator, and brain-score all source from it. Adds Opus 4.8 ($5/$25) so --max-cost-usd and the dream-cycle budget meter enforce on 4.8 runs; fixes a stale Opus 4.7 $15/$75 in the takes-quality gate and reconciles Gemini 2.0 Flash to $0.10/$0.40. Because every table derives from canonical, cross-table price drift is structurally impossible. * chore: bump version and changelog (v0.42.25.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document canonical chat-pricing table for v0.42.25.0 Add KEY_FILES.md entries for src/core/model-pricing.ts (canonical CANONICAL_PRICING + canonicalLookup) and refresh the now-derived anthropic-pricing.ts + takes-quality-eval/pricing.ts entries to current-state. Add the "one canonical chat-pricing table" cross-cutting invariant to CLAUDE.md. Fix the stale model-price snapshot pointer in SEARCH_MODE_METHODOLOGY.md (anthropic-pricing.ts -> model-pricing.ts). Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f868257405 |
v0.42.24.0 fix(minions): route lock claim/renewLock through direct session pool (#1822)
* fix(minions): route lock claim/renewLock through direct session pool The Minion lock heartbeat (claim + renewLock) ran every UPDATE through engine.executeRaw(), which is hardcoded to the read pool. On Supabase that is the transaction-mode pooler (6543), which recycles connections per transaction. A lock is held for minutes, so the pooler periodically reaps the socket mid-heartbeat -> CONNECTION_ENDED -> the lock looks expired -> the worker force-evicts its own job and the claim loop wedges silently. Add BrainEngine.executeRawDirect(): same contract as executeRaw, but routes to the direct session-mode pool (5432, GBRAIN_DIRECT_DATABASE_URL) when dual-pool is active. No-op delegation on PGLite / non-Supabase / kill-switch. claim/renewLock now use it. Single-statement UPDATEs only, so the double-claim guard and the renewLock no-inline-retry contract are preserved. Statements inside an open transaction keep their tx connection (in-transaction guard keys on peekReadPool() !== _sql); the lock hot-path never runs inside transaction(). The Postgres impl shares its cancellation plumbing with executeRaw via a private runUnsafe helper. New test/postgres-execute-raw-direct.test.ts covers the routing decision (dual-pool on/off x in-tx/not + abort short-circuit) without a live Postgres; queue-lock-retry.test.ts gains a guard that claim can never fall back to executeRaw. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.24.0 chore: bump version and changelog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.42.24.0 Document executeRawDirect on the BrainEngine contract and the claim/renewLock direct-session-pool routing in KEY_FILES.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make D3 executeRaw-no-retry guard refactor-aware The DRY refactor in this PR extracted executeRaw/executeRawDirect's shared cancellation plumbing into a private runUnsafe(conn, ...) helper, so the single conn.unsafe() call moved out of executeRaw's body. The D3 guard read executeRaw's source and asserted conn.unsafe( appeared exactly once there, which now fails (it's zero — executeRaw delegates). The D3 invariant (no per-call retry wrapper) is unchanged; it just spans the delegate now. Update the guard to check both public methods delegate to runUnsafe without reconnect/retry, and assert the exactly-once conn.unsafe + cancel-only catch in runUnsafe. Also extends coverage to executeRawDirect so the lock hot-path can't reintroduce a retry wrapper either. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f11d56cfca |
v0.42.23.0 feat(jobs): --nice scheduling-priority flag for jobs work/supervisor (#1815) (#1820)
* feat(jobs): niceness core, worker registry, shared supervisor-pid reader OS scheduling-priority primitives for issue #1815: - niceness.ts: parseNiceValue (whole-string), applyNiceness (re-reads effective in success AND failure paths), getEffectiveNiceness, formatNice. - worker-registry.ts: live workers self-register pid + requested/effective nice under gbrainPath('workers'); readWorkers prunes ESRCH (keeps EPERM) with a pid-reuse start-time guard. - supervisor-pid.ts: readSupervisorPid extracted from the copy-pasted PID-file + liveness block. * feat(jobs): --nice flag for jobs work/supervisor + doctor niceness check Wires the --nice <n> flag (and GBRAIN_NICE env) through the CLI (issue #1815): - jobs work: applies niceness + registers the worker; cleanup on finally and process.on('exit'). - jobs supervisor: applies in the foreground-start path only (after the --detach fork), passes the apply result into MinionSupervisor. - supervisor.ts: nice opts, extracted testable buildWorkerArgs (appends --nice), emits niceness on started/worker_spawned audit events. - jobs stats / supervisor status: surface effective worker + supervisor nice. - doctor: separate supervisor_niceness check (warns on requested != effective) so it can't clobber the supervisor crash-check precedence; registered in doctor-categories. * test(jobs): cover niceness, worker registry, supervisor-pid, build args Unit tests for issue #1815: parseNiceValue rejects 3.5/10abc that parseInt would accept; applyNiceness re-reads effective on EPERM; registry ESRCH/EPERM + pid-reuse guard + brain-isolated path; readSupervisorPid states; parseNiceFlag flag>env precedence; buildWorkerArgs --nice propagation. * chore: bump version and changelog (v0.42.23.0) --nice flag for jobs work/supervisor (issue #1815). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document --nice flag for jobs work/supervisor (v0.42.23.0) - minions-deployment.md: niceness tuning section (full concurrency, low priority). - KEY_FILES.md: entries for niceness.ts, worker-registry.ts, supervisor-pid.ts; supervisor.ts entry notes buildWorkerArgs + nice opts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): add enrich_thin to dream cycle EXPECTED_PHASES The enrich_thin cycle phase (src/core/cycle.ts ALL_PHASES, between conversation_facts_backfill and skillopt) shipped without updating the e2e phase-order expectation, so dream-cycle-phase-order-pglite failed on master. Sync the expected list to the real ALL_PHASES order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): align sync-lock-recovery with the shipped --break-lock --all contract v0.41.13.0 intentionally dropped the "--break-lock + --all is refused" guard so cron can self-heal every source in one call (sync.ts runBreakLock iterates sources under --all). The e2e test still asserted the old exit-1 refusal and failed on master. Assert the current contract: the combination is accepted and takes the iterate / no-active-sources path (exit 0, no refusal message). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): de-flake ingestion-roundtrip chokidar first-drop race The native fsevents watcher occasionally missed a freshly written file, timing out the 15s waitFor (~1/3 on master under load). Three fixes: - inject a polling chokidar watcher via the source's _watchFactory seam (usePolling, 20ms interval) so detection never depends on fsevents timing; - drop deterministic fixtures BEFORE start so the initial scan (ignoreInitial:false) emits them, keeping live-watch coverage only where it's robust; - poll for the dedup hit instead of a fixed 600ms sleep. 15/15 green under stress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): make hermetic-PGLite serve tests actually hermetic connect-bearer and serve-stdio-roundtrip init a PGLite brain and spawn serve, but passed {...process.env} through — leaking an ambient DATABASE_URL / GBRAIN_DATABASE_URL into the subprocess, which then came up on Postgres and failed the `engine: pglite` assertion. Strip both DB vars from the spawned env so the tests are deterministic whether or not the shell/CI has a DB URL set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): type the hermetic-PGLite env so tsc passes The DATABASE_URL/GBRAIN_DATABASE_URL strip used `delete` on a narrowly-typed env literal (tsc-only failure; bun test doesn't typecheck). Annotate connect-bearer's env as Record<string,string|undefined> and build serve-stdio's as a concrete Record<string,string> (StdioClientTransport.env rejects undefined). Runtime behavior unchanged (7/7 + 3/3 green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: quarantine worker-registry to *.serial (R1 env-mutation isolation) worker-registry.test.ts sets process.env.GBRAIN_HOME per-test so gbrainPath resolves to a temp dir, then lazy-imports the module — a process-global mutation the parallel isolation lint (rule R1) forbids. Rename to worker-registry.serial.test.ts: it runs in the serial pass (own bun process, max-concurrency=1) where env mutation is safe, and the lint skips *.serial files. No logic change (6/6 green); fixes the failing `verify` CI job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f4959348c2 |
v0.42.22.0 fix(minions): supervisor progress watchdog + worker DB self-defense — alive-but-wedged worker self-heals (#1801) (#1824)
* fix(minions): supervisor progress watchdog + worker DB self-defense under supervision (#1801) Alive-but-wedged worker (dead DB pool, process still up) now self-heals in minutes instead of a silent 15h halt. - supervisor: progress watchdog restarts a child that makes no forward progress on claimable work (name+queue-scoped, active_healthy/due-delayed aware, startup-grace + loop-budget bounded); runtime handler-name derivation. - child-worker-supervisor: killChild gates on liveness not .killed (also fixes the existing shutdown SIGKILL no-op); restartCurrentChild kills the captured child ref; intentional restart doesn't count toward max_crashes. - worker: DB-liveness probe runs under supervision (db_dead self-exit), stall detection stays supervised-off. - doctor: standalone per-queue wedged_queue check + state->status fix in the remote queue_health check. - jobs/queue: queue-scoped getStats wedge fields + jobs stats WEDGED line. * fix(minions): wedge_restart_loop one-shot + supervised-probe comment + jobs-stats threshold (review) Pre-landing adversarial review findings: - wedge_restart_loop warn now fires once per exhausted window via a re-arming flag, not every health tick (was flooding the audit log for the full window). - Correct the stale GBRAIN_SUPERVISED comment: the DB probe runs under supervision now; only stall detection is skipped. - jobs stats WEDGED line reads GBRAIN_WEDGED_QUEUE_WARN_MINUTES so it agrees with the doctor wedged_queue threshold. * chore: bump version and changelog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: queue-ops runbook + KEY_FILES for the #1801 wedge watchdog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f3ade6c0c3 |
v0.42.21.0 fix(postgres): module-singleton ownership — canonical landing for the dream-cycle "connect() has not been called" class (#1404/#1471/#1619) (#1805)
* fix(postgres): module-singleton ownership — borrower disconnect no longer nulls the cycle's connection (#1404/#1471/#1619) gbrain dream on Postgres failed every DB phase with "No database connection: connect() has not been called": a short-lived borrower probe engine (lint/doctor config-lift, no poolSize) called db.disconnect() in its own disconnect(), nulling the shared module singleton the long-lived cycle owner was still using. The module `sql` is only ever nulled by db.disconnect() (postgres.js auto-reconnects its own pool), so the failure was always a borrower-disconnect, never an idle-pooler drop. Fix: db.connect() returns whether THIS call created the singleton (atomic — no await between the null-check and the sql=postgres() assignment), PostgresEngine stores it as _ownsModuleSingleton, and disconnect() only calls db.disconnect() when it owns the connection. Borrowers no-op. Hardening: db.disconnect() snapshots +nulls sql before awaiting end(); reconnect() shares one in-flight _reconnectPromise. Tests: new postgres-engine-singleton-ownership.test.ts; expanded DB-gated e2e matrix (owner/borrower, creation-not-role, symmetric CLI-exit, owner-reconnect- with-live-borrower); module-style getter asymmetry; #1570 shared-recovery regression updated to assert the fixed contract. Co-Authored-By: nullhex-io <noreply@github.com> Co-Authored-By: joelwp <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.15.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: nullhex-io <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
ec5fed2921 |
v0.42.20.0 fix: reliability wave — PGLite capture lock-pin + Postgres reconnect race + search embed-hang (#1762 #1745 #1775) (#1810)
* fix(core): drain fire-and-forget sinks before disconnect via a background-work registry (#1762) New src/core/background-work.ts registry (Map<name,drainer>, ordered drain, awaited abort). facts-queue (order 0, abort=shutdown), last-retrieved (1), and eval-capture (3, now self-tracked) register as sinks. Both CLI exit paths (op-dispatch finally + handleCliOnly finally) drain the registry before engine.disconnect() so a PGLite db.close() can't race in-flight work into the re-pump busy-loop that pinned the single-writer lock. Op-dispatch error path converts process.exit(1) to exitCode+return so the finally still drains. * fix(ai): bound every outbound AI call so a stalled provider can't hang (#1762/#1775) withDefaultTimeout composes a per-touchpoint default deadline (chat 300s, embed/multimodal 60s) with any caller signal via AbortSignal.any. Applied at the SDK call layer (chat generateText, expand generateObject, OCR, per-sub-batch embed) — covers native-anthropic + retries — plus per-request multimodal fetch. embedQuery forwards abortSignal. Env: GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS. * fix(postgres): module-mode reconnect preserves the shared singleton (#1745) reconnect() branches on connection style. Module-singleton engines re-establish idempotently via db.connect() (no-op when alive) + refresh the ConnectionManager read pool, never db.disconnect() — so a transient blip no longer nulls the shared sql out from under concurrent ops (which threw 'connect() has not been called'). Fail-loud on real connect failure. Instance pools keep teardown+recreate. * fix(search): bound the query-time embed so a stall falls back to keyword (#1775) search/query default to cheap-hybrid (embeds the query); a stalled provider made the embed never settle, so the keyword fallback never engaged and the command force-exited with no output. One shared QueryEmbedDeadline (6s, floored 2s per embed) covers both the cache-lookup and inner embeds via embedQueryBounded (abortSignal + Promise.race) → existing keyword fallback engages. Also registers the search-cache background-work drainer (now bounded). Env: GBRAIN_QUERY_EMBED_TIMEOUT_MS. * test+chore: reliability wave tests + v0.42.11.0 (#1762 #1745 #1775) New: background-work registry unit, query-embed deadline unit, eval-capture drain unit, postgres reconnect E2E (#1745), gbrain capture exit-clean case in the PGLite serial test. Updated fix-wave-structural assertions to the registry shape. VERSION/package.json/CHANGELOG -> 0.42.11.0; TODOS retrofit marked done. Incorporates + hardens PR #1763 (drain-before-disconnect + embed fetch timeout); the residual hung-Haiku hole is closed by the facts shutdown() abort belt. Co-Authored-By: ElliotDrel <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document background-work registry + v0.42.11.0 reliability wave in CLAUDE.md (regen llms) * chore: bump release version 0.42.11.0 -> 0.42.20.0 Rename the reliability-wave release version per request. Trio (VERSION / package.json / CHANGELOG) reconciled; in-code version-tag comments and test fixtures updated; llms regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ElliotDrel <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
3d2add15d9 |
v0.42.19.0 fix(skillopt): close the last gap in the AI SDK v6 tool-loop fix (write-capture mapper + regression test) (#1809)
* fix(ai/gateway): AI SDK v6 tool loop on non-Anthropic providers (#1782, #1764) v6's asSchema() rejected the bare { jsonSchema } object and called it as a function ("schema is not a function"), killing every tool-using agent run on non-Anthropic providers and skillopt on all providers. Wrap tool inputSchema with the SDK jsonSchema() helper, and add a pure toModelMessages() boundary adapter in chat() that converts tool results to the v6 ModelMessage shape (role:'tool' + typed output:{type:'json'|'error-text',value}, isError dropped, non-JSON-safe output normalized). toolLoop stays provider-neutral and unchanged. Both skillopt tool builders (rollout.ts, write-capture.ts) switch to the shared paramDefToSchema mapper so enum/default/items survive. Tests run the produced shapes through real generateText + MockLanguageModelV3. Co-Authored-By: michaeladair44 <michaeladair44@users.noreply.github.com> Co-Authored-By: justemu <justemu@users.noreply.github.com> Co-Authored-By: JE4NVRG <JE4NVRG@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.19.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: michaeladair44 <michaeladair44@users.noreply.github.com> Co-authored-by: justemu <justemu@users.noreply.github.com> Co-authored-by: JE4NVRG <JE4NVRG@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bde11bb18f |
v0.42.18.0 fix: sync orphan-pileup watchdog (#1633) + links-lag µs stamp (#1768) (#1807)
* fix(extract): links_extraction_lag never clears on Postgres (#1768) Stamp the full-microsecond updated_at (via to_char ... AT TIME ZONE UTC) instead of the millisecond-truncated JS Date, so links_extracted_at equals the DB updated_at exactly and the staleness predicate clears. Stamp SQL unchanged: version-arm backdating still works, D4 preserved, CDX-1 strengthened. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(core): out-of-band hard-deadline watchdog primitive (#1633) Bun eval-Worker that SIGTERM->grace->SIGKILLs its own process from a separate OS thread, so a sync whose main event loop is starved (ReDoS spin) still dies. Signals SELF (no PID-reuse footgun). Empirically validated on Bun 1.3.13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): arm hard-deadline watchdog + graceful SIGINT cancel (#1633) cli.ts installs the watchdog before connectEngine (bounds connect hangs); resolveSyncHardDeadline + composeAbortSignals in sync.ts; SIGINT graceful cancel on single-source + --all; withRefreshingLock timer unref'd. Non-TTY default 3600s makes cron orphan-pileup structurally impossible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(claude): annotate process-watchdog + #1768/#1633 fixes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: rebump v0.42.13.0 -> v0.42.18.0 (queue collision) Sibling workspaces claimed v0.42.13-v0.42.17; advance this branch's slot. VERSION + package.json + CHANGELOG header + CLAUDE.md annotations + llms bundles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): current-state phrasing for #1633/#1768 entries (fix check:doc-history) The doc-history guard bans the bolded **v0.X release-clause marker in reference docs (history belongs in CHANGELOG + git). Rewrote the extract.ts/sync.ts additions as current-state prose and de-versioned the process-watchdog entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
fd2fde9d26 |
v0.42.17.0 fix(sync): resumable incremental sync — killed mid-import no longer loses progress (#1794) (#1808)
* feat(sync): checkpoint primitives for resumable sync (#1794) - op-checkpoint.ts: syncFingerprint({sourceId, lastCommit}) keyed on the anchor (never HEAD) so the checkpoint survives a growing backlog. - source-health.ts: commitTimeMs(localPath, sha) for stamping newest_content_at against a pinned (non-HEAD) commit. - sync-concurrency.ts: resolveMaxConnections + clampWorkersForConnectionBudget for the opt-in GBRAIN_MAX_CONNECTIONS single-sync footprint clamp. * feat(sync): resumable incremental sync via pinned-target checkpoint (#1794) performSyncInner now drains a fixed lastCommit..pin range, banking completed file paths to op_checkpoints and advancing last_commit (+ last_sync_at) ONLY at full import completion. A killed/aborted/blocked run leaves the anchor untouched and resumes from the banked set next run — the convergence fix. - Pinned target: completion advances to the pin, not live HEAD, so commits landing after the pin are a clean next-sync diff (kills the staleness window). History rewrite (pin not an ancestor of HEAD) discards the checkpoint + re-pins. - Forward-progress head gate: merge-base --is-ancestor pin HEAD replaces the strict "HEAD == captured" gate that blocked on every concurrent enrich commit. - Vanished-on-disk added file -> skip + checkpoint, not a failedFiles block. - Large syncs defer extract/embed to the resumable --stale sweeps (convergence == import convergence); small syncs keep inline extract/facts/embed. - GBRAIN_MAX_CONNECTIONS clamp on the worker fan-out (opt-in). - Typed SyncLockBusyError; the Minion sync handler (jobs.ts) marks the job SKIPPED (not failed) on a held lock so cron/autopilot defers cleanly. * feat(doctor): pool_budget check for GBRAIN_MAX_CONNECTIONS (#1794) computePoolBudgetCheck + checkPoolBudget warn when the parent pool leaves no room for a parallel sync worker under GBRAIN_MAX_CONNECTIONS, pointing at GBRAIN_POOL_SIZE=2. Registered in the ops category set. * test(sync): resumable-sync regression suite + vanished-file contract (#1794) - sync-resumable-import.serial.test.ts (13 cases): convergence regression, resume-skips-checkpointed, pinned-target/forward-drift, history-rewrite re-pin, last_sync_at-not-bumped-on-block + good-file banking, vanished-file skip, dry-run/empty-diff, + pure fingerprint/clamp/pool-budget helpers. - sync-parallel.test.ts: vanished-mid-sync added file now asserts the new skip contract (supersedes the v0.22.13 CODEX-3 failedFiles behavior). * chore: bump version and changelog (v0.42.17.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
3fe449361c |
v0.42.16.0 feat(doctor): brain health as a solved problem — cause-ranked doctor + OOM-loop line + auto-drain + pool-reap (#1685) (#1802)
* feat(minions): pool-recovery audit + reconnect reason-threading + shared drain helper (#1685 GAP B, 5A) - pool-recovery-audit.ts: reap_detected (CONNECTION_ENDED) vs reconnect_other; recovered/failed split - postgres-engine reconnect(ctx?) classifies the triggering error so only true pooler reaps are tagged (CODEX #8) - retry.ts reconnect callback widened to thread the error; retry-matcher isConnectionEndedError - runExtractAtomsDrainForSource shared helper (cycleLockIdFor + withRefreshingLock) — one drain path (5A) - supervisor-audit readRecentSupervisorEvents (current+prev ISO week, CODEX #7) - extract-atoms-drain PROTECTED; autopilot.auto_drain.* config keys * feat(doctor): worker_oom_loop + pool_reap_health checks + cause-ranked top_issues (#1685 GAP A/B/C) - computeWorkerOomLoopCheck: unions supervisor rss_watchdog + minion_jobs watchdog-abort (CODEX #5), cap fallback to resolveDefaultMaxRssMb (CODEX #6) - computePoolReapHealthCheck: reaps-not-recovering fail, thrash warn - doctor-cause-rank rankIssues: tier ordering + grounded downstream_of (CODEX #9) + drift guard (4A) - supervisor causeStr + queue_health cross-reference worker_oom_loop (DRY 1C) - register both checks in doctor-categories ops * feat(autopilot): per-source extract_atoms auto-drain + handler + dream --drain refactor (#1685 GAP D) - autopilot per-source gate: enabled + !packDeclares + backlog>threshold + daily cap; time-sloted idempotency key (CODEX #2) - extract-atoms-drain Minion handler (thin wrapper, LockUnavailableError -> deferred) - dream --drain routes through the shared helper (5A) * chore: bump version and changelog (v0.42.12.0) #1685 brain-health-as-solved-problem: cause-ranked doctor, worker_oom_loop line, per-source auto-drain, pool-reap health. Layers on #1678/#1735. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(todos): file #1685 GAP E + remote-path follow-ups (v0.42.12.0) * fix(#1685): pre-landing review — multi-source auto-drain, honest pool-reap signal, lock-renewal reap labeling - autopilot: drop maxWaiting (coalesces by name+queue not source → only one source drained + cap over-count); pre-check idempotency key so only genuinely-new sources submit+count - pool_reap_health: fail on reconnect FAILURES (the real signal), not reaps>0&&failures>0 (false causality when a recovered reap + unrelated failure co-occur) - lock-renewal-tick threads its triggering error to reconnect() so a CONNECTION_ENDED pooler reap is labeled reap_detected not reconnect_other (pool_reap_health now fires for the #1678 incident path) * chore: re-version v0.42.12.0 → v0.42.16.0 (#1685) Slot collision avoidance per queue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: restore slim CLAUDE.md + move #1685 entries to KEY_FILES.md (fix check:doc-history) The master merge wrongly kept the pre-restructure 577KB CLAUDE.md; the check:doc-history guard caps it at 60KB. Take master's slim CLAUDE.md and record the #1685 files (doctor-cause-rank, pool-recovery-audit, worker_oom_loop + pool_reap_health checks, auto-drain, 5A helper) as current-state prose in docs/architecture/KEY_FILES.md (no release markers). llms regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
488f89e0dc |
v0.42.15.0 fix: decouple CLI primary output from process.stdout.isTTY (#1784) (#1806)
* feat(eval): cycle-default — single source of truth for TTY/non-TTY cycle count (#1784) * fix(jobs): decouple 'jobs watch' format (--json) from loop (--follow); non-TTY prints one human snapshot (#1784) * fix(reindex): human cost-refusal unless --json; spend guardrail unchanged (#1784) * fix(eval): annotate non-interactive cycle/budget defaults; runner core TTY-agnostic (#1784) * chore: bump version and changelog (v0.42.15.0) Decouple CLI primary output from process.stdout.isTTY (#1784): human by default, JSON only with --json; jobs watch non-TTY one-shots; eval banners annotate non-interactive defaults; reindex-code refusal is human unless --json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: CLAUDE.md Key Files notes for v0.42.15.0 isTTY-output decoupling (#1784) Annotate jobs.ts (jobs watch format/loop split + resolveWatchMode + the new cycle-default.ts), reindex-code.ts (human cost-refusal), and eval-cross-modal.ts (cycle/budget banner annotations). Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
1036f8f752 |
v0.42.14.0 fix(zero-config): code-* readiness signal + init embedding-key validation + lock self-heal (#1780) (#1804)
* feat(code-intel): readiness signal on code-def/refs/callers/callees (#1780 Gap 1) New src/core/code-graph-readiness.ts: resolveCodeReadiness() returns a typed status (not_built | indexing | ready | unknown) + ready boolean so callers can tell "graph not built / still indexing" apart from "genuinely no match" when count===0. EXISTS-based (cheap), chunk-grain, resolver-version-matching pending predicate, fail-open. Wired into the 4 CLI envelopes (+ human hint) and the 4 MCP op handlers. def/refs are 2-state brain-wide; callers/callees 3-state scoped. * feat(db-lock): automatic same-host dead-pid cycle-lock takeover (#1780 Gap 3) tryAcquireDbLock now reclaims a held, not-TTL-expired lock when the same-host holder is provably dead (process.kill ESRCH) past a 60s grace, via guarded DELETE + one normal-upsert retry returning the normal handle. New shared injectable classifyHolderLiveness/isHolderDeadLocally (EPERM treated as ALIVE — never steals a live lock). runBreakLock's safe path consumes the shared predicate, fixing its prior EPERM-as-dead bug. Cross-host stays TTL-only. * feat(init): validate the embedding key at gbrain init (#1780 Gap 2) New src/core/init-embed-check.ts: config-only diagnoseEmbedding (missing key, all providers) + best-effort 1-token live test-embed (invalid/expired key, 5s timeout, never blocks). Loud warning to stderr, init still exits 0; skipped by --no-embedding / --skip-embed-check / GBRAIN_INIT_SKIP_EMBED_CHECK=1. Builds the effective env (process.env + file-plane keys + --key) via buildGatewayConfig, extracted to src/core/ai/build-gateway-config.ts (cli.ts re-exports) so the check sees the same keys + provider base URLs as runtime. embedding_check added to --json. * chore: bump version and changelog (v0.42.14.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document the #1780 zero-config gaps for v0.42.14.0 CLAUDE.md Key Files: add src/core/code-graph-readiness.ts, init-embed-check.ts, ai/build-gateway-config.ts, and the db-lock auto-takeover + code-* readiness field behaviors. Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bea2d3e6c9 |
v0.42.13.0 fix(search): archive/ content findable by default, demoted not hard-excluded (#1777) (#1797)
* fix(search): archive/ findable by default — demote not hard-exclude (#1777) Move archive/ out of DEFAULT_HARD_EXCLUDES into a 0.5 source-boost demote so archived historical content is findable by default, ranked below curated content. Add a hidden_by_search_policy doctor check so the surviving exclude policy (test/, attachments/, .raw/) is auditable. Bump KNOBS_HASH_VERSION 8->9 so the policy change invalidates archive-excluded query_cache rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct search-exclude.test.ts annotation for archive demote (#1777) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a57d98b813 |
v0.42.12.0 feat: self-upgrading gbrain — invocation-riding update check + opt-in auto-upgrade (#1798)
* feat(self-upgrade): decision/cache/snooze foundation + atomic binary self-update Pure decideSelfUpgrade (invocation + autopilot channels), atomic untrusted cache + escalating snooze + shared marker grammar (forged-marker rejection), semver helpers, and real darwin-arm64/linux-x64 binary self-update (download -> fsync -> smoke -> atomic rename; failure leaves old binary intact). Tests incl. real-HTTP-server swap E2E. * feat(self-upgrade): check-update cache/markers, self-upgrade command, CLI heartbeat hook check-update gains gstack-style cache/snooze/markers + refreshUpdateCache + exported fetchLatestRelease. New 'gbrain self-upgrade' command. cli.ts emits the update marker on every invocation (cache-read-only hot path, detached single-flight refresh, skip-set + recursion guard + NODE_ENV=test gate). * feat(self-upgrade): autopilot silent channel, doctor check, runPostUpgrade setup, config + identity marker autopilot opt-in silent channel (auto+quiet+idle, swap-only+breadcrumb+exit-relaunch) + installSystemd Restart=always + migrateSystemdUnitToRestartAlways. doctor self_upgrade_health. runPostUpgrade applySelfUpgradeSetup (one-time consent + systemd rewrite). init defaults mode=notify. config self_upgrade plane + KNOWN_CONFIG_KEYS. get_brain_identity carries update marker. * docs(self-upgrade): gbrain-upgrade agent skill, RESOLVER/manifest, auto-update doc reversal, HEARTBEAT New skills/gbrain-upgrade agent flow (mirror gstack-upgrade) wired into RESOLVER + manifest. upgrades-auto-update.md reversed to document opt-in auto + conservative gates. HEARTBEAT self-upgrade --check-only line. llms-full regenerated. * chore: bump version and changelog (v0.42.12.0) Self-upgrading gbrain: invocation-riding update marker + opt-in autopilot silent channel + real atomic binary self-update. Mirrors gstack's mechanism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(self-upgrade): write just-upgraded-from breadcrumb + clear stale cache after upgrade Codex ship-review P3: the CLI startup hook reads just-upgraded-from to print the one-time JUST_UPGRADED confirmation, but nothing wrote it — dead path. runUpgrade now writes the breadcrumb (covers full + --swap-only) and clears the update-check cache + snooze so a now-applied 'upgrade available' marker stops nudging. * feat(self-upgrade): surface what's-new in notify + wire agent integration (AGENTS.md, HEARTBEAT) + e2e - self-upgrade --check-only --json now includes changelog_diff + release_url (export fetchChangelog); the gbrain-upgrade skill shows 3-5 what's-new bullets before the 4-option prompt instead of just version numbers. - setup injects a self-upgrade marker protocol into AGENTS.md so interactive agents (Claude Code, Codex) act on the UPGRADE_AVAILABLE stderr marker — the piece that makes notify actually fire for them. - HEARTBEAT daily beat routes through the gbrain-upgrade skill (OpenClaw/Hermes cron cadence); auto-mode daemons ride the autopilot tick. - e2e: real subprocess invocation proves the marker fires (notify emits; off/snooze/up-to-date silent; JUST_UPGRADED fires+clears; --quiet suppresses). Serial test: --check-only surfaces the changelog. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
d4211f4176 |
v0.42.11.0 feat(skillopt): held-out eval gate, honest receipts, ENFORCE + ablation opts (#1759)
* feat(skillopt): wire held-out gate, honest receipts, ENFORCE + ablation opts Wire the F11 held-out gate into the orchestrator at checkpoint acceptance (runHeldOutGate was dead code); parse + thread --held-out through CLI, batch, fleet, background job, and the run_skillopt MCP op. Populate the real receipt.baseline_sel_score (was hardcoded 0) and add a final-test eval (test_score + baseline_test_score) via a shared scoreSkillOnTasks primitive. Fix the --no-mutate proposed.md write (was a stub) and enforce maxRuntimeMin. D16 ENFORCE in core mutation policy (assertBundledMutationHeldOut): mutating a bundled skill in place requires a non-empty (>=5), benchmark-disjoint held-out set or hard-refuses. Add three eval-internal ablation opts (reflectMode, disableValidationGate, optimizerMode='one-shot-rewrite') recorded in the receipt + audit; ROLLOUT_SUCCESS_THRESHOLD named constant. Security: run_skillopt MCP op validates skill_name (kebab-only) and confines caller-supplied benchmark/held-out paths to the skills dir for remote callers. * test(skillopt): held-out gate, ENFORCE, one-shot rewrite, runtime + receipt honesty New test/skillopt/rollout.test.ts (rollout had zero coverage). Held-out ENFORCE unit cases + one-shot-rewrite fence handling (whole-response unwrap, embedded-fence preserved, error path). E2E: F11 held-out BLOCKS/ALLOWS, bundled no-mutate write, reflectMode/disableValidationGate/optimizerMode, maxRuntimeMin abort, receipt baseline/test-score honesty, held-out/benchmark disjointness, D2 no-DB-pollution. * chore: bump version and changelog (v0.42.9.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0 Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and the tutorial's bundled-skill step: mutating a bundled skill in place now requires --allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update the receipt contract to the honest baseline/test-score fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): AI SDK v6 toolLoop compat — multi-turn tool calls work again The ai@6.x bump tightened ModelMessage + tool-schema validation, which silently broke every multi-turn tool loop. Both `gbrain skillopt` rollouts and production background `subagent` jobs route through `chat()`/`toolLoop` and crashed the moment the model called a tool ("messages do not match the ModelMessage[] schema" / "schema is not a function"). Surfaced end-to-end by the SkillOpt real-LLM eval. Three fixes: - chat(): wrap tool defs with the SDK's `jsonSchema()` helper instead of a bare `{jsonSchema}` object (v6 asSchema() treated the bare object as a thunk and threw). - chat(): new exported pure `toModelMessages()` converts gbrain's provider-neutral ChatMessage[] into v6 ModelMessage[] — tool results ride a dedicated `role:'tool'` message with structured `{type,value}` output; null output preserved as json null. Load-bearing for the production subagent path, not just skillopt. - rollout.ts: replace the inline params→schema mapper (dropped `items` on array params) with the shared `paramDefToSchema` single source of truth. Pinned by test/gateway-model-messages.test.ts (8 cases). Folds into the open v0.42.9.0 PR (#1759) — these complete the eval-readiness wave by making skillopt actually run against a live model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skillopt): budget no-pricing for Haiku silently scored every rollout 0 Surfaced by the SkillOpt real-LLM eval (Track B). Two coupled bugs that made a budget-capped Haiku run report a vacuous "0/N" measurement in ~2ms with zero LLM calls — indistinguishable from a real deficient-skill score: 1. Claude Haiku 4.5's canonical dateless id (`claude-haiku-4-5`) was missing from anthropic-pricing.ts (only the dated `-20251001` was present). With `--max-cost` set, BudgetTracker.reserve() threw no_pricing on the FIRST chat() of every rollout. Added the dateless entry (sonnet already had its dateless form). 2. runValidationGate swallowed that BUDGET_EXHAUSTED error — runWithLimit settled it as {ok:false}, which the gate turned into median:0. A pricing/cap crash became a fake score. The gate now scans settled results for isMustAbortError() and re-throws so the caller aborts loudly; ordinary (non-abort) rollout errors still fail-open to 0 (judge-hiccup posture kept). Pinned by test/skillopt/validate-gate-abort.test.ts (3 cases). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): llms-full.txt over size budget — drop what-schemas-unlock from full bundle The toolLoop + budget bug-fix annotations grew CLAUDE.md, pushing llms-full.txt to 756KB over the 750KB FULL_SIZE_BUDGET (the `build-llms > size budget` test failed, failing the `test` CI job). CLAUDE.md stays inlined by design (it's the point of the one-fetch bundle), so per the budget comment's own guidance ("ship with includeInFull=false exclusions") this excludes docs/what-schemas-unlock.md (15.4KB value-explainer, not load-bearing operational reference) from llms-full.txt; it stays linked in llms.txt. Bundle now 740KB with ~9KB headroom. No budget bump — 750KB is near the ~190k-token-context fit ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ci): re-admit policy docs into ci-cache-hash before doc relocation docs/**/*.md is deny-listed from the CI cache hash (test-irrelevant). The CLAUDE.md restructure moves test/release POLICY into docs/TESTING.md + docs/RELEASING.md, which DO carry contracts the test suite reads. Without re-admitting them, a policy-only edit would produce the same cache hash and skip the test shard that runs the build-llms + doc-history guards (false-pass). Adds an ALLOW_PATTERNS re-admit step after the deny, scoped to the named policy docs (not a blanket docs un-deny). Lands FIRST, before any doc moves. Pinned by 3 new cases in test/scripts/ci-cache-hash.test.ts: TESTING.md + RELEASING.md edits MUST change the hash; docs/guide.md still must not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate Key files / thin-client / Testing out of CLAUDE.md (verbatim) CLAUDE.md had grown to 592KB / ~147k tokens auto-loaded every session (~77% of the llms-full.txt single-fetch bundle). The per-file index was append-only by mandate. This is the exact thin-dispatcher-vs-fat-blob anti-pattern gbrain exists to fix, so CLAUDE.md becomes a thin orientation + resolver that points at on-demand docs. This commit is the VERBATIM move (content-preserving — the next commit compresses): - docs/architecture/KEY_FILES.md <- ## Key files + the calibration key-files cluster + Schema Cathedral v3 impl detail - docs/architecture/thin-client.md <- ## Thin-client routing - docs/TESTING.md <- ## Testing - ## Commands DROPPED (18 'added in vX.Y' history blocks; current surface is gbrain 0.41.38.0 -- personal knowledge brain USAGE gbrain <command> [options] SETUP init [--pglite|--supabase|--url] Create brain (PGLite default, no server) migrate --to <supabase|pglite> Transfer brain between engines upgrade Self-update check-update [--json] Check for new versions doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings) integrations [subcommand] Manage integration recipes (senses + reflexes) PAGES get <slug> Read a page put <slug> [< file.md] Write/update a page delete <slug> Delete a page list [--type T] [--tag T] [-n N] List pages SEARCH search <query> Keyword search (tsvector) query <question> [--no-expand] Hybrid search (RRF + expansion) ask <question> [--no-expand] Alias for query IMPORT/EXPORT import <dir> [--no-embed] Import markdown directory sync [--repo <path>] [flags] Git-to-brain incremental sync sync --watch [--interval N] Continuous sync (loops until stopped) sync --install-cron Install persistent sync daemon export [--dir ./out/] Export to markdown export --restore-only [--repo <p>] Restore missing supabase-only files [--type T] [--slug-prefix S] With optional filters FILES files list [slug] List stored files files upload <file> --page <slug> Upload file to storage files upload-raw <file> --page <s> Smart upload (size routing + .redirect.yaml) files signed-url <path> Generate signed URL (1-hour) files sync <dir> Bulk upload directory files verify Verify all uploads EMBEDDINGS embed [<slug>|--all|--stale] Generate/refresh embeddings LINKS link <from> <to> [--type T] Create typed link unlink <from> <to> Remove link backlinks <slug> Incoming links graph <slug> [--depth N] Traverse link graph (returns nodes) graph-query <slug> [--type T] Edge-based traversal with type/direction filters [--depth N] [--direction in|out|both] TAGS tags <slug> List tags tag <slug> <tag> Add tag untag <slug> <tag> Remove tag TIMELINE timeline [<slug>] View timeline timeline-add <slug> <date> <text> Add timeline entry TOOLS extract <links|timeline|all> Extract links/timeline (idempotent) [--source fs|db] fs (default) walks .md files; db iterates engine pages [--dir <brain>] brain dir for fs source [--type T] [--since DATE] filters (db source) [--dry-run] [--json] publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256) check-backlinks <check|fix> [dir] Find/fix missing back-links across brain lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter orphans [--json] [--count] Find pages with no inbound wikilinks salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type) transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only) dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly). See also: autopilot --install (continuous daemon). check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY) report --type <name> --content ... Save timestamped report to brain/reports/ BRAIN (capture / ideate / explore — v0.37/v0.38) capture [content] [--file PATH] Single entrypoint for getting content into the brain [--stdin] [--slug s] [--type t] Inline content / file / stdin; writes to inbox/ by default [--source ID] [--quiet|--json] Multi-source brains: route to a non-default source brainstorm <question> [--json] Bisociation idea generator (hybrid search + far-set + judge) [--save|--no-save] [--limit N] lsd <question> [--json] Lateral Synaptic Drift: inverted-judge brainstorm [--save|--no-save] [--limit N] rewarding far-from-obvious + axiomatic inversions SOURCES (multi-repo / multi-brain) sources list Show registered sources sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki') sources remove <id> Remove a source + its pages sync --all Sync all sources with a local_path sync --source <id> Sync one specific source repos ... DEPRECATED alias for 'sources' (v0.19.0) CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) code-def <symbol> [--lang l] Find the definition of a symbol across code pages code-refs <symbol> [--lang l] Find all references to a symbol (JSON-first) code-callers <symbol> Who calls this symbol? (v0.20.0 A1) code-callees <symbol> What does this symbol call? (v0.20.0 A1) query <q> --lang <l> Filter hybrid search to one language (v0.20.0) query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) sync --strategy code Sync code files into the brain JOBS (Minions) jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run] jobs list [--status S] [--limit N] List jobs jobs get <id> Job details + history jobs cancel <id> Cancel job jobs retry <id> Re-queue failed/dead job jobs prune [--older-than 30d] Clean old jobs jobs stats Job health dashboard jobs work [--queue Q] Start worker daemon (Postgres only) ADMIN stats Brain statistics health Brain health dashboard history <slug> Page version history revert <slug> <version-id> Revert to version features [--json] [--auto-fix] Scan usage + recommend unused features autopilot [--repo] [--interval N] Self-maintaining brain daemon config [show|get|set] <key> [val] Brain config storage status [--repo <path>] Storage tier status and health [--json] (git-tracked vs supabase-only) serve MCP server (stdio) serve --http [--port N] HTTP MCP server with OAuth 2.1 --token-ttl N Access token TTL in seconds (default: 3600) --enable-dcr Enable Dynamic Client Registration --public-url URL Public issuer URL (required behind proxy/tunnel) call <tool> '<json>' Raw tool invocation version Version info --tools-json Tool discovery (JSON) Run gbrain <command> --help for command-specific help. + the per-command KEY_FILES entries; content stays in git) CLAUDE.md gains: a Reference map (resolver), a Maintaining section (the anti-disease rule), and a Cross-cutting invariants subsection under Architecture so the must-never-violate rules (trust fail-closed, sourceScopeOpts isolation, JSONB trap, engine parity, contract-first, migrations, multi-source) still auto-load after the index moved out. Result: CLAUDE.md 592KB -> 61KB; llms-full.txt 740KB -> 210KB (new docs link-only until compressed). build-llms drift + budget test green; verify 29/29 green. The pre-move content is recoverable at git show <this^>:CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): compress relocated docs to current-state + add recurrence guard Compresses the verbatim-relocated reference docs from append-only release-history to current-state-only (the disease cure), then makes recurrence structurally impossible via a CI guard. Compression (fan-out subagents + adversarial verify, audited mechanically): - KEY_FILES.md 453KB -> 356KB; TESTING.md 42KB -> 38KB; thin-client.md already clean. - 393/393 entries preserved; every src/test/scripts path from the verbatim original survives (mechanical comm-check); zero bolded **v0. markers remain. - Conservative ratio (~22%) because the content is invariant-dense — correctness over brevity. Dropped: **vX.Y.Z (#NNN):** clauses, codex/review tags, contributor credits, PR-numbers-as-ids, pre-fix/then/was-now history deltas. Kept: every exported symbol, invariant, and Pinned-by reference. Verbatim original recoverable at git show <relocation-commit>:docs/architecture/KEY_FILES.md. Recurrence guard (scripts/check-key-files-current-state.sh, wired into verify + check:all): - HARD: bans the bolded **v0.<digit> marker in the reference docs (scoped — plain 'as of pgvector 0.7' prose is fine, no false positives). - HARD: CLAUDE.md size cap (90KB; currently 61KB) — the structural backstop. - Pinned by test/scripts/check-key-files-current-state.test.ts (7 cases). Content contracts (test/build-llms.test.ts, +5 cases per codex outside-voice): CLAUDE.md keeps inline ship IRON RULES (version format, document-release, never-hand-roll); AGENTS.md keeps its boot order; llms indexes the new docs; KEY_FILES stays link-only (not inlined). Privacy: scrubbed the relocated 'wintermute/chat/' source-boost examples + the literal harvest-lint regex to generic placeholders (legitimate in allowlisted CLAUDE.md; genericized for the new public docs per the privacy rule). Reverts the |
||
|
|
f09f9177a9 |
v0.42.10.0 feat(extract): opt-in global-basename wikilink resolution (closes #972) (#1388)
* v0.40.8.2 fix(extract): opt-in global-basename wikilink resolution (#972) Bare wikilinks like [[struktura]] that point at pages in another folder were silently dropped from the graph. The issue reporter saw 71 wikilinks in Obsidian render to 12 in gbrain (~83% lost). Symptoms downstream: `gbrain graph` returns thin neighborhoods, `gbrain backlinks` undercounts. This release adds an opt-in mode that resolves bare wikilinks by basename match, covers all three resolver surfaces (FS-source extract, DB-source extract, put_page auto-link), and emits one edge per match — no silent winner on ambiguity. `gbrain doctor` surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve under the new mode. Enable with: gbrain config set link_resolution.global_basename true gbrain extract links Default stays off. Existing brains see zero behavior change on upgrade. Closes #972. Adapts PR #1233 from @rayers (regex shape + slug-tail index) into a multi-match, opt-in form with FS-source coverage that the original PR explicitly skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: document opt-in global-basename wikilink resolution (#972) The #972 feature shipped with no user-facing docs — only CHANGELOG + CLAUDE.md. Anyone migrating an Obsidian/Notion vault with bare [[name]] wikilinks couldn't discover the link_resolution.global_basename flag unless gbrain doctor happened to surface its hint. - README "Self-wiring knowledge graph": one sentence on the opt-in mode for Obsidian-style cross-folder bare wikilinks + the doctor pre-check, linking to the install step. - INSTALL_FOR_AGENTS Step 4.5 (Wire the Knowledge Graph): a dedicated agent- facing subsection — when bare [[name]] links need it, the enable command, re-running extract, the doctor opportunity hint, and the multi-match behavior. - Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): resolve aliased wikilinks by target slug, not display text Codex outside-voice [P1]: `[[struktura|the project]]` resolved the basename "the project" (the alias) instead of `struktura` (the target), because extractPageLinks called resolveBasenameMatches(ref.name) and the doctor check keyed basenameIndex.get(e.name). ref.name is the display alias (match[2]); ref.slug is the wikilink target (match[1]). - extractPageLinks resolves ref.slug; context excerpt locates ref.slug. - doctor link_resolution_opportunity keys e.slug so its estimate matches what extraction actually resolves. - Test: aliased wikilink calls resolveBasenameMatches with the target, never the display text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): reconcile wikilink-resolved edges in put_page auto-link Codex outside-voice [P1]: put_page's reconcilableOut filter excluded link_source='wikilink-resolved', so a basename edge written by auto-link survived after the bare wikilink was deleted from the page OR the link_resolution.global_basename flag was turned off (the stale-removal loop only iterates reconcilableOut). Add 'wikilink-resolved' to the reconcilable set; manual edges still untouched. Test: write page with [[struktura]] (flag on) → edge lands; re-put without the wikilink → edge reconciled away. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): source-scope basename resolution (no cross-source edges) Codex outside-voice [P1]: makeResolver.resolveBasenameMatches called engine.getAllSlugs() unscoped, so a bare [[name]] could resolve to a same-tail page in a DIFFERENT source and create a cross-source edge. The engine exposes getAllSlugs({sourceId}) precisely to prevent this. #972 is "global basename across folders," not "cross-source federation" — the canonical gbrain multi-source bug class. - makeResolver gains opts.sourceId; ensureBasenameIndex passes it to getAllSlugs (unscoped only when sourceId omitted — back-compat). - runAutoLink (put_page) passes opts.sourceId; extractLinksFromDB passes sourceIdFilter. FS extract is already single-source (walks one dir). - Tests: scoped index returns only the source's slugs (no cross-source); unscoped call stays brain-wide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): FS-source basename edges carry link_source='wikilink-resolved' The FS extract path is the issue's default repro (gbrain extract links with no --source db). ExtractedLink had no link_source field, so FS basename edges landed with the engine default ('markdown') instead of the 'wikilink-resolved' provenance the DB / put_page paths set and the docs promise. The e2e FS test only asserted link_type, so it was blind to this. - ExtractedLink gains link_source?; extractLinksFromFile sets it to 'wikilink-resolved' on basename edges (undefined for ordinary markdown). - Carries through the addLinksBatch snapshots automatically (LinkBatchInput already has link_source); single-row addLink fallback now passes it too. - e2e FS repro asserts link_source === 'wikilink-resolved'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(#972): one shared basename matcher across resolver/FS/doctor Codex outside-voice [P2] DRY: three surfaces each hand-rolled a basename matcher with divergent key sets — the doctor omitted the slugified key, so its link_resolution_opportunity estimate undercounted what extraction resolves, and the resolver returned matches in unsorted getAllSlugs bucket order. New shared exports in link-extraction.ts: buildBasenameIndex(slugs) + queryBasenameIndex(index, name) (keys raw/lower/slugified tail; stable sort shorter-first then lexical) + normalizeBasename. - makeResolver.resolveBasenameMatches → queryBasenameIndex (now stable-sorted). - extract.ts resolveBasenameMatchesFromSlugs → delegates to the shared pair. - doctor link_resolution_opportunity → shared builder/query (slugified key added; estimate now matches extraction). - Test: doctor counts a slugified-only match ([[Fast Weigh]] → companies/fast-weigh). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): P2 cluster — masking, code-fence, self-link, dedup decision Codex outside-voice P2 findings: - P2a markdown-label masking: a wikilink inside a markdown-link label ([see [[acme]]](companies/acme.md)) spawned a stray generic basename ref. Pass-1 can't match the nested brackets, so a new MARKDOWN_LABEL_WIKILINK_RE masks those spans out of pass 2c. Inner [[acme]] is now inert. - P2b FS code-fence: the FS path (extractMarkdownLinks on raw content) didn't strip code blocks like the DB path. extractLinksFromFile now scans stripCodeBlocks(content) so [[name]] inside a fence creates no FS edge. - P2c self-link guard: a basename [[own-tail]] on its own page resolved back to itself. Dropped in both extractPageLinks and the FS path. - P2d dedup: documented the decision to KEEP qualified + bare edges to the same target as separate rows (distinct provenance/audit trail). - P2e: skipFrontmatter unresolved-contract tests added. Tests: P2a inert-label, P2c self-link drop, P2b code-fence, P2e unresolved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(#972): bound the doctor link_resolution_opportunity scan The check did listAllPageRefs() + a getPage() per page under a 60s budget. On a large brain (the eng-review concern) it hit the budget every non-fast doctor run and returned a perpetual partial, adding ~60s. Now batch-loads the 1000 most-recent pages in ONE query (ORDER BY id DESC LIMIT SAMPLE_LIMIT) and scans in memory, with the 60s cap kept as a backstop. Mirrors the v0.40.9 sampling convention. The estimate message names the bound when the brain exceeds the sample ("scanned the 1000 most-recent of N pages"). Test: source-grep pins the bounded query + the absence of the per-page getPage walk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(#972): reconcile stale version/migration references to v112 / 0.42.6.0 Merge churn left intermediate refs: schema.sql + schema-embedded.ts said "migration v93", CLAUDE.md said "v0.41.32.0 / Migration v109", CHANGELOG said "Migration v93". Reconciled all to migration v112 / shipping 0.42.6.0. The CLAUDE.md annotation is also refreshed to describe the final behavior (shared matcher, source-scoping, alias-by-target, stale-edge reconciliation, bounded doctor scan) and credit @rayers + @ukd1. Regenerated schema-embedded + llms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): register doctor check category + bump llms budget to 800KB Two full-suite gate failures from the re-sync: - doctor-categories drift guard: the new `link_resolution_opportunity` check wasn't in any category set. Added to BRAIN_CHECK_NAMES (alongside graph_coverage / orphan_ratio — it's a graph-quality signal). - build-llms size budget: the #972 Key Files annotation (landing with master's #1696/#1699 waves) pushed llms-full.txt past 750KB. Bumped FULL_SIZE_BUDGET 750KB→800KB, the established "budget tracks CLAUDE.md's legitimate per-feature growth" pattern (600→700→750→800 across releases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Garry Tan <garrytan@gmail.com> |