mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
8f45624e55fdb8b63560b905b2cfb697e897eb47
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
8f45624e55 |
v0.42.39.0 feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981) (#2019)
* fix(integrations): parameterize resolver-row fence by recipe id The install fence was hardcoded gbrain:agent-voice:resolver-rows, so any second copy-into-host-repo recipe wrote a mislabeled block (and refresh/ uninstall keyed on recipe id would miss it). Derive it from manifest.recipe. * feat(context): Retrieval Reflex — teach the agent when/what to retrieve (#1981) Deterministic per-turn pointer layer in the context engine: a zero-LLM, precision-biased scan resolves salient entities (names, @handles) to existing brain pages and injects compact pointers (name → slug → safe synopsis). Detect + point, never auto-dump. Fail-open, capped, suppression on prior context only. Engine-aware resolver ladder (no second DB connection): host ctx.brainQuery → PGLite serve resolve IPC (unix socket) → Postgres cached direct → disabled. Synopsis runs through get_page's privacy strip. Plus the retrieval-reflex recipe + policy skill, the retrieval_reflex_health doctor check, config gate, and the init next-step hint. * chore: bump version and changelog (v0.42.38.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): KEY_FILES entries for Retrieval Reflex surface (#1981) Document the new src/core/context/ modules, the context-engine resolver ladder, the serve resolve IPC, the retrieval_reflex_health doctor check, and the recipe-id-keyed install fence. Current-state only. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
03ffc6ebdb |
v0.42.37.0 fix(jobs): reap stale locks, bound disconnect, complete cooperative-abort (#1972) (#2015)
* fix(jobs): reap stale dead-holder cycle/sync locks (#1972) A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row until something contended for it — reclaim was on-contention only. Add a host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle* namespaces only (never elections/supervisor/reindex), with a snapshot-matched delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse. Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now back inspectLock + listStaleLocks (killed the triplication). * fix(db): bound pool disconnect so teardown can't eat CLI output (#1972) pool.end() against PgBouncer transaction-mode never drained, so disconnect blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write, truncating stdout (e.g. #1959's relational query returned empty). Add a gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against a hard timer, so teardown is bounded regardless of what postgres.js does and is testable. connection-manager ends its direct + read pools concurrently so the per-pool bounds don't stack. PGLite disconnect is unaffected. * fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972) v0.42.29 made only the embed phase honor the abort signal; a 24h pull still showed force-evicts from a long non-embed phase ignoring it. Thread the signal into every cycle-reachable long loop: extract (extractForSlugs + the full-walk extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop + embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket loop. Add a terminal abort check so an aborted cycle never stamps last_full_cycle_at as a completed run (Codex #9). lint now yields + checks abort every 200 pages (it's synchronous; the yield is what lets the signal land). New phase-duration force-evict attribution log names any phase that crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start. * chore: bump version and changelog (v0.42.38.0) #1972 — stale-lock reaper, bounded pool disconnect, and complete cooperative-abort coverage across cycle phases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY), db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/ extract_facts/consolidate/lint + terminal guard) entries to current truth. * test(isolation): fix shard-order flakes exposed by #1972's new test files Adding 3 new test files reshuffled the hash-based shards, exposing two pre-existing test-isolation bugs: - cycle-consolidate.test.ts assumed the global legacy-embedding preload's 1536-d gateway config still held at initSchema, but a co-sharded test that calls resetGateway() in teardown nulls it, so initSchema fell back to the 1280-d default and built a halfvec(1280) facts column its 1536-d fixtures can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern legacy-embedding-preload.ts documents for 1536-d fixture tests). - db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation rule R1. Convert to withEnv(). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
959af1068d |
v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794) (#1980)
* fix(retry): match EMAXCONNSESSION + SQLSTATE 53300 as retryable conn errors (#1794) * feat(schema): add op_checkpoint_paths append-only delta table (migration v115) (#1794) * refactor(op-checkpoint): append-only deltas via executeRawDirect + withRetry (#1794) * fix(sync): resumable-checkpoint durability + lock-thrash fix (#1794) Durable append-only checkpoint writes (executeRawDirect + retry), fail-loud consecutive-failure abort, first-file/10s flush cadence, race-safe pending-delta under parallel workers, guaranteed final flush on every exit path incl. SIGTERM (no-retry one-shot via registerCleanup), bankedFiles/reason observability, event-loop yield to keep the lock heartbeat alive, and routing the bare (no-source) sync through withRefreshingLock. * fix(db-lock): heartbeat-aware takeover + direct-pool refresh (#1794) * fix(cycle): treat SyncLockBusyError as skip, not a phase failure (#1794) * docs(sync): document the 5 checkpoint/lock env knobs (#1794) * v0.42.36.0 fix(sync): resumable, durable, single-flight sync — converges under pool exhaustion + repeated kills (#1794) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): update sync.ts + op-checkpoint.ts entries to resumable-checkpoint current state (#1794) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
612753f318 |
v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970) (#1975)
* v0.42.35.0 fix(sync): recover from unreachable last_commit instead of full-walking forever (#1970) When a source's history is rewritten (force-push, master→main consolidation, squash), the recorded last_commit can fall outside HEAD's history. The old guard sent both "object missing" and "not an ancestor" to performFullSync — a full repo re-walk that never advances the bookmark under a cron timeout on a large cross-region brain, so the source goes silently stale. Fix: only a truly-absent object forces a full reconcile. A present-but-non- ancestor bookmark is diffed tree-to-tree directly (git diff A..B needs no ancestry), importing only the real delta. Adds: oversized-diff fallback to full reconcile (F-B); performFullSync now purges deleted files, gated to file-backed pages by source_path so manual/put_page and metafile pages are spared (F-A); rename-to-unsyncable deletes the stale old page (F-C). 7 new PGLite e2e tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): record #1970 sync bookmark recovery + full-sync delete reconcile in KEY_FILES Update the sync.ts entry to current truth: entry-time bookmark-reachability guard (gc'd anchor → full reconcile; non-ancestor-but-present → direct tree-to-tree diff), oversized-diff fallback, performFullSync now authoritative for deletes (file-backed pages by source_path), and rename-to-unsyncable delete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
b31de6613e |
v0.42.33.0 fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (#1881) (#1960)
* fix(sources): confine sync re-clone to gbrain-owned clones; never delete a user working tree (#1881) recloneIfMissing deleted local_path whenever a source had a remote_url and a non-healthy on-disk state, with no check that gbrain actually created the clone. A source whose local_path was a user's live working tree (remote_url set, no gbrain-created clone) could have its directory removed and re-cloned over. - isOwnedClone(): ownership, not path-containment. True only for a config .managed_clone marker (written by addSource --url) or exact normalized-path equality with defaultCloneDir(id) (back-compat for pre-marker default clones). - recloneIfMissing: ownership guard aborts before ANY filesystem op; EXDEV-safe sibling-temp clone + atomic swap (old aside -> new in -> drop old) with best-effort restore + a message naming where the original is preserved; symlink-leaf reject before the destructive rename. - sync.ts validate_repo_state guards reclone on isOwnedClone (no per-sync warn). - sources restore degrades to a warning for an unowned source instead of the misleading "missing clone, try sync" hint. Tests: #1881 regression (tree survives), isOwnedClone matrix, symlink reject, EXDEV swap residue-free, --clone-dir owned-via-marker, restore CV3, unownedHint healthy/degraded, sync-level refusal. * chore: bump version and changelog (v0.42.33.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document sources-ops reclone-ownership invariant for v0.42.33.0 (#1881) Add the missing src/core/sources-ops.ts entry to KEY_FILES.md capturing the must-never-violate reclone-ownership guarantee: gbrain only deletes/re-clones a clone it created (isOwnedClone), never a user working tree. Covers managed_clone marker, defaultCloneDir back-compat, EXDEV-safe swap, TOCTOU + symlink-leaf guards, unmanaged_path SourceOpError, and the read-only sources restore path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
5a06af5a57 |
v0.42.32.0 fix(sync): coerce non-string frontmatter titles + bounded auto-skip failure ledger (#1939) (#1956)
* fix(import): coerce non-string frontmatter title/slug/type (#1939) YAML `title: 2024-06-01` parses to a Date and `title: 1458` to a number; the old `(frontmatter.X as string)` cast was a compile-time lie, so downstream `.toLowerCase()` threw and (via the importer failure gate) could wedge sync indefinitely. parseMarkdown now coerces via coerceFrontmatterString (Date -> UTC ISO date, deterministic), and the pure assessContentSanity self-protects against a non-string title. * feat(sync): bounded auto-skip failure ledger; poison file can't wedge indexing (#1939) New src/core/sync-failure-ledger.ts owns the failure store + a crash-safe, multi-source, concurrent bounded auto-skip valve. A file that fails N consecutive syncs (GBRAIN_SYNC_AUTOSKIP_AFTER, default 3) auto-skips so it can't freeze all indexing forever, while fresh failures still fail-closed and a `<head>` history-rewrite sentinel hard-blocks even with --skip-failed. - (source_id, path) keying — failures never merge across sources - success clears a path so attempts are truly consecutive - advance-before-ack ordering (a crash can't mark a file skipped while wedged) - shared applySyncFailureGate used by BOTH the incremental and full-sync gates - legacy-row normalization + duplicate collapse on load - cross-process lock + atomic temp-rename, age-based stale-lock break sync.ts re-exports the ledger for existing callers; import.ts records source-scoped and defers the bookmark to the gate under managedBookmark. * fix(doctor): sync_failures severity via one shared decision on both surfaces (#1939) Local buildChecks and remote doctorReportRemote now both route through decideSyncFailureSeverity, so a stuck bookmark escalates WARN -> FAIL consistently (oldest-open age > fail cadence, or large unresolved count), auto-skipped pages stay visible (WARN, not hidden), and the acknowledged/acknowledged_at field-split that caused drift is gone. The remote surface stays subprocess-free (file read + Date.parse only). * chore(test): add trailing newline to e5-lease-cap-ab baseline fixture * fix(sync): address adversarial review findings on the failure ledger (#1939) - #1: a parse-failed file that is later deleted/renamed-away no longer leaves a permanent open ledger row. Removed paths (filtered.deleted, renamed-from, and the "gone from disk" forward-delete skip branch) are treated as resolved so the ledger self-heals instead of aging doctor to a stuck FAIL. - #3: decideSyncFailureSeverity escalates to FAIL on OPEN (blocking) failures only — auto_skipped rows already advanced the bookmark, so they stay WARN-visible regardless of count, matching the state-machine contract. * chore: bump version and changelog (v0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document sync-failure ledger + auto-skip valve for v0.42.30.0 KEY_FILES.md: new src/core/sync-failure-ledger.ts entry (bounded auto-skip state machine, decideGateAction/decideSyncFailureSeverity/applySyncFailureGate, GBRAIN_SYNC_AUTOSKIP_AFTER); update sync.ts (failure store moved to ledger, re-exported), doctor.ts (sync_failures severity via shared rule on both surfaces), markdown.ts (coerceFrontmatterString), import.ts (managedBookmark). live-sync.md: poison-file auto-skip tricky-spot. Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.31.0 (queue collision on 0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.32.0 (queue collision) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f401d7407e |
v0.42.31.0 feat(links): open link_source provenance + link-add/link-rm/link-sources (#1941) (#1957)
* feat(links): relax link_source CHECK to kebab-case provenance + migration v114 Open link_source from a closed allowlist to a kebab-case format gate (^[a-z][a-z0-9]*(-[a-z0-9]+)*$, char_length<=64) so external derivers stamp their own provenance (e.g. citation-graph) without a per-deriver migration. Migration v114: Postgres NOT VALID + VALIDATE (lock-friendly, transaction:false); PGLite plain DROP+ADD. Updates the schema.sql + engine provenance contract comments. (#1941) * feat(links): expose link provenance on link ops + link-add/link-rm/link-sources add_link/remove_link now accept --link-source/--link-type; add_link guards the reconciliation-managed built-ins (markdown/frontmatter/mentions/ wikilink-resolved) and defaults omitted provenance to 'manual' (was the misleading engine default 'markdown'). New cliHints.aliases mechanism with a startup collision guard registers link-add/link-rm; printOpHelp shows the invoked alias name. New list_link_sources read op + listLinkSources engine method (both engines, {sourceId?,sourceIds?}, deterministic order) powers `gbrain link-sources`, added to the minion read allowlist. (#1941) * test(links): kebab provenance, op guard, link-sources, aliases + parity Covers the v114 regex/length boundaries, upgrade-path constraint swap on existing data, the managed-built-in op guard + manual default, remove_link type/source filters, list_link_sources scoping (scalar + federated) and PG/PGLite parity, and alias resolution/collision/help. Fixes the prior 'inferred'-rejection assertion (now valid kebab) in the mentions test. (#1941) * chore: bump version and changelog (v0.42.31.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update KEY_FILES for v0.42.31.0 link provenance surface KEY_FILES.md current-state updates for #1941: link_source now an open kebab-case provenance (migration v114), the add_link/remove_link guard + defaults, list_link_sources + listLinkSources, and cliHints.aliases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minions): supervisor queue-singleton keying + pidfile cleanup (follow-up #1849) Two correctness bugs in the v0.42.29.0 supervisor-singleton work, caught by adversarial review: - supervisorLockId mixed a config-derived DB identity into the key, but the lock row already lives inside the target database. Two supervisors on the same physical DB via different-but-equivalent URLs (pooler vs direct port, host alias, trailing params) computed different ids and BOTH acquired the "singleton" lock. Key on the queue alone; the database half of the mutex is physical. Removes the now-dead currentDbIdentity() from worker-registry. - The pidfile-cleanup process.on('exit') listener was installed AFTER the DB-lock acquire, so the LOCK_HELD early-exit stranded the pidfile this process had just created. Install the listener first. Regression test pins the listener-before-lock ordering; updates the lockId test to the queue-only invariant; KEY_FILES updated to current state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
613da94093 |
v0.42.29.0 fix(minions): long-job abort-honoring + attempt accounting + supervisor singleton; topic-aware voice (#1737, #1849, #1851) (#1943)
* fix(minions): honest attempt accounting + cooperative abort-honoring + per-handler timeouts (#1737) - Wall-clock and stall dead-letter paths now increment attempts_made (terminal, no retry — wall-clock fires at 2x cumulative timeout; retrying non-idempotent embed/subagent work would duplicate side effects). Surface stalled_counter in jobs get so 'started 3 / stalled 2 / attempts 0' reads true instead of looking like broken accounting. - Thread AbortSignal through embed-backfill/autopilot-cycle -> runPhaseEmbed -> runEmbedCore -> embedAll(Stale)/embedPage, checking it on BOTH --stale and --all paths and between embed batches. A timed-out embed phase now bails within a batch, so the cycle finally releases gbrain_cycle_locks instead of running the full 10-15 min after the job was killed (the daily cycle-wedge). New shared src/core/abort-check.ts (isAborted/throwIfAborted/anySignal). - Per-handler default wall-clock budget (handler-timeouts.ts) stamped at submit for long handlers without an explicit timeout_ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minions): queue-scoped DB supervisor singleton + canonical pidfile + doctor max-rss check (#1849) - Acquire a queue-scoped DB lock (tryAcquireDbLock, keyed on the raw DB identity + queue) on supervisor.start(): a second supervisor on the same (db, queue) fails fast with exit 2 regardless of $HOME/--pid-file. Refresh on a dedicated timer; on refresh failure past the threshold, fail SAFE (exit non-zero) before the TTL could lapse and let a second supervisor take over. Release on shutdown. - Canonical default pidfile keyed on brain id (currentBrainId, config-only, no DB connect) so two brains under one HOME no longer share supervisor.pid. - doctor: new supervisor_singleton check surfaces the effective --max-rss (from the started audit event) and warns when the lock holder's (host,pid) differs from the local pidfile — comparing host+pid, not bare pid. Registered in doctor-categories. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(agent-voice): topic-aware persona context via server-resolved topicId (#1851) Summon Mars/Venus into a specific conversation topic so they boot already knowing the recent thread. A per-topic call link carries only topicId (+ optional display topicName); the server resolves the recent-conversation context from the brain (topics/<topicId>.md) — topic CONTENT is never accepted over the wire (that would be prompt injection + a leak into URLs/referrers/logs). topicId is a strict slug with a path-traversal guard. New '# Topic Context' prompt slot injected after the persona body so identity-first ordering still wins; persona identity unchanged. No topicId -> generic behavior. Contract doc + persona skill docs updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: file #1737 slot-reservation fair-scheduling follow-up TODO (F7) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.29.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync KEY_FILES.md for v0.42.29.0 minions + abort wave (#1737, #1849) Fold the #1737/#1849 behavior into the existing per-file entries and add the two new core files, keeping the doc at current-state truth: - queue.ts: honest attempt accounting on wall-clock + stall dead-letter paths; defaultTimeoutMsFor stamping at submit. - supervisor.ts: queue-scoped DB singleton lock (supervisorLockId, classifySupervisorSingleton, LOCK_LOST, refresh-fail-safe, brain-id pidfile, max_rss_mb audit). - worker-registry.ts: currentDbIdentity(). - New entries: src/core/abort-check.ts, src/core/minions/handler-timeouts.ts. - New doctor.ts extension: supervisor_singleton check. - cycle.ts / embed.ts extensions: AbortSignal threading note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f7f8512b14 |
v0.42.28.0 fix(engine): batch inserts use jsonb_to_recordset, not text[] array literals (#1861) (#1927)
* fix(engine): batch inserts use jsonb_to_recordset, not text[] array literals (#1861) addLinksBatch/addTimelineEntriesBatch/addTakesBatch passed free text through unnest(${arr}::text[]); postgres.js serialized it to a Postgres text[] literal that array_in rejected ("malformed array literal") on calendar/Zoom context, aborting the whole `extract links --stale` sweep. Bind the batch as one JSONB doc via jsonb_to_recordset(($1::jsonb)->'rows') through the audited executeRawJsonb contract instead. Shared row builders (src/core/batch-rows.ts) keep both engines byte-identical; NUL is stripped only from free-text body fields (context/summary/detail/claim), while identity/security fields (slugs/source_ids/holder/kind/dates) still reject NUL. addTakesBatch is now batchRetry-wrapped ('addTakesBatch' audit site) and its BrainEngine signature takes BatchOpts. Scalar addLink context is NUL-stripped too. Regression tests on both engines: PGLite always-on poison/NUL/parity suite + DATABASE_URL-gated Postgres lane (the engine that actually crashed). * test: make "no Anthropic key" tests hermetic via withoutAnthropicKey hasAnthropicKey() reads both ANTHROPIC_API_KEY and ~/.gbrain config; tests that only deleted the env var fired a real LLM call on configured machines (warning flipped NO_ANTHROPIC_API_KEY -> LLM_OUTPUT_NOT_JSON). New test/helpers/no-anthropic-key.ts neutralizes both sources (env + GBRAIN_HOME temp dir) for the duration of the call. Refactors the five no-key tests in think-pipeline + takes-mcp-allowlist to use it, including two that previously passed only by luck of the live LLM output. * chore: docs + version bump (v0.42.28.0) KEY_FILES.md/RETRIEVAL.md describe the jsonb_to_recordset batch path; TODOS.md files the #1861 follow-ups (element-isolation, remaining ::text[] sites, shared SQL-string hoist, batch-insert edge-case tests). CHANGELOG + VERSION + package.json to 0.42.28.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync TESTING.md batch-insert references for v0.42.28.0 The #1861 fix migrated links/timeline/takes batch inserts from unnest(::text[]) to jsonb_to_recordset. Update the stale "postgres-js unnest() binding" note and add the two new poison-regression test files (test/links-timeline-jsonb-poison.test.ts PGLite half, test/e2e/jsonb-batch-poison-postgres.test.ts Postgres lane) to the inventory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sql-query): reject top-level array jsonb params in executeRawJsonb (#1861 P2a) The "no top-level array" rule was only a comment. A bare JS array bound to a $N::jsonb position can serialize as a Postgres array literal (not jsonb) through postgres.js, silently re-entering the "malformed array literal" class #1861 just escaped. executeRawJsonb now throws a clear error steering callers to the { rows: [...] } object wrapper. Verified breaks zero call sites (all pass objects or null). Codex adversarial P2a; batch-size enforcement (P2b) filed as a TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9a0bae8d62 |
v0.42.25.0 fix(pricing): unify chat-model pricing into one canonical source; add Opus 4.8 (#1819) (#1827)
* fix(pricing): unify chat-model pricing into one canonical source; add Opus 4.8 (#1819) Single canonical CANONICAL_PRICING table (src/core/model-pricing.ts) with canonicalLookup; ANTHROPIC_PRICING and takes-quality MODEL_PRICING become derived views. cost-tracker, cross-modal runner, skillopt preflight, brainstorm orchestrator, and brain-score all source from it. Adds Opus 4.8 ($5/$25) so --max-cost-usd and the dream-cycle budget meter enforce on 4.8 runs; fixes a stale Opus 4.7 $15/$75 in the takes-quality gate and reconciles Gemini 2.0 Flash to $0.10/$0.40. Because every table derives from canonical, cross-table price drift is structurally impossible. * chore: bump version and changelog (v0.42.25.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document canonical chat-pricing table for v0.42.25.0 Add KEY_FILES.md entries for src/core/model-pricing.ts (canonical CANONICAL_PRICING + canonicalLookup) and refresh the now-derived anthropic-pricing.ts + takes-quality-eval/pricing.ts entries to current-state. Add the "one canonical chat-pricing table" cross-cutting invariant to CLAUDE.md. Fix the stale model-price snapshot pointer in SEARCH_MODE_METHODOLOGY.md (anthropic-pricing.ts -> model-pricing.ts). Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f868257405 |
v0.42.24.0 fix(minions): route lock claim/renewLock through direct session pool (#1822)
* fix(minions): route lock claim/renewLock through direct session pool The Minion lock heartbeat (claim + renewLock) ran every UPDATE through engine.executeRaw(), which is hardcoded to the read pool. On Supabase that is the transaction-mode pooler (6543), which recycles connections per transaction. A lock is held for minutes, so the pooler periodically reaps the socket mid-heartbeat -> CONNECTION_ENDED -> the lock looks expired -> the worker force-evicts its own job and the claim loop wedges silently. Add BrainEngine.executeRawDirect(): same contract as executeRaw, but routes to the direct session-mode pool (5432, GBRAIN_DIRECT_DATABASE_URL) when dual-pool is active. No-op delegation on PGLite / non-Supabase / kill-switch. claim/renewLock now use it. Single-statement UPDATEs only, so the double-claim guard and the renewLock no-inline-retry contract are preserved. Statements inside an open transaction keep their tx connection (in-transaction guard keys on peekReadPool() !== _sql); the lock hot-path never runs inside transaction(). The Postgres impl shares its cancellation plumbing with executeRaw via a private runUnsafe helper. New test/postgres-execute-raw-direct.test.ts covers the routing decision (dual-pool on/off x in-tx/not + abort short-circuit) without a live Postgres; queue-lock-retry.test.ts gains a guard that claim can never fall back to executeRaw. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.24.0 chore: bump version and changelog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.42.24.0 Document executeRawDirect on the BrainEngine contract and the claim/renewLock direct-session-pool routing in KEY_FILES.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make D3 executeRaw-no-retry guard refactor-aware The DRY refactor in this PR extracted executeRaw/executeRawDirect's shared cancellation plumbing into a private runUnsafe(conn, ...) helper, so the single conn.unsafe() call moved out of executeRaw's body. The D3 guard read executeRaw's source and asserted conn.unsafe( appeared exactly once there, which now fails (it's zero — executeRaw delegates). The D3 invariant (no per-call retry wrapper) is unchanged; it just spans the delegate now. Update the guard to check both public methods delegate to runUnsafe without reconnect/retry, and assert the exactly-once conn.unsafe + cancel-only catch in runUnsafe. Also extends coverage to executeRawDirect so the lock hot-path can't reintroduce a retry wrapper either. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f11d56cfca |
v0.42.23.0 feat(jobs): --nice scheduling-priority flag for jobs work/supervisor (#1815) (#1820)
* feat(jobs): niceness core, worker registry, shared supervisor-pid reader OS scheduling-priority primitives for issue #1815: - niceness.ts: parseNiceValue (whole-string), applyNiceness (re-reads effective in success AND failure paths), getEffectiveNiceness, formatNice. - worker-registry.ts: live workers self-register pid + requested/effective nice under gbrainPath('workers'); readWorkers prunes ESRCH (keeps EPERM) with a pid-reuse start-time guard. - supervisor-pid.ts: readSupervisorPid extracted from the copy-pasted PID-file + liveness block. * feat(jobs): --nice flag for jobs work/supervisor + doctor niceness check Wires the --nice <n> flag (and GBRAIN_NICE env) through the CLI (issue #1815): - jobs work: applies niceness + registers the worker; cleanup on finally and process.on('exit'). - jobs supervisor: applies in the foreground-start path only (after the --detach fork), passes the apply result into MinionSupervisor. - supervisor.ts: nice opts, extracted testable buildWorkerArgs (appends --nice), emits niceness on started/worker_spawned audit events. - jobs stats / supervisor status: surface effective worker + supervisor nice. - doctor: separate supervisor_niceness check (warns on requested != effective) so it can't clobber the supervisor crash-check precedence; registered in doctor-categories. * test(jobs): cover niceness, worker registry, supervisor-pid, build args Unit tests for issue #1815: parseNiceValue rejects 3.5/10abc that parseInt would accept; applyNiceness re-reads effective on EPERM; registry ESRCH/EPERM + pid-reuse guard + brain-isolated path; readSupervisorPid states; parseNiceFlag flag>env precedence; buildWorkerArgs --nice propagation. * chore: bump version and changelog (v0.42.23.0) --nice flag for jobs work/supervisor (issue #1815). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document --nice flag for jobs work/supervisor (v0.42.23.0) - minions-deployment.md: niceness tuning section (full concurrency, low priority). - KEY_FILES.md: entries for niceness.ts, worker-registry.ts, supervisor-pid.ts; supervisor.ts entry notes buildWorkerArgs + nice opts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): add enrich_thin to dream cycle EXPECTED_PHASES The enrich_thin cycle phase (src/core/cycle.ts ALL_PHASES, between conversation_facts_backfill and skillopt) shipped without updating the e2e phase-order expectation, so dream-cycle-phase-order-pglite failed on master. Sync the expected list to the real ALL_PHASES order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): align sync-lock-recovery with the shipped --break-lock --all contract v0.41.13.0 intentionally dropped the "--break-lock + --all is refused" guard so cron can self-heal every source in one call (sync.ts runBreakLock iterates sources under --all). The e2e test still asserted the old exit-1 refusal and failed on master. Assert the current contract: the combination is accepted and takes the iterate / no-active-sources path (exit 0, no refusal message). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): de-flake ingestion-roundtrip chokidar first-drop race The native fsevents watcher occasionally missed a freshly written file, timing out the 15s waitFor (~1/3 on master under load). Three fixes: - inject a polling chokidar watcher via the source's _watchFactory seam (usePolling, 20ms interval) so detection never depends on fsevents timing; - drop deterministic fixtures BEFORE start so the initial scan (ignoreInitial:false) emits them, keeping live-watch coverage only where it's robust; - poll for the dedup hit instead of a fixed 600ms sleep. 15/15 green under stress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): make hermetic-PGLite serve tests actually hermetic connect-bearer and serve-stdio-roundtrip init a PGLite brain and spawn serve, but passed {...process.env} through — leaking an ambient DATABASE_URL / GBRAIN_DATABASE_URL into the subprocess, which then came up on Postgres and failed the `engine: pglite` assertion. Strip both DB vars from the spawned env so the tests are deterministic whether or not the shell/CI has a DB URL set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): type the hermetic-PGLite env so tsc passes The DATABASE_URL/GBRAIN_DATABASE_URL strip used `delete` on a narrowly-typed env literal (tsc-only failure; bun test doesn't typecheck). Annotate connect-bearer's env as Record<string,string|undefined> and build serve-stdio's as a concrete Record<string,string> (StdioClientTransport.env rejects undefined). Runtime behavior unchanged (7/7 + 3/3 green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: quarantine worker-registry to *.serial (R1 env-mutation isolation) worker-registry.test.ts sets process.env.GBRAIN_HOME per-test so gbrainPath resolves to a temp dir, then lazy-imports the module — a process-global mutation the parallel isolation lint (rule R1) forbids. Rename to worker-registry.serial.test.ts: it runs in the serial pass (own bun process, max-concurrency=1) where env mutation is safe, and the lint skips *.serial files. No logic change (6/6 green); fixes the failing `verify` CI job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f4959348c2 |
v0.42.22.0 fix(minions): supervisor progress watchdog + worker DB self-defense — alive-but-wedged worker self-heals (#1801) (#1824)
* fix(minions): supervisor progress watchdog + worker DB self-defense under supervision (#1801) Alive-but-wedged worker (dead DB pool, process still up) now self-heals in minutes instead of a silent 15h halt. - supervisor: progress watchdog restarts a child that makes no forward progress on claimable work (name+queue-scoped, active_healthy/due-delayed aware, startup-grace + loop-budget bounded); runtime handler-name derivation. - child-worker-supervisor: killChild gates on liveness not .killed (also fixes the existing shutdown SIGKILL no-op); restartCurrentChild kills the captured child ref; intentional restart doesn't count toward max_crashes. - worker: DB-liveness probe runs under supervision (db_dead self-exit), stall detection stays supervised-off. - doctor: standalone per-queue wedged_queue check + state->status fix in the remote queue_health check. - jobs/queue: queue-scoped getStats wedge fields + jobs stats WEDGED line. * fix(minions): wedge_restart_loop one-shot + supervised-probe comment + jobs-stats threshold (review) Pre-landing adversarial review findings: - wedge_restart_loop warn now fires once per exhausted window via a re-arming flag, not every health tick (was flooding the audit log for the full window). - Correct the stale GBRAIN_SUPERVISED comment: the DB probe runs under supervision now; only stall detection is skipped. - jobs stats WEDGED line reads GBRAIN_WEDGED_QUEUE_WARN_MINUTES so it agrees with the doctor wedged_queue threshold. * chore: bump version and changelog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: queue-ops runbook + KEY_FILES for the #1801 wedge watchdog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f3ade6c0c3 |
v0.42.21.0 fix(postgres): module-singleton ownership — canonical landing for the dream-cycle "connect() has not been called" class (#1404/#1471/#1619) (#1805)
* fix(postgres): module-singleton ownership — borrower disconnect no longer nulls the cycle's connection (#1404/#1471/#1619) gbrain dream on Postgres failed every DB phase with "No database connection: connect() has not been called": a short-lived borrower probe engine (lint/doctor config-lift, no poolSize) called db.disconnect() in its own disconnect(), nulling the shared module singleton the long-lived cycle owner was still using. The module `sql` is only ever nulled by db.disconnect() (postgres.js auto-reconnects its own pool), so the failure was always a borrower-disconnect, never an idle-pooler drop. Fix: db.connect() returns whether THIS call created the singleton (atomic — no await between the null-check and the sql=postgres() assignment), PostgresEngine stores it as _ownsModuleSingleton, and disconnect() only calls db.disconnect() when it owns the connection. Borrowers no-op. Hardening: db.disconnect() snapshots +nulls sql before awaiting end(); reconnect() shares one in-flight _reconnectPromise. Tests: new postgres-engine-singleton-ownership.test.ts; expanded DB-gated e2e matrix (owner/borrower, creation-not-role, symmetric CLI-exit, owner-reconnect- with-live-borrower); module-style getter asymmetry; #1570 shared-recovery regression updated to assert the fixed contract. Co-Authored-By: nullhex-io <noreply@github.com> Co-Authored-By: joelwp <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.15.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: nullhex-io <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
ec5fed2921 |
v0.42.20.0 fix: reliability wave — PGLite capture lock-pin + Postgres reconnect race + search embed-hang (#1762 #1745 #1775) (#1810)
* fix(core): drain fire-and-forget sinks before disconnect via a background-work registry (#1762) New src/core/background-work.ts registry (Map<name,drainer>, ordered drain, awaited abort). facts-queue (order 0, abort=shutdown), last-retrieved (1), and eval-capture (3, now self-tracked) register as sinks. Both CLI exit paths (op-dispatch finally + handleCliOnly finally) drain the registry before engine.disconnect() so a PGLite db.close() can't race in-flight work into the re-pump busy-loop that pinned the single-writer lock. Op-dispatch error path converts process.exit(1) to exitCode+return so the finally still drains. * fix(ai): bound every outbound AI call so a stalled provider can't hang (#1762/#1775) withDefaultTimeout composes a per-touchpoint default deadline (chat 300s, embed/multimodal 60s) with any caller signal via AbortSignal.any. Applied at the SDK call layer (chat generateText, expand generateObject, OCR, per-sub-batch embed) — covers native-anthropic + retries — plus per-request multimodal fetch. embedQuery forwards abortSignal. Env: GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS. * fix(postgres): module-mode reconnect preserves the shared singleton (#1745) reconnect() branches on connection style. Module-singleton engines re-establish idempotently via db.connect() (no-op when alive) + refresh the ConnectionManager read pool, never db.disconnect() — so a transient blip no longer nulls the shared sql out from under concurrent ops (which threw 'connect() has not been called'). Fail-loud on real connect failure. Instance pools keep teardown+recreate. * fix(search): bound the query-time embed so a stall falls back to keyword (#1775) search/query default to cheap-hybrid (embeds the query); a stalled provider made the embed never settle, so the keyword fallback never engaged and the command force-exited with no output. One shared QueryEmbedDeadline (6s, floored 2s per embed) covers both the cache-lookup and inner embeds via embedQueryBounded (abortSignal + Promise.race) → existing keyword fallback engages. Also registers the search-cache background-work drainer (now bounded). Env: GBRAIN_QUERY_EMBED_TIMEOUT_MS. * test+chore: reliability wave tests + v0.42.11.0 (#1762 #1745 #1775) New: background-work registry unit, query-embed deadline unit, eval-capture drain unit, postgres reconnect E2E (#1745), gbrain capture exit-clean case in the PGLite serial test. Updated fix-wave-structural assertions to the registry shape. VERSION/package.json/CHANGELOG -> 0.42.11.0; TODOS retrofit marked done. Incorporates + hardens PR #1763 (drain-before-disconnect + embed fetch timeout); the residual hung-Haiku hole is closed by the facts shutdown() abort belt. Co-Authored-By: ElliotDrel <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document background-work registry + v0.42.11.0 reliability wave in CLAUDE.md (regen llms) * chore: bump release version 0.42.11.0 -> 0.42.20.0 Rename the reliability-wave release version per request. Trio (VERSION / package.json / CHANGELOG) reconciled; in-code version-tag comments and test fixtures updated; llms regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ElliotDrel <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bde11bb18f |
v0.42.18.0 fix: sync orphan-pileup watchdog (#1633) + links-lag µs stamp (#1768) (#1807)
* fix(extract): links_extraction_lag never clears on Postgres (#1768) Stamp the full-microsecond updated_at (via to_char ... AT TIME ZONE UTC) instead of the millisecond-truncated JS Date, so links_extracted_at equals the DB updated_at exactly and the staleness predicate clears. Stamp SQL unchanged: version-arm backdating still works, D4 preserved, CDX-1 strengthened. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(core): out-of-band hard-deadline watchdog primitive (#1633) Bun eval-Worker that SIGTERM->grace->SIGKILLs its own process from a separate OS thread, so a sync whose main event loop is starved (ReDoS spin) still dies. Signals SELF (no PID-reuse footgun). Empirically validated on Bun 1.3.13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): arm hard-deadline watchdog + graceful SIGINT cancel (#1633) cli.ts installs the watchdog before connectEngine (bounds connect hangs); resolveSyncHardDeadline + composeAbortSignals in sync.ts; SIGINT graceful cancel on single-source + --all; withRefreshingLock timer unref'd. Non-TTY default 3600s makes cron orphan-pileup structurally impossible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(claude): annotate process-watchdog + #1768/#1633 fixes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: rebump v0.42.13.0 -> v0.42.18.0 (queue collision) Sibling workspaces claimed v0.42.13-v0.42.17; advance this branch's slot. VERSION + package.json + CHANGELOG header + CLAUDE.md annotations + llms bundles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(key-files): current-state phrasing for #1633/#1768 entries (fix check:doc-history) The doc-history guard bans the bolded **v0.X release-clause marker in reference docs (history belongs in CHANGELOG + git). Rewrote the extract.ts/sync.ts additions as current-state prose and de-versioned the process-watchdog entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
fd2fde9d26 |
v0.42.17.0 fix(sync): resumable incremental sync — killed mid-import no longer loses progress (#1794) (#1808)
* feat(sync): checkpoint primitives for resumable sync (#1794) - op-checkpoint.ts: syncFingerprint({sourceId, lastCommit}) keyed on the anchor (never HEAD) so the checkpoint survives a growing backlog. - source-health.ts: commitTimeMs(localPath, sha) for stamping newest_content_at against a pinned (non-HEAD) commit. - sync-concurrency.ts: resolveMaxConnections + clampWorkersForConnectionBudget for the opt-in GBRAIN_MAX_CONNECTIONS single-sync footprint clamp. * feat(sync): resumable incremental sync via pinned-target checkpoint (#1794) performSyncInner now drains a fixed lastCommit..pin range, banking completed file paths to op_checkpoints and advancing last_commit (+ last_sync_at) ONLY at full import completion. A killed/aborted/blocked run leaves the anchor untouched and resumes from the banked set next run — the convergence fix. - Pinned target: completion advances to the pin, not live HEAD, so commits landing after the pin are a clean next-sync diff (kills the staleness window). History rewrite (pin not an ancestor of HEAD) discards the checkpoint + re-pins. - Forward-progress head gate: merge-base --is-ancestor pin HEAD replaces the strict "HEAD == captured" gate that blocked on every concurrent enrich commit. - Vanished-on-disk added file -> skip + checkpoint, not a failedFiles block. - Large syncs defer extract/embed to the resumable --stale sweeps (convergence == import convergence); small syncs keep inline extract/facts/embed. - GBRAIN_MAX_CONNECTIONS clamp on the worker fan-out (opt-in). - Typed SyncLockBusyError; the Minion sync handler (jobs.ts) marks the job SKIPPED (not failed) on a held lock so cron/autopilot defers cleanly. * feat(doctor): pool_budget check for GBRAIN_MAX_CONNECTIONS (#1794) computePoolBudgetCheck + checkPoolBudget warn when the parent pool leaves no room for a parallel sync worker under GBRAIN_MAX_CONNECTIONS, pointing at GBRAIN_POOL_SIZE=2. Registered in the ops category set. * test(sync): resumable-sync regression suite + vanished-file contract (#1794) - sync-resumable-import.serial.test.ts (13 cases): convergence regression, resume-skips-checkpointed, pinned-target/forward-drift, history-rewrite re-pin, last_sync_at-not-bumped-on-block + good-file banking, vanished-file skip, dry-run/empty-diff, + pure fingerprint/clamp/pool-budget helpers. - sync-parallel.test.ts: vanished-mid-sync added file now asserts the new skip contract (supersedes the v0.22.13 CODEX-3 failedFiles behavior). * chore: bump version and changelog (v0.42.17.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
3fe449361c |
v0.42.16.0 feat(doctor): brain health as a solved problem — cause-ranked doctor + OOM-loop line + auto-drain + pool-reap (#1685) (#1802)
* feat(minions): pool-recovery audit + reconnect reason-threading + shared drain helper (#1685 GAP B, 5A) - pool-recovery-audit.ts: reap_detected (CONNECTION_ENDED) vs reconnect_other; recovered/failed split - postgres-engine reconnect(ctx?) classifies the triggering error so only true pooler reaps are tagged (CODEX #8) - retry.ts reconnect callback widened to thread the error; retry-matcher isConnectionEndedError - runExtractAtomsDrainForSource shared helper (cycleLockIdFor + withRefreshingLock) — one drain path (5A) - supervisor-audit readRecentSupervisorEvents (current+prev ISO week, CODEX #7) - extract-atoms-drain PROTECTED; autopilot.auto_drain.* config keys * feat(doctor): worker_oom_loop + pool_reap_health checks + cause-ranked top_issues (#1685 GAP A/B/C) - computeWorkerOomLoopCheck: unions supervisor rss_watchdog + minion_jobs watchdog-abort (CODEX #5), cap fallback to resolveDefaultMaxRssMb (CODEX #6) - computePoolReapHealthCheck: reaps-not-recovering fail, thrash warn - doctor-cause-rank rankIssues: tier ordering + grounded downstream_of (CODEX #9) + drift guard (4A) - supervisor causeStr + queue_health cross-reference worker_oom_loop (DRY 1C) - register both checks in doctor-categories ops * feat(autopilot): per-source extract_atoms auto-drain + handler + dream --drain refactor (#1685 GAP D) - autopilot per-source gate: enabled + !packDeclares + backlog>threshold + daily cap; time-sloted idempotency key (CODEX #2) - extract-atoms-drain Minion handler (thin wrapper, LockUnavailableError -> deferred) - dream --drain routes through the shared helper (5A) * chore: bump version and changelog (v0.42.12.0) #1685 brain-health-as-solved-problem: cause-ranked doctor, worker_oom_loop line, per-source auto-drain, pool-reap health. Layers on #1678/#1735. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(todos): file #1685 GAP E + remote-path follow-ups (v0.42.12.0) * fix(#1685): pre-landing review — multi-source auto-drain, honest pool-reap signal, lock-renewal reap labeling - autopilot: drop maxWaiting (coalesces by name+queue not source → only one source drained + cap over-count); pre-check idempotency key so only genuinely-new sources submit+count - pool_reap_health: fail on reconnect FAILURES (the real signal), not reaps>0&&failures>0 (false causality when a recovered reap + unrelated failure co-occur) - lock-renewal-tick threads its triggering error to reconnect() so a CONNECTION_ENDED pooler reap is labeled reap_detected not reconnect_other (pool_reap_health now fires for the #1678 incident path) * chore: re-version v0.42.12.0 → v0.42.16.0 (#1685) Slot collision avoidance per queue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: restore slim CLAUDE.md + move #1685 entries to KEY_FILES.md (fix check:doc-history) The master merge wrongly kept the pre-restructure 577KB CLAUDE.md; the check:doc-history guard caps it at 60KB. Take master's slim CLAUDE.md and record the #1685 files (doctor-cause-rank, pool-recovery-audit, worker_oom_loop + pool_reap_health checks, auto-drain, 5A helper) as current-state prose in docs/architecture/KEY_FILES.md (no release markers). llms regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
488f89e0dc |
v0.42.15.0 fix: decouple CLI primary output from process.stdout.isTTY (#1784) (#1806)
* feat(eval): cycle-default — single source of truth for TTY/non-TTY cycle count (#1784) * fix(jobs): decouple 'jobs watch' format (--json) from loop (--follow); non-TTY prints one human snapshot (#1784) * fix(reindex): human cost-refusal unless --json; spend guardrail unchanged (#1784) * fix(eval): annotate non-interactive cycle/budget defaults; runner core TTY-agnostic (#1784) * chore: bump version and changelog (v0.42.15.0) Decouple CLI primary output from process.stdout.isTTY (#1784): human by default, JSON only with --json; jobs watch non-TTY one-shots; eval banners annotate non-interactive defaults; reindex-code refusal is human unless --json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: CLAUDE.md Key Files notes for v0.42.15.0 isTTY-output decoupling (#1784) Annotate jobs.ts (jobs watch format/loop split + resolveWatchMode + the new cycle-default.ts), reindex-code.ts (human cost-refusal), and eval-cross-modal.ts (cycle/budget banner annotations). Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
1036f8f752 |
v0.42.14.0 fix(zero-config): code-* readiness signal + init embedding-key validation + lock self-heal (#1780) (#1804)
* feat(code-intel): readiness signal on code-def/refs/callers/callees (#1780 Gap 1) New src/core/code-graph-readiness.ts: resolveCodeReadiness() returns a typed status (not_built | indexing | ready | unknown) + ready boolean so callers can tell "graph not built / still indexing" apart from "genuinely no match" when count===0. EXISTS-based (cheap), chunk-grain, resolver-version-matching pending predicate, fail-open. Wired into the 4 CLI envelopes (+ human hint) and the 4 MCP op handlers. def/refs are 2-state brain-wide; callers/callees 3-state scoped. * feat(db-lock): automatic same-host dead-pid cycle-lock takeover (#1780 Gap 3) tryAcquireDbLock now reclaims a held, not-TTL-expired lock when the same-host holder is provably dead (process.kill ESRCH) past a 60s grace, via guarded DELETE + one normal-upsert retry returning the normal handle. New shared injectable classifyHolderLiveness/isHolderDeadLocally (EPERM treated as ALIVE — never steals a live lock). runBreakLock's safe path consumes the shared predicate, fixing its prior EPERM-as-dead bug. Cross-host stays TTL-only. * feat(init): validate the embedding key at gbrain init (#1780 Gap 2) New src/core/init-embed-check.ts: config-only diagnoseEmbedding (missing key, all providers) + best-effort 1-token live test-embed (invalid/expired key, 5s timeout, never blocks). Loud warning to stderr, init still exits 0; skipped by --no-embedding / --skip-embed-check / GBRAIN_INIT_SKIP_EMBED_CHECK=1. Builds the effective env (process.env + file-plane keys + --key) via buildGatewayConfig, extracted to src/core/ai/build-gateway-config.ts (cli.ts re-exports) so the check sees the same keys + provider base URLs as runtime. embedding_check added to --json. * chore: bump version and changelog (v0.42.14.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document the #1780 zero-config gaps for v0.42.14.0 CLAUDE.md Key Files: add src/core/code-graph-readiness.ts, init-embed-check.ts, ai/build-gateway-config.ts, and the db-lock auto-takeover + code-* readiness field behaviors. Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bea2d3e6c9 |
v0.42.13.0 fix(search): archive/ content findable by default, demoted not hard-excluded (#1777) (#1797)
* fix(search): archive/ findable by default — demote not hard-exclude (#1777) Move archive/ out of DEFAULT_HARD_EXCLUDES into a 0.5 source-boost demote so archived historical content is findable by default, ranked below curated content. Add a hidden_by_search_policy doctor check so the surviving exclude policy (test/, attachments/, .raw/) is auditable. Bump KNOBS_HASH_VERSION 8->9 so the policy change invalidates archive-excluded query_cache rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct search-exclude.test.ts annotation for archive demote (#1777) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
d4211f4176 |
v0.42.11.0 feat(skillopt): held-out eval gate, honest receipts, ENFORCE + ablation opts (#1759)
* feat(skillopt): wire held-out gate, honest receipts, ENFORCE + ablation opts Wire the F11 held-out gate into the orchestrator at checkpoint acceptance (runHeldOutGate was dead code); parse + thread --held-out through CLI, batch, fleet, background job, and the run_skillopt MCP op. Populate the real receipt.baseline_sel_score (was hardcoded 0) and add a final-test eval (test_score + baseline_test_score) via a shared scoreSkillOnTasks primitive. Fix the --no-mutate proposed.md write (was a stub) and enforce maxRuntimeMin. D16 ENFORCE in core mutation policy (assertBundledMutationHeldOut): mutating a bundled skill in place requires a non-empty (>=5), benchmark-disjoint held-out set or hard-refuses. Add three eval-internal ablation opts (reflectMode, disableValidationGate, optimizerMode='one-shot-rewrite') recorded in the receipt + audit; ROLLOUT_SUCCESS_THRESHOLD named constant. Security: run_skillopt MCP op validates skill_name (kebab-only) and confines caller-supplied benchmark/held-out paths to the skills dir for remote callers. * test(skillopt): held-out gate, ENFORCE, one-shot rewrite, runtime + receipt honesty New test/skillopt/rollout.test.ts (rollout had zero coverage). Held-out ENFORCE unit cases + one-shot-rewrite fence handling (whole-response unwrap, embedded-fence preserved, error path). E2E: F11 held-out BLOCKS/ALLOWS, bundled no-mutate write, reflectMode/disableValidationGate/optimizerMode, maxRuntimeMin abort, receipt baseline/test-score honesty, held-out/benchmark disjointness, D2 no-DB-pollution. * chore: bump version and changelog (v0.42.9.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0 Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and the tutorial's bundled-skill step: mutating a bundled skill in place now requires --allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update the receipt contract to the honest baseline/test-score fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): AI SDK v6 toolLoop compat — multi-turn tool calls work again The ai@6.x bump tightened ModelMessage + tool-schema validation, which silently broke every multi-turn tool loop. Both `gbrain skillopt` rollouts and production background `subagent` jobs route through `chat()`/`toolLoop` and crashed the moment the model called a tool ("messages do not match the ModelMessage[] schema" / "schema is not a function"). Surfaced end-to-end by the SkillOpt real-LLM eval. Three fixes: - chat(): wrap tool defs with the SDK's `jsonSchema()` helper instead of a bare `{jsonSchema}` object (v6 asSchema() treated the bare object as a thunk and threw). - chat(): new exported pure `toModelMessages()` converts gbrain's provider-neutral ChatMessage[] into v6 ModelMessage[] — tool results ride a dedicated `role:'tool'` message with structured `{type,value}` output; null output preserved as json null. Load-bearing for the production subagent path, not just skillopt. - rollout.ts: replace the inline params→schema mapper (dropped `items` on array params) with the shared `paramDefToSchema` single source of truth. Pinned by test/gateway-model-messages.test.ts (8 cases). Folds into the open v0.42.9.0 PR (#1759) — these complete the eval-readiness wave by making skillopt actually run against a live model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skillopt): budget no-pricing for Haiku silently scored every rollout 0 Surfaced by the SkillOpt real-LLM eval (Track B). Two coupled bugs that made a budget-capped Haiku run report a vacuous "0/N" measurement in ~2ms with zero LLM calls — indistinguishable from a real deficient-skill score: 1. Claude Haiku 4.5's canonical dateless id (`claude-haiku-4-5`) was missing from anthropic-pricing.ts (only the dated `-20251001` was present). With `--max-cost` set, BudgetTracker.reserve() threw no_pricing on the FIRST chat() of every rollout. Added the dateless entry (sonnet already had its dateless form). 2. runValidationGate swallowed that BUDGET_EXHAUSTED error — runWithLimit settled it as {ok:false}, which the gate turned into median:0. A pricing/cap crash became a fake score. The gate now scans settled results for isMustAbortError() and re-throws so the caller aborts loudly; ordinary (non-abort) rollout errors still fail-open to 0 (judge-hiccup posture kept). Pinned by test/skillopt/validate-gate-abort.test.ts (3 cases). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): llms-full.txt over size budget — drop what-schemas-unlock from full bundle The toolLoop + budget bug-fix annotations grew CLAUDE.md, pushing llms-full.txt to 756KB over the 750KB FULL_SIZE_BUDGET (the `build-llms > size budget` test failed, failing the `test` CI job). CLAUDE.md stays inlined by design (it's the point of the one-fetch bundle), so per the budget comment's own guidance ("ship with includeInFull=false exclusions") this excludes docs/what-schemas-unlock.md (15.4KB value-explainer, not load-bearing operational reference) from llms-full.txt; it stays linked in llms.txt. Bundle now 740KB with ~9KB headroom. No budget bump — 750KB is near the ~190k-token-context fit ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ci): re-admit policy docs into ci-cache-hash before doc relocation docs/**/*.md is deny-listed from the CI cache hash (test-irrelevant). The CLAUDE.md restructure moves test/release POLICY into docs/TESTING.md + docs/RELEASING.md, which DO carry contracts the test suite reads. Without re-admitting them, a policy-only edit would produce the same cache hash and skip the test shard that runs the build-llms + doc-history guards (false-pass). Adds an ALLOW_PATTERNS re-admit step after the deny, scoped to the named policy docs (not a blanket docs un-deny). Lands FIRST, before any doc moves. Pinned by 3 new cases in test/scripts/ci-cache-hash.test.ts: TESTING.md + RELEASING.md edits MUST change the hash; docs/guide.md still must not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate Key files / thin-client / Testing out of CLAUDE.md (verbatim) CLAUDE.md had grown to 592KB / ~147k tokens auto-loaded every session (~77% of the llms-full.txt single-fetch bundle). The per-file index was append-only by mandate. This is the exact thin-dispatcher-vs-fat-blob anti-pattern gbrain exists to fix, so CLAUDE.md becomes a thin orientation + resolver that points at on-demand docs. This commit is the VERBATIM move (content-preserving — the next commit compresses): - docs/architecture/KEY_FILES.md <- ## Key files + the calibration key-files cluster + Schema Cathedral v3 impl detail - docs/architecture/thin-client.md <- ## Thin-client routing - docs/TESTING.md <- ## Testing - ## Commands DROPPED (18 'added in vX.Y' history blocks; current surface is gbrain 0.41.38.0 -- personal knowledge brain USAGE gbrain <command> [options] SETUP init [--pglite|--supabase|--url] Create brain (PGLite default, no server) migrate --to <supabase|pglite> Transfer brain between engines upgrade Self-update check-update [--json] Check for new versions doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings) integrations [subcommand] Manage integration recipes (senses + reflexes) PAGES get <slug> Read a page put <slug> [< file.md] Write/update a page delete <slug> Delete a page list [--type T] [--tag T] [-n N] List pages SEARCH search <query> Keyword search (tsvector) query <question> [--no-expand] Hybrid search (RRF + expansion) ask <question> [--no-expand] Alias for query IMPORT/EXPORT import <dir> [--no-embed] Import markdown directory sync [--repo <path>] [flags] Git-to-brain incremental sync sync --watch [--interval N] Continuous sync (loops until stopped) sync --install-cron Install persistent sync daemon export [--dir ./out/] Export to markdown export --restore-only [--repo <p>] Restore missing supabase-only files [--type T] [--slug-prefix S] With optional filters FILES files list [slug] List stored files files upload <file> --page <slug> Upload file to storage files upload-raw <file> --page <s> Smart upload (size routing + .redirect.yaml) files signed-url <path> Generate signed URL (1-hour) files sync <dir> Bulk upload directory files verify Verify all uploads EMBEDDINGS embed [<slug>|--all|--stale] Generate/refresh embeddings LINKS link <from> <to> [--type T] Create typed link unlink <from> <to> Remove link backlinks <slug> Incoming links graph <slug> [--depth N] Traverse link graph (returns nodes) graph-query <slug> [--type T] Edge-based traversal with type/direction filters [--depth N] [--direction in|out|both] TAGS tags <slug> List tags tag <slug> <tag> Add tag untag <slug> <tag> Remove tag TIMELINE timeline [<slug>] View timeline timeline-add <slug> <date> <text> Add timeline entry TOOLS extract <links|timeline|all> Extract links/timeline (idempotent) [--source fs|db] fs (default) walks .md files; db iterates engine pages [--dir <brain>] brain dir for fs source [--type T] [--since DATE] filters (db source) [--dry-run] [--json] publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256) check-backlinks <check|fix> [dir] Find/fix missing back-links across brain lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter orphans [--json] [--count] Find pages with no inbound wikilinks salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type) transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only) dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly). See also: autopilot --install (continuous daemon). check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY) report --type <name> --content ... Save timestamped report to brain/reports/ BRAIN (capture / ideate / explore — v0.37/v0.38) capture [content] [--file PATH] Single entrypoint for getting content into the brain [--stdin] [--slug s] [--type t] Inline content / file / stdin; writes to inbox/ by default [--source ID] [--quiet|--json] Multi-source brains: route to a non-default source brainstorm <question> [--json] Bisociation idea generator (hybrid search + far-set + judge) [--save|--no-save] [--limit N] lsd <question> [--json] Lateral Synaptic Drift: inverted-judge brainstorm [--save|--no-save] [--limit N] rewarding far-from-obvious + axiomatic inversions SOURCES (multi-repo / multi-brain) sources list Show registered sources sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki') sources remove <id> Remove a source + its pages sync --all Sync all sources with a local_path sync --source <id> Sync one specific source repos ... DEPRECATED alias for 'sources' (v0.19.0) CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) code-def <symbol> [--lang l] Find the definition of a symbol across code pages code-refs <symbol> [--lang l] Find all references to a symbol (JSON-first) code-callers <symbol> Who calls this symbol? (v0.20.0 A1) code-callees <symbol> What does this symbol call? (v0.20.0 A1) query <q> --lang <l> Filter hybrid search to one language (v0.20.0) query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) sync --strategy code Sync code files into the brain JOBS (Minions) jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run] jobs list [--status S] [--limit N] List jobs jobs get <id> Job details + history jobs cancel <id> Cancel job jobs retry <id> Re-queue failed/dead job jobs prune [--older-than 30d] Clean old jobs jobs stats Job health dashboard jobs work [--queue Q] Start worker daemon (Postgres only) ADMIN stats Brain statistics health Brain health dashboard history <slug> Page version history revert <slug> <version-id> Revert to version features [--json] [--auto-fix] Scan usage + recommend unused features autopilot [--repo] [--interval N] Self-maintaining brain daemon config [show|get|set] <key> [val] Brain config storage status [--repo <path>] Storage tier status and health [--json] (git-tracked vs supabase-only) serve MCP server (stdio) serve --http [--port N] HTTP MCP server with OAuth 2.1 --token-ttl N Access token TTL in seconds (default: 3600) --enable-dcr Enable Dynamic Client Registration --public-url URL Public issuer URL (required behind proxy/tunnel) call <tool> '<json>' Raw tool invocation version Version info --tools-json Tool discovery (JSON) Run gbrain <command> --help for command-specific help. + the per-command KEY_FILES entries; content stays in git) CLAUDE.md gains: a Reference map (resolver), a Maintaining section (the anti-disease rule), and a Cross-cutting invariants subsection under Architecture so the must-never-violate rules (trust fail-closed, sourceScopeOpts isolation, JSONB trap, engine parity, contract-first, migrations, multi-source) still auto-load after the index moved out. Result: CLAUDE.md 592KB -> 61KB; llms-full.txt 740KB -> 210KB (new docs link-only until compressed). build-llms drift + budget test green; verify 29/29 green. The pre-move content is recoverable at git show <this^>:CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): compress relocated docs to current-state + add recurrence guard Compresses the verbatim-relocated reference docs from append-only release-history to current-state-only (the disease cure), then makes recurrence structurally impossible via a CI guard. Compression (fan-out subagents + adversarial verify, audited mechanically): - KEY_FILES.md 453KB -> 356KB; TESTING.md 42KB -> 38KB; thin-client.md already clean. - 393/393 entries preserved; every src/test/scripts path from the verbatim original survives (mechanical comm-check); zero bolded **v0. markers remain. - Conservative ratio (~22%) because the content is invariant-dense — correctness over brevity. Dropped: **vX.Y.Z (#NNN):** clauses, codex/review tags, contributor credits, PR-numbers-as-ids, pre-fix/then/was-now history deltas. Kept: every exported symbol, invariant, and Pinned-by reference. Verbatim original recoverable at git show <relocation-commit>:docs/architecture/KEY_FILES.md. Recurrence guard (scripts/check-key-files-current-state.sh, wired into verify + check:all): - HARD: bans the bolded **v0.<digit> marker in the reference docs (scoped — plain 'as of pgvector 0.7' prose is fine, no false positives). - HARD: CLAUDE.md size cap (90KB; currently 61KB) — the structural backstop. - Pinned by test/scripts/check-key-files-current-state.test.ts (7 cases). Content contracts (test/build-llms.test.ts, +5 cases per codex outside-voice): CLAUDE.md keeps inline ship IRON RULES (version format, document-release, never-hand-roll); AGENTS.md keeps its boot order; llms indexes the new docs; KEY_FILES stays link-only (not inlined). Privacy: scrubbed the relocated 'wintermute/chat/' source-boost examples + the literal harvest-lint regex to generic placeholders (legitimate in allowlisted CLAUDE.md; genericized for the new public docs per the privacy rule). Reverts the |