mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
8f45624e55fdb8b63560b905b2cfb697e897eb47
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
1eb430a2df |
v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts (#1999)
* fix(security): scope cross-source reads to the caller grant; close get_page exact-path leak One shared resolveRequestedScope() routes every source-scoped read op (query, code_callers/callees, search_by_image, code_blast/flow, get_page) through a single fail-closed trust+grant ladder: a remote caller's __all__ collapses to its granted sources (never the whole brain) and an explicit out-of-grant source_id is rejected. get_page's exact-match path now honors a federated grant via getPage(sourceIds[]) in both engines. Legacy bearer tokens carry their stored permissions.source_id grant (bounded, never widened). Also retries getConfig on transient connection loss. Closes #1924, #1371, #1393, #1336, #1603. * fix(ingest): non-string frontmatter no longer aborts lint/sync; embed/hook/catalog papercuts Parser coerces a non-string title to a string and falls back to inference for slug/type (never fabricating a "123" slug), with a lint NON_STRING_FIELD finding surfacing the malformed frontmatter; a defensive guard in content-sanity stops a non-string title from crashing the whole lint/sync run brain-wide. Plus: embed --catch-up no longer arms the overflowed 32-bit budget timer (and surfaces unembeddable chunks); the frontmatter pre-commit hook ships a correct .md/.mdx regex; and the skill catalog parses YAML block-scalar descriptions. Closes #1883, #1658, #1556, #1948, #1946, #1840, #1711. * v0.42.37.0 fix(security,ingest): source-isolation grant enforcement + non-string frontmatter guard + papercuts Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add NON_STRING_FIELD frontmatter validation class to docs for v0.42.37.0 The v0.42.37.0 non-string-frontmatter fix added an eighth validation class (NON_STRING_FIELD / lint code frontmatter-non-string-field). Update the two current-state docs that enumerate the validation classes: - skills/frontmatter-guard/SKILL.md (seven->eight + table row) - docs/integrations/pre-commit.md (seven->eight + table row) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
dab441f59f |
v0.41.4.0 wave: local providers + cross-platform stdin + gateway-routed dream judge (6 community PRs) (#1377)
* fix(cli): use fd 0 instead of '/dev/stdin' for cross-platform stdin reads
`readFileSync('/dev/stdin', 'utf-8')` works on Unix but fails on Windows
(Git Bash, PowerShell, cmd) with `ENOENT: no such file or directory,
open '/dev/stdin'`. Windows doesn't expose `/dev/stdin` as a filesystem
path.
Reading file descriptor 0 directly (`readFileSync(0, 'utf-8')`) is the
documented Node.js idiom and works on every platform. No behavior change
on Unix — same syscall path, same semantics.
Repro on Windows before the fix:
echo "test" | gbrain put my-page
ENOENT: no such file or directory, open '/dev/stdin'
After: round-trip put/search/delete works on Windows Git Bash.
* v0.40.6.1 feat: llama-server reranker — local Qwen3 / self-hosted ZE via llama.cpp
Adds local reranker support so users can point gbrain's reranker call at their
own llama.cpp server instead of ZeroEntropy's hosted API. One new recipe
(`llama-server-reranker`), a `path?: string` + `default_timeout_ms?: number`
extension on `RerankerTouchpoint`, env passthrough wiring, budget-tracker
`FREE_LOCAL_RERANK_PROVIDERS` set so `--max-cost` callers don't TX2 hard-fail on
local rerank, and a doctor-probe divergence fix (probe and live search now read
the same `search.reranker.model` path via `loadSearchModeConfig` + `resolveSearchMode`).
ZE-hosted users are unchanged. Voyage / Cohere / vLLM rerankers stay out of
scope — different wire shapes need adapter hooks designed against their actual
shapes in a follow-up plan.
Verification:
- `bun run verify` (typecheck + 13 pre-checks): clean
- `bun run check:all` (15 historical checks): clean
- 107/107 expect() calls pass across 5 affected test files
- /codex review against the full diff: GATE PASS (caught one [P2] /v1 path
doubling bug pre-merge; fixed by changing recipe path to leaf `/rerank`)
- Claude adversarial subagent: 7 net-new findings filed as v0.40.7+ TODOs
(none currently exploitable; hardening for future contributor traps)
Test surface (107 cases, 5 files):
- test/ai/rerank.test.ts: path override (exact URL match), default_timeout_ms
honored, empty models[] accepts any id, ZE regression
- test/ai/recipe-llama-server-reranker.test.ts: recipe shape regression guard
+ base_url + path concat assertion (codex-caught /v1/v1/ regression)
- test/search-mode.test.ts: timeout precedence chain (per-call > config >
recipe > bundle), ZE no-recipe-default regression, unknown provider fallthrough
- test/models-doctor-reranker.test.ts: divergence-fix helper across DB-plane
read, mode default, disabled, override, DB-error graceful fallback
- test/core/budget/budget-tracker.test.ts: free-local rerank pricing + arbitrary
model id + chat-kind TX2 hard-fail preserved
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs: post-ship documentation sync
* docs: index docs/ai-providers/ in llms.txt (zeroentropy + llama-server-reranker)
The hand-curated llms-config.ts doc map never included docs/ai-providers/, so
both zeroentropy.md (since v0.35.0.0) and the new llama-server-reranker.md were
invisible to the AI-facing llms.txt / llms-full.txt index. Adds an "AI providers"
section with both. Marked includeInFull: false (setup walkthroughs belong in the
index but would push the single-fetch bundle past FULL_SIZE_BUDGET) — same
treatment CHANGELOG.md gets.
Caught by the /ship document-release subagent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: recipe-aware embedding-provider check for local providers
doctor --remediation-plan and autopilot both judged the embedding
provider with a hosted-only key check, so a brain on ollama: or
llama-server: was reported "blocked" on a missing API key it never
needed, contradicting doctor --json's 100%-coverage health.
Extract a shared embeddingProviderConfigured() helper into
brain-score-recommendations.ts: empty auth_env.required (local
providers) is configured with no key; hosted providers check their
OWN required key. Both producers (doctor, autopilot) call it,
killing the DRY violation that caused the bug. Hosted brains with a
missing key still block.
* fix(budget): price local embed providers at $0
A --max-cost-bounded embed/reindex job configured for ollama: or
llama-server: TX2 hard-failed with no_pricing because
lookupEmbeddingPrice has no entry for local models. Add
FREE_LOCAL_EMBED_PROVIDERS (sibling to FREE_LOCAL_RERANK_PROVIDERS)
so a pricing miss on a local-inference provider returns $0 instead
of null. lmstudio/litellm intentionally excluded.
* feat(models): embedding reachability probe in gbrain models doctor
A down/misconfigured local embed server was invisible until first
embed. Add probeEmbeddingReachability() (mirrors the reranker probe):
a 1-input embed with a 5s abort timeout, classified via classifyError,
under a new 'embedding_reachability' touchpoint, gated on the
zero-network config probe returning ok first.
* fix: don't count config-plane voyage/google keys as configured
codex review caught a false positive: HOSTED_EMBED_KEY_CONFIG mapped
VOYAGE_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY to config fields, but
buildGatewayConfig only threads openai/anthropic/zeroentropy config
keys into the gateway env. A Voyage/Google brain with the key only in
config.json would be judged "configured" and dispatch an embed.stale
job that then fails auth at the gateway. Drop those two from the map so
the producer closures resolve them by env var only, matching what the
gateway can actually use. Pinned by a regression test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(dream): route significance judge through gateway.chat for multi-provider support
Replaces the hardcoded `new Anthropic()` client in the dream-cycle synthesize
phase with a gateway-routed JudgeClient adapter. Mirrors the v0.35.5.0 pattern
that closed #952 for runThink: construction-time provider/key probe returns null
on a clear miss (cheap pre-flight); the verdict loop wraps the chat call in
try/catch for AIConfigError mid-run.
Any provider with a registered gateway recipe (Anthropic, DeepSeek, OpenRouter,
Voyage, Ollama, llama-server, etc.) is now reachable via:
gbrain config set models.dream.synthesize_verdict <provider>:<model>
The canonical config key `models.dream.synthesize_verdict` (per PER_TASK_KEYS
in src/core/model-config.ts) is used unchanged. The exported JudgeClient
interface signature is preserved for test-seam stability.
The original community PR (#1349) shipped a custom fetch adapter that
bypassed the gateway entirely. This reworked landing routes through the
canonical seam so future provider additions automatically benefit, and a
CI guard (T7) will land in this wave to prevent the bug class from
re-opening (the same one that bit src/core/think/index.ts before v0.35.5.0).
Co-Authored-By: justemu <206393437+justemu@users.noreply.github.com>
* test(dream): synthesize-gateway-adapter unit tests + R3 parsed-verdict parity
11 cases pin the gateway-routed JudgeClient adapter from T5:
- A1: makeJudgeClient returns null on missing Anthropic key (legacy short-circuit preserved)
- A2: returns a JudgeClient when chat provider is reachable
- A3: JudgeClient.create routes through gateway.chat (via __setChatTransportForTests)
- A4: ChatResult.text → Anthropic.Message.content[0].text mapping
- A5: empty text from gateway → graceful empty-text Anthropic.Message
- A6: non-AIConfigError from gateway propagates to caller (no swallow)
- A7: AIConfigError from gateway propagates as AIConfigError (caught per-transcript in production loop)
- A8: makeJudgeClient returns null on unknown provider prefix
- A9: returns a JudgeClient for non-anthropic providers without env-probing (delegates to gateway at call time)
- R3: parsed-verdict SEMANTIC parity — gateway-routed and legacy SDK-shape JudgeClients produce same {worth_processing, reasons} given identical canned LLM text
- R3 corollary: unparseable LLM output → both paths fall through to cheap-fallback verdict
Codex flagged byte-identical-Anthropic.Message as a meaningless gate; R3 is
parsed-verdict semantic parity instead. Mirror pattern of
test/think-gateway-adapter.test.ts for cross-site consistency with the
v0.35.5.0 runThink migration.
* ci: guard against direct Anthropic SDK construction in gateway-routed files
New scripts/check-gateway-routed-no-direct-anthropic.sh greps two guarded
files (src/core/cycle/synthesize.ts and src/core/think/index.ts) for
`new Anthropic()` constructor calls and runtime imports of @anthropic-ai/sdk.
Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay
allowed because both files use Anthropic.Message / .MessageCreateParamsNonStreaming
as adapter types.
Comment lines (starting with `//` or ` *`) are excluded so historical
references in JSDoc don't false-fire. Negative test in this commit's
verification confirms: injecting `new Anthropic()` into synthesize.ts
makes the guard exit 1 with a clear error pointing at the gateway adapter
pattern; reverting restores the OK state.
Wired into both `bun run verify` and `bun run check:all`. Closes the bug
class that bit synthesize.ts in PR #1349 (which would have shipped a
parallel fetch stack instead of routing through the canonical gateway).
The same class previously bit think/index.ts and was fixed structurally
in v0.35.5.0; this guard prevents either file from regressing.
Extend GUARDED_FILES in the script when migrating another file off
direct SDK construction.
* docs(put_page): point Windows / pipe-buffer users at gbrain capture --file
Extends the put_page op description (surfaced by `gbrain put --help`) with a
one-line pointer to `gbrain capture --file PATH --slug SLUG` for the file-
as-input use case. Capture (v0.39.3.0) is the canonical Windows-pipe-buffer
escape route: reads files as a Buffer first, scans the first 8KB for NUL bytes
to refuse binary content, decodes to UTF-8 only after the safety check, and
adds provenance write-through.
Lands the user-facing value the closed PR #1365 was reaching for, without
duplicating the CLI surface. Credits the original contributor.
Co-Authored-By: ecat2010 <90021101+ecat2010@users.noreply.github.com>
* test: R1+R2+R4 critical regression pins for the community-PR-wave landing
Per the wave's eng-review plan (IRON RULE — mandatory):
R1 — get_page handler accepts calls without `content` param. Pre-wave
PR #1365 landed its `!p.content → throw` check in the WRONG handler
(get_page instead of put_page), which would have broken every read
in the system. Pin: get_page MUST NOT require content + the schema
carries no `content` or `file` param.
R2 — put_page schema content stays `required: true`. PR #1365 also
flipped `content` from required→optional in the schema. Pin: the
contract stays at `required: true` + the closed PR's `file` param
is NOT in the schema.
R4 — Cross-platform stdin via fd 0 (PR #1325 regression pin). Source-grep
asserts src/cli.ts uses `readFileSync(0, ...)` and NOT the legacy
`readFileSync('/dev/stdin', ...)`. Belt-and-suspenders pattern
assertions confirm the parseOpArgs branch shape (cliHints.stdin
check, 5MB cap, isTTY gate) hasn't drifted.
R3 (gateway-adapter parsed-verdict parity) lives in the sibling file
test/cycle/synthesize-gateway-adapter.test.ts.
* test(e2e): update dream-synthesize no-key reason text + harden hermeticity
After T5's gateway-adapter rework, the "no API key" verdict text changed from
'no ANTHROPIC_API_KEY for significance judge' to
'no configured provider for verdict model: <model>' (broader + names the
actual model so the user sees WHICH provider failed). Update both assertions
that check the old text.
Hermeticity bug fix in the same commit: `withoutAnthropicKey` previously only
cleared the env var. After the rework, `makeJudgeClient` ALSO checks
`loadConfig().anthropic_api_key` (same hasAnthropicKey() pattern think/index.ts
uses since v0.35.5.0). If the developer running the test has the key set in
~/.gbrain/config.json, the test would behave non-deterministically. Fix:
override GBRAIN_HOME to a fresh tmpdir for the duration of the body, restore
on return (even on throw).
* test(e2e): pin verdict-loop AIConfigError catch from T5 rework end-to-end
Drives runPhaseSynthesize against a real PGLite engine with the gateway
chat transport stubbed to throw AIConfigError on every call (simulates a
revoked/misconfigured provider surfacing mid-run). Asserts:
- Phase does NOT crash; converts the throw to a per-transcript verdict
with worth=false and reasons[0] matching "gateway error: ...".
- status='ok' so subsequent transcripts in the loop would continue
being judged (not visible in 1-transcript test, but the loop shape is
proven not to abort).
Pre-rework (T5), this code path didn't exist — judgeSignificance threw
directly to runPhaseSynthesize and crashed the whole phase. Pin so a
future regression that removes the try/catch fires loudly.
* docs(claude.md): annotate v0.41+ community-PR-wave changes
Two additions to the Key files section:
- src/core/cycle/synthesize.ts — appends a v0.41+ paragraph documenting
the gateway-adapter rework (makeJudgeClient + AIConfigError catch loop +
canonical config key + JudgeClient interface preserved + CI guard
reference + test file references).
- scripts/check-gateway-routed-no-direct-anthropic.sh — new entry
documenting the CI guard's contract, scope, and how to extend
GUARDED_FILES when migrating another file off direct SDK construction.
CLAUDE.md drives /sync-gbrain and llms.txt generation; both need the
wave's annotations to land BEFORE the llms regeneration step (T10).
* docs(llms): regenerate llms.txt + llms-full.txt for v0.41+ wave
Refreshes the auto-generated llms.txt bundles to pick up the CLAUDE.md
annotations landed earlier in this wave (gateway-adapter synthesize.ts
+ check-gateway-routed-no-direct-anthropic.sh + the cherry-picked
llama-server-reranker recipe). Pinned by test/build-llms.test.ts.
* fix(providers): dynamic-width id column accommodates llama-server-reranker
v0.40.6.1 introduced `llama-server-reranker` (21 chars), which overflowed
formatRecipeTable's static 14-char PROVIDER column. When the id is longer
than the column, padEnd is a no-op — the row starts with the tier name
directly, no space delimiter. test/providers.test.ts 'each recipe appears
at most once' iterates every recipe and asserts at least one row starts
with `${id} ` or `${id} `; with no space after `llama-server-reranker`,
the assertion fails and the recipe appears effectively missing from the
human-readable list.
Fix: compute column width dynamically as `max(14, max(id.length) + 1)` so
every id is followed by at least one space, regardless of length. Also
widens the separator rule to match. 14 stays as the floor so the existing
short-id rows (openai 6, ollama 6, anthropic 9, ...) keep their familiar
layout when llama-server-reranker isn't in the active recipe set.
10/10 cases in test/providers.test.ts pass after the fix.
* chore: pre-landing review polish — refresh models doctor tip + file embed timeout TODO
Two pre-landing review absorptions:
- `src/commands/models.ts:154` — the help-text tip said `gbrain models doctor`
"spends ~1 token per model" but the wave added an `embed(['probe'])` call
AND a reranker probe. Generalize to "spends a minimal request per configured
chat/embed/rerank surface" so the cost expectation matches reality.
- `TODOS.md` — file a follow-up to widen `default_timeout_ms` from
RerankerTouchpoint to EmbeddingTouchpoint so `probeEmbeddingReachability`
doesn't hardcode 5000ms while the sibling reranker probe reads the
recipe's configured timeout. Local CPU embedding endpoints (llama-server)
hit the same cold-start curve as Qwen3-Reranker-4B; workaround today is
"re-run the probe" per the existing JSDoc.
Other informational findings from pre-landing review either match
established patterns (no behavioral test for `probeEmbeddingReachability`,
matching `probeRerankerReachability`), are intentional choices documented
in JSDoc (the `as unknown as Anthropic.Message` cast), or are micro-perf
in non-hot paths (autopilot's 4 sequential `getConfig` awaits per
5-minute tick). All non-blocking.
* ci: tighten gateway-routed guard against import bypass shapes + honest JSDoc
Adversarial review caught two soft spots in the wave's new contracts:
1. `scripts/check-gateway-routed-no-direct-anthropic.sh` only matched the
default-import shape `import Anthropic from '@anthropic-ai/sdk'`. A future
contributor (or, more realistically, a future refactor) could bypass with:
- `import { Anthropic } from '@anthropic-ai/sdk'`
- `import { Anthropic as A } from '@anthropic-ai/sdk'`
- `import * as Anthropic from '@anthropic-ai/sdk'`
- `const x = await import('@anthropic-ai/sdk')`
Tightened the regex to match ANY value-shaped import from the SDK module
(excluding only the explicit `import type ... from '@anthropic-ai/sdk'`
form which the adapter's Anthropic.Message return type needs). Added a
second grep for dynamic imports. Verified all four bypass shapes now
trigger the guard against synthesize.ts; type-only import still passes.
2. `synthesize.ts:makeJudgeClient` JSDoc claimed the adapter "tolerates the
array-of-blocks shape for future flexibility" — but the mapping flattens
ONLY text blocks; `tool_use`, `tool_result`, image blocks silently
become empty strings. Today only `judgeSignificance` calls this and it
only sends string content, so no behavior bug. But the comment was
marketing future flexibility the code doesn't deliver. Narrowed to call
out the silent-drop and say to extend the mapping if a future caller
wires non-text content through.
Both wave-scope: the CI guard was added by the wave, the JSDoc was added
by the wave's T5 rework. Adversarial review caught them before merge.
* fix(models doctor): reranker probe timeout matches live search precedence chain
Codex Pass-9 adversarial review caught a probe-vs-production divergence:
production `hybridSearch` resolves reranker timeout via the full chain
(per-call > config > recipe > bundle) by going through
`loadSearchModeConfig + resolveSearchMode`, but `probeRerankerReachability`
was reading ONLY the recipe's `default_timeout_ms` — so an operator who
set `search.reranker.timeout_ms=1000` would see doctor wait 30s and report
"reachable" while production search timed out at 1s and fail-opened.
A higher configured timeout produces the opposite false failure (probe
gives up at 5s when production would have waited longer).
Fix: extract `resolveLiveRerankerTimeoutMs(engine)` parallel to the
existing `resolveLiveRerankerModel(engine)` — same precedence chain,
same DB-plane consistency posture. The probe now reads the SAME timeout
live search reads, on the same lookup path.
The codex P1 finding about `FREE_LOCAL_*_PROVIDERS` zero-pricing being
bypassable via redirected `LLAMA_SERVER_BASE_URL` is filed as a TODO under
community-pr-wave follow-ups — couples with the existing
FREE_LOCAL_PROVIDERS unification TODO so both close in one v0.41+ PR.
* ci(guard): handle mixed type+value imports + macOS BSD sed POSIX classes
Codex structured review [P3] caught a bypass in the freshly-tightened
gateway-routed guard:
import { type Message, Anthropic } from '@anthropic-ai/sdk';
new Anthropic();
The previous regex `^\s*import\s+[^t][^y]*from ...` was meant to exclude
`import type ...` but stops at the `y` in `type` inside the brace list,
silently allowing the value-import `Anthropic` through. Two fixes:
1. Replace the brittle regex-based type-exclusion with a clause-level
parse: extract the brace-list specifiers, allow the import iff EVERY
non-empty specifier is `type`-prefixed. Catches mixed-import bypasses
(`{ type Foo, Bar }`) while keeping all-type braces (`{ type Foo, type Bar }`)
passing. Default + namespace imports remain always-value-shaped.
2. Replace `\s` with POSIX `[[:space:]]` in the sed extract — macOS BSD sed
doesn't honor `\s` in extended-regex mode (it silently no-ops the pattern
so `specifiers` comes back empty and the script falls through to the
default/namespace branch's wrong error message).
Hermetic 7-shape regression matrix now verifies every TypeScript import
shape against the expected ALLOW/BLOCK verdict; all 7 pass:
- ALLOW: `import type Anthropic from '...'`
- ALLOW: `import type { Foo } from '...'`
- ALLOW: `import { type Message, type Foo } from '...'`
- BLOCK: `import { type Message, Anthropic } from '...'`
- BLOCK: `import { Anthropic } from '...'`
- BLOCK: `import Anthropic from '...'`
- BLOCK: `import * as A from '...'`
Subshell-trap fix in the same commit: the previous "exit 1 inside while-pipe"
pattern doesn't propagate to the outer `$?` because the pipe spawns a
subshell. Switched to a tmpfile-flagged sentinel so the verdict survives
the subshell boundary cleanly.
* chore: bump version and changelog (v0.41.4.0)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(audit-writer): route log() to file matching event ts, not real-now
CI failure surfaced a time-dependent test flake in
`test/audit/audit-writer.test.ts` "returns events from current week,
filtered by ts cutoff" (added in v0.40.4.0 PR #1300). The test pinned
synthetic `now = 2026-05-22T12:00:00Z` (ISO week 21), logged 3 events
with synthetic ts values, then called `readRecent(7, now)` expecting
to find 2 events in window.
Root cause: `log()` ignored the caller-supplied `ts` for filename
routing and ALWAYS wrote to the file matching real-time-now's ISO
week. When real CI time crossed into 2026-W22 (this Monday), the
events went to W22's file but `readRecent` walked W21 + W20 → 0 hits.
Fix:
- `log()` parses `event.ts` (when provided) and routes to the file
matching that ts's ISO week. Falls back to real-now when ts is
missing or unparseable.
- No behavior change for production callers — none of the 5 audit
consumers pass `ts` explicitly (rerank-audit, audit-slug-fallback,
content-sanity-audit, graph-signals, supervisor-audit). The writer
stamps real-now → both ts and filename use real-now → same file
as before.
- Sibling test "honors caller-supplied ts override" also pinned a
fixed ts and would have broken from the opposite angle (test
read from `computeFilename()` default = real-now). Updated to
read from `computeFilename(new Date(fixedTs))` so it asserts the
per-row file routing the wave now provides.
22/22 audit-writer cases pass. Production callers (5 sites) unchanged.
Pre-existing on master since v0.40.4.0; surfaced when real time
crossed into a different ISO week than the test's synthetic now.
NOT introduced by this PR (#1377 community-PR-wave) — audit-writer
files aren't touched by the wave.
---------
Co-authored-by: Tobias <34135750+tobbecokta@users.noreply.github.com>
Co-authored-by: kohai-ut <chris@tincreek.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: justemu <noreply@github.com>
Co-authored-by: justemu <206393437+justemu@users.noreply.github.com>
Co-authored-by: ecat2010 <90021101+ecat2010@users.noreply.github.com>
|
||
|
|
d0d0e2a64a |
v0.37.11.0: fresh-install PGLite embedding setup fix wave (#1286)
* chore(test): preload gateway to OpenAI/1536 so 1536-dim test fixtures keep working The v0.37 fix wave changes the canonical gateway defaults to zeroentropyai:zembed-1 / 1280 (matching what v0.36 already chose as the system default). 20+ test files have hardcoded new Float32Array(1536) fixtures that match the OLD schema default. Without this preload, those tests fail with a vector-dim-mismatch on insert. The preload is gateway-only — it doesn't change which model gbrain ships to production users. Tests that want the new ZE/1280 defaults call configureGateway() explicitly in their own beforeAll. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai): canonical embedding defaults + sweep across schema/engines/registry Closes the v0.36 defaults drift bug class. The gateway shipped zeroentropyai:zembed-1 / 1280 as the system default in v0.36 but eight other places kept hardcoding 1536 / text-embedding-3-large. Fresh gbrain init --pglite sized the column to 1536, the embed pipeline used ZE/1280, and every page failed with dim mismatch. - New src/core/ai/defaults.ts leaf module is the canonical source for DEFAULT_EMBEDDING_MODEL / DEFAULT_EMBEDDING_DIMENSIONS. Schema and registry helpers import from this lean module instead of pulling the full gateway (which loads every provider SDK). - src/core/ai/gateway.ts re-exports the constants for back-compat. - src/core/pglite-schema.ts getPGLiteSchema() defaults track gateway. - src/core/postgres-engine.ts getPostgresSchema() default args track gateway (same drift on the Postgres path — codex round 1 CDX-1). - Both engine.initSchema() fallbacks track gateway constants (no more stale OpenAI/1536 catch-block defaults). - Schema seed stops stripping the provider prefix; full provider:model is stored in the DB config table (codex round 1 CDX-4). - Chunk-row INSERT defaults track gateway (codex round 2 CDX2-4 — pglite-engine:1611 + postgres-engine:1647 were production write sites previously hardcoded to text-embedding-3-large). - src/core/search/embedding-column.ts loadRegistry + isCacheSafe gain the cfg > gateway > DEFAULT resolution chain (codex round 2 CDX2-3). The gateway tier matters because callers that configure the gateway (init paths, tests, programmatic SDK) expect the registry to mirror that state when cfg doesn't have an explicit embedding_model. Tests: - schema-templating: default expectation flips to ZE/1280 (v0.37 truth). - embedding-dim-check: 3 new engine-kind branching cases + updated fresh-brain expectation (under legacy preload). - embedding-column: registry + isCacheSafe expectations match new chain. - v0_28_5-fix-wave E2E: engineKind required arg propagated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(init+config+cli): always-configure gateway, file-only loader, honest config-set, sync/reinit help Closes the "fresh init doesn't work + config-set silently lies" bug class end-to-end. Six related changes that ship together because the file-plane/DB-plane contract only holds when init paths, config-set, the gateway env mapping, and the recipe text all agree. Lane B (init paths): - initPGLite, initPostgres, initMigrateOnly always configureGateway() before engine.initSchema(). Pre-fix the call was gated on flags, so bare `gbrain init --pglite` left the gateway unconfigured and the engine fell through to stale OpenAI/1536 defaults instead of the ZE/1280 the gateway would have resolved. - New configureGatewayWithMergedPrecedence() helper applies the locked precedence chain `CLI > env > existing file > gateway internal`. - printResolvedAIChoice() shows the resolved model/dim at init time + surfaces a ZE setup hint inline when the API key is missing. - B.4: saveConfig merge uses loadConfigFileOnly() so transient env state (DATABASE_URL, etc.) never poisons ~/.gbrain/config.json (codex round 2 CDX-5). - B.5: extend the v0.28.5 dim-mismatch detector so it fires when the gateway-resolved dim differs from the existing column, not only when --embedding-dimensions is explicit (codex round 2 CDX-6). Lane C (config plane): - New `loadConfigFileOnly()` reads ~/.gbrain/config.json only — no env merge, no DATABASE_URL inference. Safe write-back source for init. - GBrainConfig gains `zeroentropy_api_key?: string`. loadConfig merges process.env.ZEROENTROPY_API_KEY. buildGatewayConfig at cli.ts:1401 maps it into env.ZEROENTROPY_API_KEY so ZE recipes finally see it (codex round 2 CDX2-5+6 — the v1 fix landed in the wrong file). - `gbrain config set embedding_model` and `... embedding_dimensions` refuse unconditionally and print a paste-ready wipe-and-reinit recipe. No --force escape (codex round 2 CDX2-13). - migrate-engine.ts adds a contract comment at the DB-plane write site documenting "DB stores schema-applied metadata; file plane is canonical for runtime gateway config" + preserves the existing file-plane config across engine migration. Lane D.1 (recipe text): - embeddingMismatchMessage() takes an `engineKind` arg. PGLite branch emits a wipe-and-reinit recipe using gbrainPath('brain.pglite') or the caller's databasePath override. Postgres branch keeps the SQL ALTER recipe. - The PGLite recipe recommends `gbrain reinit-pglite` (new sugar command below) as the one-line path before falling back to the by-hand mv + init + sync sequence. Lane D.4 (sync help dispatch): - `sync` and `reinit-pglite` added to CLI_ONLY_SELF_HELP so their own --help branches reach the user (pre-fix the generic short-circuit fired first and the dedicated usage was unreachable; codex round 2 CDX2-12). - `gbrain sync --help` short-circuits BEFORE engine bind so users on a fresh tmpdir (no config) can read the help without hitting no-such-config errors. Sugar: - New `gbrain reinit-pglite --embedding-model X --embedding-dimensions N` wraps the wipe + init + sync dance into one command. Backs up the brain to <path>.bak. TTY confirmation unless --yes. --no-sync to defer the resync. --json for scripts. Tests: - test/cli.test.ts sync-help test rewritten for the new per-command-usage output (lists --no-embed which is the v0.37 user-visible flag the wave wanted to surface). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(embed+sync): pre-flight dim-mismatch guard + sync hint at both catch sites embedding-pipeline error UX. Pre-fix, a fresh-install dim mismatch produced raw Postgres "expected N dimensions, not M" errors page after page, surfacing only after the worker pool drained the entire corpus. Sync swallowed embed errors at TWO catch sites and never surfaced the recovery recipe. embed.ts: - New `EmbeddingDimMismatchError` tagged class with the paste-ready recipe baked in. - `runEmbedCore` pre-flights via `readContentChunksEmbeddingDim` + gateway.getEmbeddingDimensions() before the worker pool spins up. On mismatch, throws the typed error which the CLI wrapper catches and prints. Dry-run skips the check (no embed risk). - Catches the headline fresh-install bug class at first call instead of letting it hammer N parallel API calls into dim-rejected inserts. sync.ts: - Both embed catches at sync.ts:990 (incremental) and sync.ts:1129 (first-sync) detect EmbeddingDimMismatchError and surface the recipe + a `--no-embed` tip on stderr (codex round 2 CDX2-8: incremental path was previously silent; only the first-sync path was flagged). - Non-mismatch embed failures still stay best-effort (rate limits, transient network) — those shouldn't break sync. - Sync calls runEmbedCore directly instead of runEmbed (which calls process.exit on error and bypasses sync's catch). - Sync gets a proper --help block listing every meaningful flag: --no-embed, --workers, --source, --skip-failed, --retry-failed, --watch, --interval, --no-pull, --all, --json, --yes, --dry-run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): read gateway for schema-sizing checks + provider-aware key lookup Doctor's embedding checks were reading the DB config table for embedding_model / embedding_dimensions / zeroentropy_api_key. Post v0.37 the file plane is canonical (the DB plane is schema-applied metadata, not runtime gateway config) so those reads produced stale verdicts on fresh installs whose DB row hadn't been written. - checkEmbeddingWidthConsistency reads gateway.getEmbeddingDimensions() and gateway.getEmbeddingModel() instead of engine.getConfig(...). Reuses readContentChunksEmbeddingDim from the same shared helper init + embed use. On mismatch, the fix hint threads engineKind + databasePath into the new branched recipe (codex round 1 CDX-8 + Lane E.1/E.2). - checkZeEmbeddingHealth reads gateway for the model + loadConfigFileOnly for the key. Fires when (a) resolved model starts with zeroentropyai: AND (b) ZEROENTROPY_API_KEY is unset in env AND (c) file plane has no zeroentropy_api_key (codex round 2 CDX2-10). - loadRecommendationContext reads gateway for both fields and recognizes the ZE key alongside OpenAI/Anthropic in the hasEmbeddingApiKey check, so brains on ZE no longer look "healthy" just because OPENAI_API_KEY happens to be set (codex round 2 CDX2-11). Tests rewritten for the gateway-source-of-truth contract via configureGateway() in beforeAll. Added a "gateway unconfigured: skips with ok" case so doctor doesn't false-warn on cold-boot brains. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test+docs(v0.37): fix-wave unit coverage + PGLite-first migration recipe + TODOS Lands the v0.37 PGLite fresh-install fix wave's structural tests and the user-facing migration recipe overhaul. test/v0_37_fix_wave.test.ts (new): 22 unit cases pinning the lanes: - Lane A: defaults module exports, getPGLiteSchema/getPostgresSchema default-args, registry + isCacheSafe under the `cfg > gateway > DEFAULT` chain (both gateway-set and gateway-reset branches). - Lane B: loadConfigFileOnly env isolation + DATABASE_URL inference refusal + null-on-missing. - Lane C.3: buildGatewayConfig maps zeroentropy_api_key + process.env wins over config (operator escape hatch contract). - Lane D.2: EmbeddingDimMismatchError shape + tag. - Lane D.4: structural assertion that `sync` is in CLI_ONLY_SELF_HELP. - Deferred-TODO ship: reinit-pglite is registered correctly + embeddingMismatchMessage PGLite branch recommends it. docs/embedding-migrations.md: PGLite section moved to top (the default install). The recommended path is `gbrain reinit-pglite` one-liner; the by-hand mv + init + sync sequence stays as the fallback recipe. Postgres SQL ALTER recipe preserved. New section on `gbrain config set` refusal explains the file-plane vs DB-plane contract so users don't follow stale documentation. TODOS.md: 4 deferred follow-ups filed with concrete file pointers: - gbrain embed --try-fallback (provider auto-switch with consent gate) - Full plane unification for non-schema-sizing fields - Worker-pool shared AbortController for mid-run dim drift - Cleanup of back-compat constants in src/core/embedding.ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.37): fill behavior gaps + headline fresh-install E2E The structural fix-wave tests in test/v0_37_fix_wave.test.ts pin lane-level invariants (exports, registry chain, signature shapes). The audit found 10+ END-TO-END behaviors that the structural tests didn't actually reach. This file fills the highest-leverage gaps. Unit coverage (test/v0_37_gap_fill.test.ts, 12 cases): - Lane A.7: chunk-row INSERT default tracks DEFAULT_EMBEDDING_MODEL constant (pre-fix this was the literal 'text-embedding-3-large' at pglite-engine.ts:1611 + postgres-engine.ts:1647 — production write sites that were never directly tested; codex round 2 CDX2-4). - Lane A.8: schema seed stores full provider:model in DB config (pre-fix the .split(':') strip dropped the prefix; codex round 1 CDX-4). Asserts a fresh ZE init stores `zeroentropyai:zembed-1` in the config table, not bare `zembed-1`. - Lane B precedence: explicit CLI > env > existing file > default test (codex round 2 CDX2-7 contradiction guard). - Lane C.3 env merge: process.env.ZEROENTROPY_API_KEY threads through loadConfig → cfg.zeroentropy_api_key; loadConfigFileOnly does NOT. - Lane D.2 end-to-end: schema=1536 + gateway=1280 → EmbeddingDimMismatchError fires AND the embed transport is never called (the whole point of pre-flight). Plus dry-run skips the check. - Lane D.3 source-text grep: both sync.ts catch sites detect the typed error + the `--no-embed` tip is present (CDX2-8). - Lane E.4 source-text grep: loadRecommendationContext is provider-aware (reads gateway + branches on ZE/OpenAI key). - reinit-pglite contract: refuses on non-PGLite engines + refuses when required flags are missing. E2E (test/e2e/fresh-install-pglite.test.ts, 2 cases): - Bare `gbrain init --pglite` produces a `vector(1280)` schema, prints the resolved choice, persists defaults to config.json — the headline scenario that v0.37 ships to fix. - init → seed page → embed end-to-end: chunks have non-null embeddings; no dim mismatch despite the wave's defaults change. Both E2E cases are IN-PROCESS (per CDX2-12: CLI-subprocess E2E can't inherit `__setEmbedTransportForTests`). They run with stubbed transport returning synthetic 1280-dim vectors so we never hit real provider APIs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.37): defensive gateway restore in reinit-pglite describe block Adds an afterAll that restores the gateway to OpenAI/1536 (matching the bunfig preload) at the end of the reinit-pglite describe. Belt-and- suspenders: earlier describe blocks in this file already restore, but if the reinit-pglite tests ever start mutating the gateway in the future, this protects downstream test files in the same bun-test shard from inheriting a non-default state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.37.10.0) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: scrub stale config-set recipes for embedding model (v0.37.10.0) README + topologies + embedding-providers were still pointing users at `gbrain config set embedding_model X` / `embedding_dimensions N`. As of v0.37.10.0 those writes are refused — the schema column has to resize alongside the config. Point at `gbrain reinit-pglite` (PGLite) and the SQL recipe in `docs/embedding-migrations.md` (Postgres) instead. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: bump version to v0.37.11.0 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: quarantine v0.37 fix-wave tests to .serial.test.ts CI's `check:test-isolation` lint flagged R1 violations (direct `process.env.GBRAIN_HOME` mutation) in both new fix-wave test files. Per the documented quarantine pattern in CLAUDE.md, rename to `*.serial.test.ts` instead of refactoring through `withEnv()` — both files use beforeEach/afterEach env wiring that's already serial-safe. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
a55de71221 |
v0.37.10.0 feat(init): env-detection + interactive picker + preflight invariants (#1278)
* feat(core): Levenshtein helper + preflight schema-dim resolvers Foundation for v0.37.10.0 env-detection wave. Two pure modules: - src/core/levenshtein.ts: editDistance(a,b) + suggestNearest(input, candidates, maxDistance). Used by config-set "did you mean" suggestions and env-var typo detection at init. - src/core/embedding-dim-check.ts: resolveSchemaEmbeddingDim() + resolveSchemaMultimodalDim() pure functions. Validate resolved dim against recipe default_dims + per-provider Matryoshka allow-lists (OpenAI text-3, Voyage flexible-dim, ZeroEntropy zembed-1) BEFORE any DB write. Plus EmbeddingDisabledError + assertEmbeddingEnabled() runtime guard for the deferred-setup path (D9). New PGVECTOR_COLUMN_MAX_DIMS=16000 exported. Tests: 41 unit cases across both modules. * feat(providers): extract formatRecipeTable + add init provider picker Two changes prepping the env-detection wave: - providers.ts: extract formatRecipeTable() helper from runList(). Picker reuses it so UI can't drift from \`gbrain providers list\`. Also adds the codex finding #10 warn-line to \`providers test\` when the tested model differs from the configured default ("Note: tested X in isolation; gbrain's configured embedding is Y — this test does NOT verify your brain's active path."). envReady() takes an explicit env arg for testing. - init-provider-picker.ts (NEW): interactive picker mirroring init-mode-picker.ts. Filters candidate recipes to env-ready ones (codex finding #3), prompts via readLineSafe, exports printSubagentAnthropicCaveat() for shared use from initPGLite/initPostgres. Tests: 17 unit cases (10 providers + 7 picker). * feat(config): embedding_disabled sentinel + strict unknown-key rejection Two changes for the v0.37.10.0 wave: - src/core/config.ts: add embedding_disabled?:boolean to GBrainConfig (D9 deferred-setup sentinel, mutually exclusive with embedding_model). Export KNOWN_CONFIG_KEYS (60+ canonical keys, file-plane + DB-plane) and KNOWN_CONFIG_KEY_PREFIXES (search., models., dream., cycle., etc.) for validation use. - src/commands/config.ts: D6 strict-default unknown-key rejection. Unknown key + no --force → exit 1 with Levenshtein suggestion against KNOWN_CONFIG_KEYS. Prefix matches accepted without --force. --force escape hatch accepts arbitrary keys with stderr WARN. Closes the silent-no-op class the bug reporter hit (embedding.provider, embedding.model, embedding.dimensions all exit 1 with right suggestion). Tests: 19 unit cases pinning the bug-reporter regression + gate logic. * feat(init): env-detection auto-pick + preflight + atomic persist + --no-embedding Core of the v0.37.10.0 wave (D1-D7, D9-D11). Closes the bug where a fresh \`gbrain init --pglite\` silently produced a broken brain when no provider key matched the v0.36 default. resolveAIOptions rewritten with per-touchpoint env detection: - Explicit flag → shorthand → env auto-pick (group by provider id, codex #2) - Picker fires when multiple providers env-ready (D1+D2 hybrid) - Non-TTY zero-key exits 1 with paste-ready setup hint (D3) + Levenshtein typo detection for OPENAPI_API_KEY → OPENAI_API_KEY (D13) - All three touchpoints covered (embedding + expansion + chat, D4) - Local-only providers (Ollama/llama-server) excluded from auto-pick; picking Ollama silently when user has OPENAI_API_KEY set was wrong UX initPGLite + initPostgres: - Drop conditional configureGateway gate → always call before initSchema - Preflight resolveSchemaEmbeddingDim() BEFORE engine.initSchema() (D11) — invalid dim refuses with paste-ready hint, no disk write - Atomic embedding-config persistence (codex #13): either resolved tuple or embedding_disabled:true sentinel, never partial state - Post-initSchema invariant assertion stays as regression guardrail - --no-embedding opt-in flag (D9) for deferred-setup mode - Subagent-Anthropic caveat (D7) fires post-init when chat_model is non-Anthropic AND ANTHROPIC_API_KEY missing Exported groupReadyByProvider() + findEnvKeyTypos() for unit testing. Tests: 21 unit cases covering provider grouping + typo detection edge cases. * feat(embed,import): refuse cleanly when --no-embedding deferred-setup is active T7 of the v0.37.10.0 wave. Both runEmbedCore and runImport now call assertEmbeddingEnabled(loadConfig()) at entry. When the brain was init'd with --no-embedding (config has embedding_disabled:true), they exit 1 with a paste-ready hint: gbrain config set embedding_model <provider>:<model> gbrain config set embedding_dimensions <N> gbrain init --force --embedding-model <provider>:<model> \`gbrain import --no-embed\` flag still works (chunks land without vectors), so users can still ingest in deferred-setup mode and backfill embeddings later with \`gbrain embed --stale\`. * feat(doctor): empty-config drift detection + subagent-Anthropic caveat extension Two doctor check extensions for v0.37.10.0: T9 — embedding_provider check extended for the v0.36 silent-default repair case. When config is empty AND schema column dim differs from the gateway-resolved default, surface the mismatch with empty-brain vs non-empty-brain repair branching (codex finding #7 nuance): - Empty brain (0 embedded chunks) → \`gbrain init --force --pglite --embedding-model <id> --embedding-dimensions <N>\` (drop and re-init) - Non-empty brain → \`gbrain retrieval-upgrade --to <id> --reindex\` Gated on totalChunks > 0 so pristine empty brains aren't pre-warned. Never recommend rm -rf ~/.gbrain. T10 — subagent_provider check (v0.31.12) extended per D7. When chat_model is non-Anthropic AND ANTHROPIC_API_KEY is missing, warn that subagent features (gbrain dream, gbrain agent run, gbrain autopilot) will fail at job submission. Chat alone (gbrain think) still works. * feat(reindex,test): multimodal preflight + E2E suite for fresh PGLite init T11 — reindex-multimodal.ts: hook resolveSchemaMultimodalDim() preflight BEFORE the reindex sweep. Mirrors the text-side contract from initPGLite — if the configured multimodal model can't produce a dim matching the schema column, fail loud here with a \`gbrain config set\` hint rather than mid-reindex with a vector(N) INSERT error. T12 — test/e2e/init-fresh-pglite.test.ts (NEW, 14 cases): subprocess-driven E2E verification of the bug-reporter's repro scenarios: - Happy path: OPENAI_API_KEY set → auto-pick OpenAI, persists config - D3 non-TTY fail-loud (with and without env-key typos) - D6 regression: bug-reporter's three no-op config keys all exit 1 with Levenshtein suggestions - D9 deferred-setup mode + gbrain import refusal (and --no-embed bypass) - D11 preflight refuses BEFORE any disk write - Explicit --embedding-model wins over env detection Each test uses its own throw-away GBRAIN_HOME for hermetic runs. * docs: env-detection + headless-install + close v0.32 picker TODO T13/T14 docs sync for v0.37.10.0: - docs/integrations/embedding-providers.md: TL;DR table refreshed to reflect ZE as v0.36 default; added "Init resolves your provider from env keys" section explaining the auto-pick → picker → fail-loud chain; added "If first import fails" troubleshooting block pointing at gbrain doctor instead of \`rm -rf ~/.gbrain\`. - docs/operations/headless-install.md (NEW): Docker/CI sequencing guide. Two acceptable patterns — provider key at build time (Pattern 1) or --no-embedding opt-in + runtime config (Pattern 2). Codex finding #11. - README.md: Troubleshooting section with one-paragraph repair hint and links to embedding-providers.md + headless-install.md. - TODOS.md: closed v0.32.x "interactive provider chooser" entry as SUPERSEDED by this wave. Added four follow-up entries (dedicated v0.36 broken-install migration, namespaced ext fields, runtime config-key audit, value-level Levenshtein on config set). * chore: bump version and changelog (v0.37.10.0) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(doctor): empty brain scores 100/100 + hermetic doctor-report-remote test Two fixes coupled because the test couldn't pass without the formula fix: src/core/pglite-engine.ts + src/core/postgres-engine.ts — empty brain (pageCount === 0) now gets FULL marks (100/100), not 0/100. Semantically an empty brain has no coverage problem to penalize — there's nothing to embed, nothing to link, nothing to orphan. Vacuous truth applies. The pre-fix "empty = 0" caused fresh-init brains to score as critically unhealthy on \`gbrain doctor\`, which was a structural surprise to users who'd just run init successfully. Same fix on both engines. test/brain-score-breakdown.test.ts — updated the "empty brain" assertion to match the new contract (was: 0/0/0/0/0/0; is: 100/35/25/15/15/10). test/doctor-report-remote.test.ts → renamed to .serial.test.ts and made hermetic. The pre-fix test pulled audit data from the host ~/.gbrain (reranker_health, sync_failures, etc.), which made the assertion non-deterministic depending on whoever ran the suite. Now isolates GBRAIN_HOME to a tempdir via beforeAll/afterAll; env mutation requires serial-quarantine per scripts/check-test-isolation.sh R1. Closes the master-state flake that was failing on every \`bun run test\` run regardless of my branch contents. * docs: update CLAUDE.md and TODOS.md for v0.37.10.0 empty-brain fix - CLAUDE.md: annotate src/core/pglite-engine.ts + src/core/postgres-engine.ts entries with v0.37.10.0 empty-brain 100/100 contract. Vacuous truth: an empty brain has no coverage to penalize, so getBrainScore returns full marks (35/25/15/15/10 breakdown) when pageCount === 0. Pre-fix 0/100 was structurally surprising on fresh init and caused the v0.37.8.0 doctor-report-remote.test.ts flake. - TODOS.md: mark P0 doctor-report-remote.test.ts:65 TODO completed (resolved by commit 9aa571f3's empty-brain-100/100 fix; test renamed to .serial.test.ts and made hermetic per scripts/check-test-isolation.sh R1). - llms-full.txt: regenerated from updated CLAUDE.md per CLAUDE.md "Auto-derived files" rule. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(init): D5 persisted-config-wins on re-init + CI mechanical E2E Two coupled fixes for the v0.37.10.0 wave's interaction with CI's Tier-1 mechanical E2E suite (which runs without any embedding-provider env var). src/commands/init.ts — Honor D5 properly at resolveAIOptions entry. Pre-fix the env-detection branch fired on EVERY init regardless of persisted config. A non-TTY re-init with no env keys exited 1 (D3 fail-loud) even when ~/.gbrain/config.json already had embedding_model set from a prior successful init. Now resolveAIOptions reads loadConfig() first and seeds out.embedding_model / embedding_dimensions / expansion_model / chat_model from the file plane BEFORE running env detection. Also honors embedding_disabled (D9 sentinel) on re-init so deferred-setup brains don't re-trigger fail-loud. test/e2e/mechanical.test.ts:722 — Setup Journey's first init runs against a fresh DB with no persisted config. Pass --embedding-model explicitly (openai:text-embedding-3-large) so the preflight resolves offline. After this init writes config, subsequent inits in the file (RLS self-heal v24, RLS event-trigger probes, etc.) honor the persisted config via the D5 fix above. Verified locally: full test/e2e/mechanical.test.ts → 78 pass / 0 fail. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
|
54a0629745 |
v0.37.8.0 feat: voyage-code-3 discoverability + reindex-code cost-preview fix (#1267)
* v0.37.6.0 feat: voyage-code-3 discoverability + reindex-code cost-preview fix For agents indexing source code with gbrain, the right embedding model is now obvious — and the brain tells you so out loud. Decision tree + Topology 3 doc + Topology 3 setup-skill pointer + runtime stderr nudge from `gbrain reindex --code` against non-code-tuned models. Same diff fixes the stale hardcoded `text-embedding-3-large` in the cost preview that would have made the nudge land badly. Tests: 3 new files (recipe regression, nudge logic + CLI integration, cost-preview IRON-RULE regression). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * todos: file pre-existing doctor-report-remote score regression (#1215 follow-up) Noticed during /ship of v0.37.6.0. The skill_brain_first check added in v0.37.3.0 (#1215) appears to tank the doctor health score on fresh PGLite test brains, causing test/doctor-report-remote.test.ts:65 to fail with health_score: 50 (expects >=70). Pre-existing on master; not in this branch's touch surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
430d784a76 |
v0.37.6.0 feat(ai): OpenRouter recipe + generic default_headers seam (cherry-pick #1210) (#1246)
* feat(ai): add default_headers / resolveDefaultHeaders seam to Recipe Generalizes per-recipe header attachment so attribution headers (OpenRouter's HTTP-Referer + X-OpenRouter-Title) ride alongside Bearer auth on every openai-compatible touchpoint. Two safety guards fire at applyResolveAuth time: declaring both default_headers AND resolveDefaultHeaders throws AIConfigError (mutual exclusion); a default header whose key shadows the resolved auth header (Authorization, the resolver's custom header) also throws. Reranker HTTP path at gateway.ts:2281 now merges both Authorization Bearer AND auth.headers (where default_headers flow) into the request Headers map. Pre-fix the ternary picked one or the other; default_headers would have been silently dropped on the manual rerank path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai): add OpenRouter provider recipe One key, many hosted models. Configures openrouter:<provider>/<model> for chat (GPT-5.2 family, Claude 4.5/4.6/4.7, Gemini 3 Flash Preview, DeepSeek) and embedding (OpenAI text-embedding-3-small with Matryoshka dims_options). max_batch_tokens=300_000 (OpenAI's aggregate per-request token cap, not the per-input 8192 the original PR conflated). resolveDefaultHeaders returns HTTP-Referer + X-OpenRouter-Title + X-Title (back-compat alias) so traffic is attributed to gbrain on OR's leaderboard. Forks override via OPENROUTER_REFERER / OPENROUTER_TITLE env vars. supports_subagent_loop: false is informational — gbrain's subagent infra is hard-pinned to Anthropic-direct via isAnthropicProvider() upstream regardless of this flag. Filed as TODO to verify tool_use_id stability through OR. Cherry-picked from PR #1210. Contributed by @davemorin. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): export buildGatewayConfig + thread OPENROUTER_BASE_URL Exports buildGatewayConfig for unit-test access. Adds one-line passthrough for OPENROUTER_BASE_URL matching the existing LITELLM/OLLAMA/LMSTUDIO/ LLAMA_SERVER pattern so users can point at a self-hosted OR-compatible proxy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(ai): cover OpenRouter recipe + default_headers seam + wire-level headers Four test additions: - test/ai/recipe-openrouter.test.ts (11 cases) — recipe shape, Matryoshka dims_options, max_batch_tokens=300K, arbitrary-ID acceptance via assertTouchpoint, defaultResolveAuth happy/error, resolveDefaultHeaders defaults + fork-override path, setup_hint coverage. Shape regression on every chat/embedding model ID (catches typos without pinning the dynamic catalog). - test/ai/recipes-existing-regression.test.ts (+6 cases) — IRON RULE preserved; adds default_headers contract: Bearer+defaults returns both apiKey AND headers, custom-header+defaults merges with resolver winning, mutual-exclusion guard, Authorization-shadow guard, custom-auth-shadow guard, cross-touchpoint parity for all four (embedding/expansion/chat/ reranker). - test/ai/header-transport.test.ts (3 cases) — proves headers actually reach the wire. Synthetic recipes with resolveOpenAICompatConfig fetch wrappers capture outgoing Headers on embed/chat/rerank. Asserts Authorization + HTTP-Referer + X-OpenRouter-Title + X-Title all present. Codex flagged the return-shape-only coverage gap during plan review. - test/ai/build-gateway-config.test.ts (7 cases) — 5-way env-baseURL passthrough sweep through the now-exported buildGatewayConfig. Uses withEnv() from test/helpers/with-env.ts for isolation compliance. Mops up pre-existing untested drift on LLAMA_SERVER/OLLAMA/LMSTUDIO/LITELLM in the same pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: add OpenRouter to embedding-providers + bump recipe count 15 -> 16 recipes. Adds OpenRouter row to the TL;DR table, a setup section covering the value-prop (one key, many hosted models), env-var overrides (OPENROUTER_BASE_URL, OPENROUTER_REFERER, OPENROUTER_TITLE), the subagent- loop limitation (isAnthropicProvider() gate), and a "One key for many hosted models" bullet under the decision tree. README updated to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump v0.37.2.0 version refs to v0.37.4.0 across in-tree comments v0.37.2.0 was claimed by master's takes_resolution_consistency hotfix (#1211) before this branch could land. This commit re-stamps the source comments that reference the OpenRouter recipe / default_headers seam to v0.37.4.0 so the in-tree version markers match the actual landing version. No behavior change — comments only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.37.4.0) One key, many hosted models — OpenRouter recipe lands. Cherry-picked from #1210 (@davemorin), with codex review corrections folded in: - recipe count math (16 not 17) - current OR attribution header name (X-OpenRouter-Title, X-Title back-compat) - max_batch_tokens semantic (300K aggregate per-request, not 8192 per-input) - Matryoshka dims_options for text-embedding-3-small - auth-shadow guard at applyResolveAuth Adds the generic Recipe.default_headers / resolveDefaultHeaders seam so attribution headers ride alongside Bearer auth. Future Together/Groq adoption tracked in TODOS.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump to v0.37.6.0 (queue moved past v0.37.4/v0.37.5) VERSION + package.json + CHANGELOG header + CLAUDE.md + TODOS.md + in-tree source comments + llms regen. No code-behavior change. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
71ed8d0d21 |
v0.32.0 feat: 5 new embedding recipes + discoverability pass (closes 17-PR cluster) (#810)
* feat(ai/types): add resolveAuth + probe + user_provided_models fields
Foundation commit for the embedding-provider fix-wave (5 API-key recipes
+ discoverability pass). Three optional additions to the recipe contract:
- `EmbeddingTouchpoint.user_provided_models?: true` (D8=A): flag for
recipes that ship without a fixed model list. Consumed by the contract
test (permits empty `models[]`), gateway.ts:223 (replaces hardcoded
`recipe.id === 'litellm'` check in a follow-up commit), and
init.ts:resolveAIOptions (refuses implicit "first model" pick for
shorthand `--model <provider>`).
- `Recipe.resolveAuth?(env): {headerName, token}` (D12=A): unified auth
seam across embed / expansion / chat. Default behavior (returns
`Authorization: Bearer <env-key>`) covers the existing 9 recipes
unchanged. Recipes deviating (Azure with `api-key:`; future OAuth
providers) override this single seam instead of adding parallel
mechanisms in 3 places. Codex review caught that auth was triplicated
at gateway.ts:281/728/931; D12=A unifies all three in one follow-up
commit.
- `Recipe.probe?(): Promise<{ready, hint?}>` (D13=A): recipe-owned
readiness check for local-server providers (ollama, llama-server).
Replaces the hardcoded `recipe.id === 'ollama'` special case in
providers.ts. Wrapped in 200ms timeout at the call sites.
Pure type additions — no behavior change. Typecheck green; existing 9
recipes work unchanged because all three fields are optional.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (decisions
D8=A, D11=C, D12=A, D13=A).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai/gateway): unify openai-compatible auth via Recipe.resolveAuth (D12=A)
Pre-v0.32, openai-compatible auth was duplicated 3 times in gateway.ts at
instantiateEmbedding, instantiateExpansion, instantiateChat — with subtle
drift (embedding had a `${recipe.id.toUpperCase()}_API_KEY` fallback the
other two lacked). Codex outside-voice review caught this during /plan-eng-review.
D12=A: unify all three through `Recipe.resolveAuth?(env)` (declared in the
prior commit). Two new module-level helpers:
- `defaultResolveAuth(recipe, env, touchpoint)` — applied when a recipe
doesn't declare its own resolver. Returns Authorization Bearer with
`auth_env.required[0]`, falling back to the first present
`auth_env.optional` env var, or 'unauthenticated' for no-auth recipes
like Ollama. Throws AIConfigError with the recipe's setup_hint when
required env is missing.
- `applyResolveAuth(recipe, cfg, touchpoint)` — returns
`createOpenAICompatible` options. Bearer-via-Authorization paths use
the SDK's native `apiKey` field; custom-header paths (Azure: api-key)
use `headers` and OMIT apiKey to avoid double-auth leaks.
The 3 `case 'openai-compatible':` branches in instantiateEmbedding (line
~281), instantiateExpansion (line ~728), instantiateChat (line ~931) each
collapse from ~10 lines of bespoke auth handling to a single
`applyResolveAuth(recipe, cfg, '<touchpoint>')` call.
Also: the litellm-template hardcode at gateway.ts:223 (`recipe.id ===
'litellm'`) is replaced with a union check for
`EmbeddingTouchpoint.user_provided_models === true` (D8=A wire-through
per Codex finding #3). Pre-v0.32 builds keep working via back-compat
`recipe.id === 'litellm'` clause; new recipes declaring
user_provided_models pick up the same gating automatically.
Existing 9 recipes (openai, anthropic, google, deepseek, groq, ollama,
litellm-proxy, together, voyage) gain zero per-recipe edits — the
default resolver covers their existing behavior. Behavior change for
ollama expansion/chat only: now reads OLLAMA_API_KEY when set (pre-v0.32
silently passed 'unauthenticated' for those touchpoints; embedding
already read it). Ollama servers ignore the header so no real-world
impact; this aligns the 3 touchpoints.
Tests: bun test test/ai/ — 77/77 pass.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (D8=A,
D12=A; addresses Codex findings #3, #4).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai): IRON RULE regression test for v0.32 resolveAuth refactor
Pins the contract that the v0.32 D2/D12=A resolveAuth refactor preserves
auth behavior for the 9 existing recipes (openai, anthropic, google,
deepseek, groq, ollama, litellm-proxy, together, voyage).
10 cases covering:
- the 9 expected recipe ids are still registered
- every recipe with non-empty required[] returns Authorization Bearer <key>
- missing required env throws AIConfigError naming recipe + touchpoint + env-var
- Ollama (empty required, optional set) reads first present optional env
- Ollama (no env) falls back to "Bearer unauthenticated"
- all 3 touchpoints (embedding/expansion/chat) produce identical auth
shape for the same recipe + env (this is the core regression: pre-v0.32,
embedding had a fallback the other two lacked)
- applyResolveAuth converts Authorization Bearer to {apiKey} (SDK-native)
- applyResolveAuth respects a custom-header override (Azure preview; the
recipe ships in commit 8) and emits {headers} WITHOUT apiKey to avoid
double-auth
- native-* recipes (openai, anthropic, google) intentionally have no
resolveAuth declared (they use AI-SDK adapters directly)
- all openai-compatible recipes ship without resolveAuth in v0.32 (default
applies); the first override is Azure in commit 8
Also: export `defaultResolveAuth` and `applyResolveAuth` as @internal
gateway helpers so tests can pin them directly. Mirrors the pattern of
`splitByTokenBudget` and `isTokenLimitError` already exported with the
same @internal annotation.
Tests: bun test test/ai/ — 87/87 pass (10 new + 77 existing).
Typecheck: clean.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (IRON RULE
per Section 3 test review).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add llama-server recipe (#702 reworked)
10th recipe in the registry; first to ship Recipe.probe (D13=A) and the
second user_provided_models recipe (litellm-proxy is the first).
llama.cpp's llama-server exposes an OpenAI-compatible /v1/embeddings
endpoint. Distinct from Ollama: different default port (8080), different
model-management story (you launch it with --model <path>; the server
serves whatever was passed). Recipe ships with `models: []`,
`user_provided_models: true`, `default_dims: 0` so the wizard refuses
implicit defaults and forces explicit --embedding-model + --embedding-dimensions.
Added:
- src/core/ai/recipes/llama-server.ts (61 lines)
- probeLlamaServer() in src/core/ai/probes.ts; reads
LLAMA_SERVER_BASE_URL with default http://localhost:8080/v1
- Registered in src/core/ai/recipes/index.ts (10 recipes total now)
- test/ai/recipe-llama-server.test.ts (8 cases): registered + shape,
user_provided_models flag, probe declared + reachability fail-with-hint,
default-auth covering no-env / API_KEY / URL-shaped-only paths
Hardening: defaultResolveAuth in gateway.ts now skips URL-shaped optional
env entries (names ending in _URL or _BASE_URL) when picking a fallback
auth token. Pre-fix, OLLAMA_BASE_URL=http://my-ollama would have become
the Bearer token; Ollama ignores it but llama-server (and future
local-server recipes) shouldn't depend on the server tolerating garbage
auth. The regression test (recipes-existing-regression) gains one case
pinning this contract.
Per-recipe test file follows D7=B (per-recipe over DRY for readability).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 4
of 11). Reworked from #702 because the original PR didn't model the
recipe-owned probe pattern (D13=A) or user_provided_models (D8=A).
Tests: bun test test/ai/ — 95/95 pass (8 new + 87 existing).
Co-Authored-By: SiyaoZheng <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add MiniMax recipe (#148 reworked)
11th recipe. embo-01 model, 1536 dims, $0.07/1M tokens.
OpenAI-compatible at api.minimax.chat. MiniMax requires a `type:
'db' | 'query'` field for asymmetric retrieval (documents indexed with
type='db', queries embedded with type='query'). gbrain has no
query/document signal at the embed-call site today, so v1 defaults to
type='db' for both indexing and retrieval — same vector space, symmetric
similarity. Asymmetric query support is a follow-up TODO that needs the
embed seam to thread query/document context.
Plumbed via src/core/ai/dims.ts: dimsProviderOptions returns
{openaiCompatible: {type: 'db'}} for modelId === 'embo-01'.
Conservative max_batch_tokens=4096 declared (MiniMax docs don't publish
the limit). Recursive halving in the gateway catches token-limit errors
at runtime.
Tests: bun test test/ai/ — 101/101 (6 new + 95 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 5
of 11). Reworked from #148.
Co-Authored-By: cacity <20351699+cacity@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add Alibaba DashScope recipe (#59 split, part 1/2)
12th recipe. text-embedding-v3 (current) + text-embedding-v2; 1024
default dims with Matryoshka options [64, 128, 256, 512, 768, 1024].
OpenAI-compatible at dashscope-intl.aliyuncs.com. China-region users
override via cfg.base_urls['dashscope']; v0.32 ships with the
international default.
Conservative max_batch_tokens=8192 + chars_per_token=2 declared because
Alibaba doesn't publish a hard batch limit and text-embedding-v3 mixes
English + CJK heavily (CJK density closer to Voyage than OpenAI tiktoken).
Tests: bun test test/ai/ — 106/106 (5 new + 101 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 6
of 11). Reworked from #59 (DashScope+Zhipu split into 2 commits per
the plan; Zhipu lands next).
Co-Authored-By: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add Zhipu AI (BigModel) recipe (#59 split, part 2/2)
13th recipe. embedding-3 (current) + embedding-2; 1024 default dims
with Matryoshka options [256, 512, 1024, 2048].
OpenAI-compatible at open.bigmodel.cn. embedding-3 at 2048 dims exceeds
pgvector's HNSW cap of 2000 — those brains fall back to exact vector
scans via the existing chunkEmbeddingIndexSql policy at
src/core/vector-index.ts. Default stays at 1024 (HNSW-fast); users who
want maximum fidelity opt into 2048 via --embedding-dimensions and
accept the slower retrieval.
Tests pin the HNSW boundary: 1024 returns the index SQL, 2048 returns
the skip-index/exact-scan SQL.
Tests: bun test test/ai/ — 112/112 (6 new + 106 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 7
of 11). Reworked from #59. Together with DashScope (commit 6), closes
the China-region embedding gap users repeatedly reported (DashScope
covers Alibaba, Zhipu covers BigModel; both ship with international
endpoints by default).
Co-Authored-By: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add Azure OpenAI recipe (#459 reworked)
14th recipe and the first to exercise both v0.32 architectural seams:
- resolveAuth (D12=A) returns `{headerName: 'api-key', token: <key>}`
instead of the default Authorization Bearer. Azure rejects double-auth,
so applyResolveAuth puts the key in `headers` and OMITS apiKey.
- A new `Recipe.resolveOpenAICompatConfig?(env)` seam (Recipe.ts) lets
the recipe template the baseURL from env (Azure: ENDPOINT + DEPLOYMENT
combine into a non-/v1 path) and inject a custom fetch wrapper that
splices ?api-version= onto every request URL.
The fetch wrapper is type-safe via `as unknown as typeof fetch`; AI SDK
never calls TS's strict `preconnect()` method on the wrapper so the cast
is sound. `applyOpenAICompatConfig` (new gateway helper) routes through
the recipe override or falls back to the pre-v0.32 base_urls/base_url_default
behavior — existing 13 recipes get zero behavior change.
API version defaults to `2024-10-21` (current stable as of 2026-05);
override via AZURE_OPENAI_API_VERSION env. Endpoint trailing slash gets
stripped during URL construction so users can copy-paste from the Azure
portal.
Tests (12 cases in test/ai/recipe-azure-openai.test.ts):
- resolveAuth returns api-key NOT Authorization Bearer
- applyResolveAuth puts key in headers, NOT apiKey (no double-auth)
- baseURL templating from endpoint + deployment, with trailing-slash strip
- AIConfigError on missing endpoint OR deployment
- fetch wrapper splices api-version (default + AZURE_OPENAI_API_VERSION override)
- fetch wrapper does NOT double-add api-version when caller already set it
- applyOpenAICompatConfig honors recipe override
IRON RULE regression test updated: now asserts azure-openai is the
documented exception that overrides resolveAuth; any future override
needs review.
Tests: bun test test/ai/ — 124/124 (12 new + 112 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 8
of 11, plus the resolveOpenAICompatConfig seam discovered during fold-in).
Reworked from #459. The original PR proposed a hardcoded AzureOpenAI
client switch; this implementation routes through the unified seams so
future Azure-shaped providers (other custom-URL services) can reuse them.
Co-Authored-By: JamesJZhang <32652444+JamesJZhang@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): adjacent fixes — no_batch_cap (#779) + config-key fallbacks (#121)
Two small ergonomics fixes folded together (#765 deferred — see TODOS.md
follow-up; the CJK PGLite extraction was bigger than the plan estimated).
#779 reworked (alexandreroumieu-codeapprentice): silence the
missing-max_batch_tokens startup warning for recipes with genuinely
dynamic batch capacity. New `EmbeddingTouchpoint.no_batch_cap?: true`
field. Set on ollama (capacity depends on locally loaded model +
OLLAMA_NUM_PARALLEL), litellm-proxy (depends on backend), llama-server
(set by --ctx-size at server launch). Three less stderr warnings on
every gateway configure; google still warns (it's a real fixed-cap
provider that ought to ship a max_batch_tokens declaration).
Bonus: litellm-proxy now declares `user_provided_models: true`, removing
the last consumer of the legacy `recipe.id === 'litellm'` hardcode in
gateway.ts:223 (D8=A wire-through completion).
#121 reworked (vinsew): self-contained API keys. Two parts:
1. config.ts: ANTHROPIC_API_KEY env merge was silently missing.
loadConfig() merged OPENAI_API_KEY but not ANTHROPIC_API_KEY into
the file-config-shape result. One-line addition.
2. cli.ts:buildGatewayConfig: when ~/.gbrain/config.json declares
openai_api_key / anthropic_api_key but the process env doesn't
have those env vars set (common for launchd-spawned daemons,
agent subprocess tools, containers that don't propagate
~/.zshrc), fold the config-file values into the gateway env
snapshot. Process env still wins (loaded last) so per-process
overrides keep working.
Tests (4 cases in test/ai/no-batch-cap-suppression.test.ts):
- Ollama / LiteLLM / llama-server all declare no_batch_cap: true
- configureGateway does NOT warn for those three
- configureGateway STILL warns for google (regression guard)
- Cross-cutting invariant: empty-models recipes declare user_provided_models
Tests: bun test test/ai/ — 128/128 (4 new + 124 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 9 of 11).
#765 (Hunyuan PGLite + CJK keyword fallback) deferred to TODOS.md
follow-up; the CJK extraction (~150 lines + scoring logic + tests) is
larger than the wave's adjacent-fix lane should carry. Closes that PR
with a deferral note.
Co-Authored-By: alexandreroumieu-codeapprentice <noreply@github.com>
Co-Authored-By: vinsew <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(discoverability): doctor alt-provider advisory + init user_provided_models refusal
Two small but high-leverage changes that address the discoverability
problem the v0.32 wave is trying to fix.
src/commands/doctor.ts: new `alternative_providers` check (8c). After
the existing embedding-provider smoke test, walks listRecipes() and
surfaces any recipe whose required env vars are ALL present in the
process env but is not the currently configured provider. Reports as
status: 'ok' with an informational message — never errors. Helps users
discover that, e.g., `OPENAI_API_KEY=x DASHSCOPE_API_KEY=y` configured
for openai means they have a Chinese-region alternative ready without
extra setup.
src/commands/init.ts: user_provided_models recipes (litellm, llama-server)
now refuse the implicit "first model" pick from shorthand --model with
a structured setup hint pointing the user at the explicit form
`--embedding-model <provider>:<your-model-id> --embedding-dimensions <N>`.
Pre-fix, shorthand --model litellm threw "no embedding models listed"
which was technically correct but unhelpful. The new error includes the
recipe's setup_hint when available.
Tests: bun test test/ai/ — 128/128 pass; typecheck clean.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 10
of 11). The full interactive provider chooser in init.ts (the bigger
piece of the discoverability lane) is deferred to a v0.32.x follow-up;
this commit ships the doctor advisory + cleaner refusal that close the
80% case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(v0.32.0): embedding-providers.md + README callout + CHANGELOG + TODOS.md
Final commit of the v0.32 wave. Closes the discoverability gap that
generated the 17-PR community cluster.
- New docs/integrations/embedding-providers.md: capability matrix, decision
tree, per-recipe one-pagers, OAuth provider notes, "my provider isn't
listed" pointer to LiteLLM proxy. Voice: capability not marketing per
CLAUDE.md voice rules.
- README.md: embedding-providers callout near the top, naming the count
(14 recipes) and pointing at the new doc.
- CHANGELOG.md: v0.32.0 entry following the verdict-headline format from
CLAUDE.md voice rules. Lead-with-numbers ("14 providers, 5 new"), what-this-
means-for-users closer, "to take advantage" upgrade block, itemized
changes, contributor credits, deferred-with-context list.
- VERSION + package.json: 0.31.1 → 0.32.0. Minor bump justified by the
new public Recipe surface (resolveAuth, resolveOpenAICompatConfig, probe,
user_provided_models, no_batch_cap fields), the new OAuth subsystem
scaffold (deferred to v0.32.x but typed in v0.32.0), and the 5 new
recipes.
- TODOS.md: 7 follow-up entries for the v0.32 wave's deferred work
(Vertex ADC, Copilot OAuth, Codex OAuth, CJK PGLite, interactive
wizard, real-credentials CI matrix, MiniMax asymmetric retrieval,
multimodal hardcode un-stuck). Each entry has full context + the
exact file paths + the spike work needed so a future contributor can
pick up cleanly.
Tests: bun test test/ai/ — 128/128 pass; typecheck clean.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 11
of 11). Wave complete: 11 commits, ~1500 net lines, 5 new recipes, full
docs, doctor advisory, IRON RULE regression test, 7 TODOS for the
v0.32.x follow-up wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: regenerate llms.txt + llms-full.txt for v0.32.0
After commit
|
||
|
|
4fc1246606 |
v0.24.0: production-hardening pass on the skillify loop (#387)
* feat: v0.19.0 — skillify loop + AGENTS.md compat + brain-first convention This is the v0.19.0 release. The branch ships four new CLI commands, a refactor to check-resolvable, and an expansion of the brain-first convention for sub-agent tool discovery. The original commit message described only the convention expansion, undercounting the scope by ~5x; this amend captures the full release. NEW COMMANDS - gbrain skillify scaffold <name> — 4 stub files + idempotent resolver row - gbrain skillify check [path] — 10-item post-task audit (promoted) - gbrain skillpack list / install — curated 25-skill bundle, atomic install - gbrain skillpack diff <name> — per-file diff preview - gbrain routing-eval — dedicated CI verb for Check 5 fixtures CHECK-RESOLVABLE REFACTOR - Accepts AGENTS.md as a resolver file alongside RESOLVER.md, at either the skills directory or one level up (workspace root layout). - Auto-derives the skill manifest by walking skills/*/SKILL.md when manifest.json is missing. - Splits ResolvableReport into errors[] + warnings[] so advisory checks (filing audit, routing gaps, DRY violations) don't break CI by default. - New --strict opt-in flag promotes warnings to exit 1. BRAIN-FIRST CONVENTION - skills/conventions/brain-first.md expanded from 5-step lookup guide to full sub-agent reference: tool inventory, lookup chain, score thresholds, authority hierarchy, sync rules, entity page conventions, sub-agent propagation rule. PRODUCTION-READINESS HARDENING (this branch's review pass) - routing-eval --llm: emits stderr placeholder notice + runs structural layer only. README, CHANGELOG, CLI help all rewritten consistently. Was a silent no-op against documented contract. - skillpack installer: receipt comment in fence (cumulative-slugs="...") preserves single-skill-install accumulation while letting install --all prune removed bundle skills cleanly. Unknown rows preserved + stderr warning for the operating agent. Pre-v0.19 fences upgrade silently. - skillify scaffold: resolver-row regex broadened to detect backticked, quoted, and bare path forms. No duplicate row on --force after the user normalizes formatting. - scripts/check-privacy.sh: now wired into package.json test chain so the wintermute-ban rule is actually enforced. New regression test. - E2E Tier 2 (LLM skills) promoted from schedule-only to required per-PR CI. Local Tier 1 + Tier 2 verified clean. - Stale v0.17/v0.18 version labels rewritten across new files. TESTS - test/routing-eval-cli.test.ts: 4 cases covering --llm warn semantics - test/privacy-script-wired.test.ts: regression guard for CI wiring - test/skillpack-install.test.ts: 4 new cases for receipt + cumulative + unknown-row preserve+warn + pre-v0.19 upgrade path - test/skillify-scaffold.test.ts: 4 new cases for broadened regex VERIFICATION - bun test: 2237 pass / 18 known PGLite-contention flakes (CI green; documented as P3 dev-experience in TODOS.md) - bun run typecheck: clean - bun run test:e2e: 18/19 files green (1 pre-existing flake on master, not caused by this branch — verified via git stash) - llms.txt + llms-full.txt regenerated to match README + CHANGELOG Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: scrub banned fork name from public artifacts The privacy guard wired into the test chain in this branch caught 5 pre-existing references to the banned OpenClaw fork name in CHANGELOG.md (2x), skills/migrations/v0.19.0.md (1x), src/cli.ts (1x), and src/commands/sync.ts (1x). All originated in master's v0.19.0 release notes and migration doc when the privacy script existed but wasn't wired into CI yet. Replacements per CLAUDE.md privacy mapping: - Origin-story copy (CHANGELOG layer narratives, code comments naming the production deployment that drove the feature) → "Garry's OpenClaw" - Reader-facing migration step → "your OpenClaw" No code semantics changed. Comments + headings only. Verification: scripts/check-privacy.sh exits 0, full CI guard chain green (privacy + jsonb + progress + wasm + typecheck). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump VERSION to 0.24.0 + new CHANGELOG entry Bump branch version above master's v0.21.0 per CLAUDE.md "CHANGELOG + VERSION are branch-scoped" rule. The new v0.24.0 entry at the top of CHANGELOG covers what THIS branch adds vs master: - routing-eval --llm honesty pass (4-surface contract drift fix) - skillpack installer cumulative-receipt + unknown-row preserve+warn (the Codex-caught regression that would have shipped in master if the original v0.19.0 had landed without this branch's review pass) - skillify scaffold resolver-row regex broadening (backtick + quoted + bare forms; idempotency contract preserved under hand-editing) - 5 banned-name leaks scrubbed from public artifacts - check-privacy.sh wired into CI test chain + regression guard test - 7 stale v0.17/v0.18 version labels rewritten across 5 files - Tier 2 (LLM-skills E2E) promoted from schedule-only to required per-PR VERSION 0.21.0 → 0.24.0 package.json version field synced. llms.txt + llms-full.txt regenerated (no content drift; sizes match). Test suite: 62/62 green across the 5 test files this branch added or extended (routing-eval-cli, privacy-script-wired, skillpack-install, skillify-scaffold, build-llms). CI guards: privacy + jsonb + progress + wasm + typecheck all clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.24.0 Auto-discovered drift via /document-release after the v0.24.0 hardening pass landed. All factual corrections clearly warranted by the diff. CLAUDE.md: - Skillpack installer: documented the cumulative-slugs receipt comment, install --all prune semantics, unknown-row preserve+warn behavior, and pre-v0.24 silent upgrade. Was previously vague about "tracks a skill manifest so install --update diffs cleanly" without explaining what the receipt is or why it matters. - routing-eval: replaced the false claim that --llm "opts into a Haiku tie-break layer for CI." Now correctly describes the placeholder semantic landed in v0.24.0 (stderr notice + structural-only run). README.md: - Skillpack section: added one paragraph on the receipt comment + the user-visible stderr message for hand-added rows. Connects the safe rerun promise to the v0.24.0 implementation that actually enforces it. CONTRIBUTING.md: - Running tests section: now recommends `bun run test` (full CI guard chain + typecheck + tests) before pushing. Names each guard so new contributors understand what catches what. The privacy guard (newly wired in v0.24.0) is one of these — without `bun run test` you'd skip it locally and find out from CI. llms-full.txt: regenerated to reflect CLAUDE.md changes. Verification: full guard chain green locally (privacy + jsonb + progress + wasm + typecheck). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Garry Tan <garry@ycombinator.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
891c28b582 |
v0.22.4 feat: frontmatter-guard — 0 resolver warnings + validate/audit/install-hook CLI (#448)
* fix: resolve check-resolvable warnings on master
- skills/maintain/SKILL.md: drop "citation audit" trigger; the focused
citation-fixer skill is the single owner. Silences the MECE overlap
warning surfaced by src/core/check-resolvable.ts.
- skills/RESOLVER.md: add citation-audit disambiguation row pointing
citation-fixer (focused fix) and chain-into maintain for broader audit.
Broaden query triggers ("who is", "background on", "notes on") so
the failing routing-eval fixtures resolve.
- skills/enrich/SKILL.md: replace inlined Citation Requirements block with
backtick-wrapped `skills/conventions/quality.md` reference (the format
extractDelegationTargets recognizes). Silences the dry_violation warning.
- skills/citation-fixer/routing-eval.jsonl: rewrite the two failing fixtures
to embed "fix citations" verbatim so the substring matcher passes.
- skills/query/SKILL.md frontmatter: mirror the broadened RESOLVER.md
triggers so the trigger round-trip test passes.
Result: gbrain check-resolvable reports 0 warnings, 0 errors against
the actual checked-in skills/ tree.
* feat: extend parseMarkdown + lint with frontmatter validation surface
Add an opt-in validation surface to parseMarkdown(): when called with
{ validate: true }, returns errors[] populated with seven canonical
ParseValidationError codes:
MISSING_OPEN, MISSING_CLOSE, YAML_PARSE, SLUG_MISMATCH,
NULL_BYTES, NESTED_QUOTES, EMPTY_FRONTMATTER
Existing callers are unaffected — validation is opt-in via the new
opts argument. The validation logic lives here as the single source of
truth for what counts as malformed brain-page frontmatter.
src/commands/lint.ts now consumes parseMarkdown(..., { validate: true })
and emits stable lint rule names (frontmatter-missing-close,
frontmatter-yaml-parse, frontmatter-null-bytes, frontmatter-nested-quotes,
frontmatter-slug-mismatch, frontmatter-empty). MISSING_OPEN is suppressed
to avoid double-reporting with the legacy no-frontmatter rule.
Tests: test/markdown-validation.test.ts (NEW, all 7 codes) +
test/lint-frontmatter.test.ts (NEW, lint integration + suppression).
* feat: add brain-writer.ts orchestrator (scan / autoFix / writeBrainPage)
Thin orchestrator (~280 lines) on top of parseMarkdown(..., {validate:true})
and isSyncable() (the canonical brain-page filter from src/core/sync.ts).
Three consumers call into this module: the gbrain frontmatter CLI, the
frontmatter_integrity doctor subcheck, and the v0.22.4 migration audit
phase. Single source of truth — no parallel validation stack.
Public API:
- autoFixFrontmatter(content, opts?): { content, fixes }
Mechanical auto-repair for the fixable subset (NULL_BYTES,
MISSING_CLOSE, NESTED_QUOTES, SLUG_MISMATCH). Idempotent.
- writeBrainPage(filePath, content, opts): path-guarded, .bak backup
before any in-place mutation. Path guard refuses writes outside
sourcePath. .bak is the safety contract for non-git brain repos.
- scanBrainSources(engine, opts?): walks every registered source via
direct SQL on sources.local_path, uses isSyncable() to filter,
blocks symlinks (matches sync's no-symlink policy), respects
AbortSignal.
The dirty-tree guard from src/core/dry-fix.ts:getWorkingTreeStatus() is
NOT used here — it rejects non-git repos as unsafe, but brain repos
aren't always git repos. .bak backups are the contract that works
universally.
Tests: test/brain-writer.test.ts (NEW, 16 cases) — autoFix idempotency,
path-guard reject, .bak backup, per-source rollup, AbortSignal mid-scan,
single-source filter, missing-source-path graceful skip, symlink no-loop.
* feat: gbrain frontmatter CLI (validate / audit / install-hook)
New top-level command surface for the frontmatter-guard feature:
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
Validate one .md file or recursively scan a directory. --fix writes
.bak then rewrites in place. No git-tree-clean guard — .bak is the
safety contract (works for both git and non-git brain repos).
gbrain frontmatter audit [--source <id>] [--json]
Read-only scan via scanBrainSources(). Per-source rollup grouped by
error code. --fix is intentionally NOT available here; use validate
--fix on the source path to repair.
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
Drops a pre-commit hook in each source that's a git repo (skips
non-git sources with a one-line note). Hook script gracefully
degrades when gbrain is missing on PATH (prints a warning, exits 0).
Refuses to clobber existing hooks without --force; writes <hook>.bak.
--uninstall reverses cleanly.
src/cli.ts wires frontmatter through handleCliOnly so --help works
without a DB connection. The audit subcommand instantiates an engine
internally only when needed.
Tests: test/frontmatter-cli.test.ts (NEW, 9 cases) +
test/frontmatter-install-hook.test.ts (NEW, 6 cases) — --help no-DB,
clean/broken validate, --fix dry-run, --fix non-git, --json envelope,
recursive directory scan with isSyncable filter parity, hook install
+ overwrite-protection + --force + --uninstall + silent-refresh.
* feat: doctor frontmatter_integrity subcheck
Adds a frontmatter_integrity subcheck under gbrain doctor that calls
scanBrainSources() (the same shared scanner the CLI and migration use).
Reports per-source counts grouped by error code, with a fix hint
pointing at `gbrain frontmatter validate <path> --fix`. Wrapped in
a doctor progress phase with heartbeat so 50K-page brain scans stay
visible.
Tests: test/doctor.test.ts (UPDATE) — assertion that the subcheck
calls scanBrainSources and the fix hint references the correct CLI.
* feat: frontmatter-guard skill (registered in manifest + RESOLVER)
New skill at skills/frontmatter-guard/SKILL.md that wraps the gbrain
frontmatter CLI for agent-driven workflows. Agent-agnostic — no
references to private host libraries. Registered in skills/manifest.json
and skills/RESOLVER.md (the trigger row was added in the Part A commit).
Triggers: "validate frontmatter", "check frontmatter", "fix frontmatter",
"frontmatter audit", "brain lint".
Includes routing-eval fixtures that pass the substring matcher. The
SKILL.md has the conformance-required Output Format and Anti-Patterns
sections. Anti-patterns explicitly call out: don't auto-fix MISSING_OPEN
or EMPTY_FRONTMATTER without user input, don't skip .bak backups, don't
install the pre-commit hook on non-git brain dirs.
* feat: v0.22.4 migration orchestrator (audit-only, source-aware)
Adds the v0.22.4 migration that surveys every registered source for
frontmatter issues and queues per-source repair commands without ever
mutating brain content. Three idempotent phases:
- schema: no-op (no DB changes in v0.22.4)
- audit: scanBrainSources() across ALL registered sources; writes
JSON report to ~/.gbrain/migrations/v0.22.4-audit.json
- emit-todo: appends one entry per source-with-issues to
~/.gbrain/migrations/pending-host-work.jsonl, each with the exact
`gbrain frontmatter validate <source-path> --fix` command
The agent reads skills/migrations/v0.22.4.md after upgrade, surfaces
the report counts to the user, and runs the fix command only with
explicit consent. `apply-migrations --yes` never silently rewrites
brain pages.
Filename convention: TS orchestrator at v0_22_4.ts (underscores, since
TS module paths can't have dots); user-facing migration doc at
skills/migrations/v0.22.4.md (dotted, matches existing convention).
The pending-host-work.jsonl skill field references the dotted-path doc.
Skips cleanly when no sources are registered (fresh install).
Tests: test/migrations-v0_22_4.test.ts (NEW, 9 cases) + updated
test/migration-orchestrator-v0_21_0.test.ts to allow v0.22.4 after,
test/apply-migrations.test.ts skippedFuture arrays extended to include
v0.22.4, test/check-resolvable.test.ts regression guard asserting the
actual checked-in skills/ tree has 0 warnings + 0 errors.
* docs: pre-commit recipe + downstream agent upgrade notes for v0.22.4
- docs/integrations/pre-commit.md (NEW): recipe doc covering install,
bypass (`git commit --no-verify`), uninstall, and downstream-fork
integration notes. Includes the full pipeline diagram showing how
the hook (write-time gate), doctor (audit gate), and CLI (fix tool)
share parseMarkdown(..., {validate:true}) as the single source of
truth.
- docs/UPGRADING_DOWNSTREAM_AGENTS.md: append v0.22.4 section with the
diff pattern for forks that had inline frontmatter validators. Covers
the five upgrade actions: replace ad-hoc validators, drop
lib/brain-writer.mjs references (it never shipped), wire the doctor
subcheck into custom health pipelines, optionally install the
pre-commit hook on git-backed brain repos, and walk
pending-host-work.jsonl after apply-migrations.
- llms.txt + llms-full.txt: regenerated from build:llms script after
the new docs landed.
* chore: bump version and changelog (v0.22.4)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: handle null loadConfig() return in frontmatter + migration paths
CI typecheck caught three call sites that passed loadConfig()'s
GBrainConfig | null result straight into toEngineConfig() (which
expects GBrainConfig, not null):
- src/commands/frontmatter.ts:64 (audit subcommand connect)
- src/commands/frontmatter-install-hook.ts:86 (install-hook connect)
- src/commands/migrations/v0_22_4.ts:59 (audit phase connect)
The frontmatter CLI and install-hook paths follow the existing
src/commands/repair-jsonb.ts pattern: throw 'No brain configured. Run:
gbrain init' so users get an actionable message instead of a TS-shaped
runtime crash.
The v0.22.4 migration audit phase takes a different shape: a fresh
install or test environment running apply-migrations shouldn't fail
hard just because there's no brain to scan yet. Return a clean
'skipped: no_brain_configured' phase result so the orchestrator
continues normally and the ledger records a complete (skipped) run.
* test: add v0.22.4 migration E2E + injection point for testability
Closes plan item B14 (the E2E that was promised but not delivered before
the original ship). Runs the v0_22_4 orchestrator end-to-end on PGLite
against a fixture brain with two registered sources and synthetic
malformed pages on disk. Asserts:
- audit phase writes ~/.gbrain/migrations/v0.22.4-audit.json with
per-source counts (NESTED_QUOTES + NULL_BYTES on alpha,
NESTED_QUOTES on beta)
- emit-todo phase appends one entry per source-with-issues to
pending-host-work.jsonl, each pointing at skills/migrations/v0.22.4.md
with the exact `gbrain frontmatter validate <source> --fix` command
- the migration is audit-only — no fixture page is mutated
during apply-migrations (no .bak created, contents byte-identical)
- re-running the orchestrator is idempotent — JSONL stays at 2 lines
Adds a small test-injection point to v0_22_4.ts:
__setTestEngineOverride(engine: BrainEngine | null): void
Mirrors src/commands/repair-jsonb.ts pattern. When set, phaseBAudit
uses the injected engine instead of loadConfig + createEngine. Production
path is unchanged: the override is null by default and the existing
loadConfig logic runs end-to-end. Required because Bun's os.homedir()
does not observe mid-process process.env.HOME mutations, so we can't
redirect loadConfig's config-file lookup via env-var overrides; the
injection point is the only hermetic way to E2E-test the orchestrator
without writing to the user's real ~/.gbrain/config.json.
Test runs unconditionally in CI's Tier 1 (no DATABASE_URL needed,
PGLite in-memory).
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
|
013b348c28 |
v0.12.3: Reliability wave — sync deadlock, search timeout scoping, wikilinks, orphans (#216)
* fix(sync): remove nested transaction that deadlocks > 10 file syncs sync.ts wraps the add/modify loop in engine.transaction(), and each importFromContent inside opens another one. PGLite's _runExclusiveTransaction is a non-reentrant mutex — the second call queues on the mutex the first is holding, and the process hangs forever in ep_poll. Reproduced with a 15-file commit: unpatched hangs, patched runs in 3.4s. Fix drops the outer wrap; per-file atomicity is correct anyway (one file's failure should not roll back the others). (cherry picked from commit |
||
|
|
7bbfc3e36a |
security: fix wave 3 — 9 vulns (file_upload, SSRF, recipe trust, prompt injection) (#174)
* feat(engine): add cap parameter to clampSearchLimit (H6) clampSearchLimit(limit, defaultLimit, cap = MAX_SEARCH_LIMIT) — third arg is a caller-specified cap so operation handlers can enforce limits below MAX_SEARCH_LIMIT. Backward compatible: existing two-arg callers still cap at MAX_SEARCH_LIMIT. This fixes a Codex-caught semantics bug: the prior signature took (limit, defaultLimit) where the second arg was misread as a cap. clampSearchLimit(x, 20) was actually allowing values up to 100, not 20. * feat(integrations): SSRF defense + recipe trust boundary (B1, B2, Fix 2, Fix 4, B3, B4) - B1: split loadAllRecipes into trusted (package-bundled) and untrusted (cwd/recipes, $GBRAIN_RECIPES_DIR) tiers. Only package-bundled recipes get embedded=true. Closes the fake trust boundary that let any cwd-local recipe bypass health-check gates. - B2: hard-block string health_checks for non-embedded recipes (was previously only blocked when isUnsafeHealthCheck regex matched, which the cwd recipe exploit bypassed). Embedded recipes still get the regex defense. - Fix 2: gate command DSL health_checks on isEmbedded. Non-embedded recipes cannot spawnSync. - Fix 4 + B3 + B4: gate http DSL health_checks on isEmbedded; for embedded recipes, validate URLs via new isInternalUrl() before fetch: - Scheme allowlist (http/https only): blocks file:, data:, blob:, ftp:, javascript: - IPv4 range check covering hex/octal/decimal/single-integer bypass forms - IPv6 loopback ::1 + IPv4-mapped ::ffff: (canonicalized hex hextets handled) - Metadata hostnames (AWS, GCP, instance-data) blocked - fetch with redirect: 'manual' + per-hop re-validation up to 3 hops Original PRs #105-109 by @garagon. Wave 3 collector branch reimplemented the fixes after Codex outside-voice review found that PRs #106/#108 alone did not actually gate cwd-local recipes (B1) and that PR #108 missed redirect-following SSRF (B3) and non-http schemes (B4). * feat(file_upload): path/slug/filename validation + remote-caller confinement (Fix 1, B5, H5, M4, Fix 5) - Fix 1 + B5 + H1: validateUploadPath uses realpathSync + path.relative to defeat symlink-parent traversal. lstatSync alone (the original PR #105 approach) only catches final-component symlinks; a symlinked parent dir still followed to /etc/passwd. Now the entire path chain is resolved. - H5: validatePageSlug uses an allowlist regex (alphanumeric + hyphens, slash-separated segments). Closes URL-encoded traversal (%2e%2e%2f), Unicode lookalikes, backslashes, control chars implicitly. - M4: validateFilename allowlist regex. Rejects control chars, backslash, RTL override (\u202E), leading dot/dash. Filename flows into storage_path so this matters for every storage backend. - Fix 5: clamp list_pages and get_ingest_log limits at the operation layer via new clampSearchLimit cap parameter (list_pages caps at 100, get_ingest_log at 50). Internal bulk commands bypass the operation layer and remain uncapped. - New OperationContext.remote flag distinguishes trusted local CLI from untrusted MCP callers. file_upload uses strict cwd confinement when remote=true (default), loose mode when remote=false (CLI). MCP stdio server sets remote=true; cli.ts and handleToolCall (gbrain call) set remote=false. Original PR #105 by @garagon. Issue #139 reported by @Hybirdss. * feat(search): query sanitization + structural prompt boundary (Fix 3, M1, M2, M3) - M1: restructure callHaikuForExpansion to use a system message that declares the user query as untrusted data, plus an XML-tagged <user_query> boundary in the user message. Layered defense with the existing tool_choice constraint (3 layers vs 1). - Fix 3 (regex sanitizer, defense-in-depth): sanitizeQueryForPrompt strips triple-backtick code fences, XML/HTML tags, leading injection prefixes, and caps at 500 chars. Original query is still used for downstream search; only the LLM-facing copy is sanitized. - M2: sanitizeExpansionOutput validates the model's alternative_queries array before it flows into search. Strips control chars, caps length, dedupes case-insensitively, drops empty/non-string items, caps to 2 items. - M3: console.warn on stripped content NEVER logs the query text — privacy-safe debug signal only. Original PR #107 by @garagon. M1/M2/M3 are wave 3 hardening per Codex review. * chore: bump version and changelog (v0.10.2) Security wave 3: 9 vulnerabilities closed across file_upload, recipe trust boundary, SSRF defense, prompt injection, and limit clamping. See CHANGELOG for full details. Contributors: - @garagon (PRs #105-109) - @Hybirdss (Issue #139) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync documentation with v0.10.2 security wave 3 - CLAUDE.md: document OperationContext.remote, new security helpers (validateUploadPath, validatePageSlug, validateFilename, isInternalUrl, parseOctet, hostnameToOctets, isPrivateIpv4, getRecipeDirs, sanitizeQueryForPrompt, sanitizeExpansionOutput), updated clampSearchLimit signature, recipe trust boundary, new test files - docs/integrations/README.md: replace string-form health_check example with typed DSL (string checks now hard-block for non-embedded recipes); add recipe trust boundary subsection - docs/mcp/DEPLOY.md: document file_upload remote-caller cwd confinement, symlink rejection, slug/filename allowlists Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
ce15062694 |
feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39)
* docs: break SKILLPACK into 17 individual guides The 1,281-line SKILLPACK monolith is now 17 individually linkable guides in docs/guides/, organized by category: core patterns, data pipelines, operations, search, and administration. GBRAIN_SKILLPACK.md becomes a structured index with categorized tables linking to each guide. The URL stays stable for backward compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add integration guides, architecture docs, and ethos New documentation directories: - docs/integrations/ — "Getting Data In" landing page, credential gateway, meeting webhooks. Includes recipe format documentation. - docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search) - docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide - docs/designs/ — "Homebrew for Personal AI" 10-star vision document Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add gbrain integrations command + voice-to-brain recipe New CLI command: gbrain integrations (list/show/status/doctor/stats/test) - Standalone command, no database connection needed - Uses gray-matter directly for recipe parsing (not parseMarkdown) - --json flag on every subcommand for agent-parseable output - Bare command shows senses/reflexes dashboard - Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl First recipe: recipes/twilio-voice-brain.md - Phone calls create brain pages via Twilio + OpenAI Realtime - Opinionated defaults: caller screening, brain-first lookup, quiet hours - Outbound call smoke test (GBrain calls the user to prove it works) - Validate-as-you-go credential testing - Twilio signature validation for webhook security Migration file for v0.7.0 with agent-readable changelog. 13 unit tests covering parseRecipe, CLI routing, and recipe validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Getting Data In to README, update CLAUDE.md and manifest README: voice calls in intro bullet list, new "Getting Data In" section with integration table (voice, email, X, calendar) and recipe philosophy. CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/, docs/integrations/, docs/architecture/, docs/ethos/). manifest.json: bump to v0.7.0, add recipes_dir field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: v0.7.0 CHANGELOG, TODOS, VERSION bump CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain, gbrain integrations command, SKILLPACK breakout, and new documentation. TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL, community recipe submission, always-on deployment recipes). VERSION + package.json: bump to 0.7.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite voice recipe with agent instructions and verified links Major improvements to recipes/twilio-voice-brain.md: - Agent preamble: explains WHY sequential execution matters (each step depends on the previous), defines 4 stop points where the agent MUST pause and verify, tells agent to never say "something went wrong" but instead explain the exact error and fix - User actions are now specific: exact URLs for every credential (Twilio console, OpenAI API keys page, ngrok dashboard), what buttons to click, what fields to copy, common failure modes - All URLs verified via web search against current 2026 documentation: Twilio SID/token at twilio.com/console, OpenAI keys at platform.openai.com/api-keys, ngrok token at dashboard.ngrok.com/get-started/your-authtoken - Cost estimate corrected: OpenAI Realtime is $0.06/min input + $0.24/min output (was understated), total ~$20-22/mo for 100 min - Validate-as-you-go: each credential tested immediately with exact curl commands, failure messages explain what went wrong and how to fix - Smoke test flow: tells user exactly what to say, verifies ALL three outputs (messaging notification + brain page + search result) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add "Homebrew for Personal AI" essay (markdown is code) New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution corollary to "Thin Harness, Fat Skills." Argues that markdown skill files are simultaneously documentation, specification, package, and source code. The agent is the package manager. The git repo is the app store. Referenced from SKILLPACK index and CLAUDE.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite agent instructions as command language, promote skills The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide. Every step is an imperative command with exact verification criteria and explicit stop-on-failure behavior. No FYI, no suggestions, just rails. Key changes: - 11-step setup with STOP points after each step - Exact user instructions for Supabase connection string (what to click, what NOT to give the agent, what the string looks like) - "Verify: run X. You must see Y. If not: Z" after every step - Skills table now links to both skill files AND guide docs - Integration recipes table simplified (no "coming soon" placeholders) - Docs section reorganized: for agents / for humans / reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 4 codex findings + add email-to-brain recipe Codex review found 4 issues, all fixed: 1. getStatus() returned "configured" if ANY secret was set (e.g. just OPENAI_API_KEY). Now requires ALL required secrets before marking configured. Prevents false "configured" status and spurious doctor runs. 2. Twilio health check hit unauthenticated endpoint (always 401). Now uses authenticated curl with SID:token, matching the setup validation. 3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md. 4. Compiled binary can't find recipes/ via import.meta.dir. Added GBRAIN_RECIPES_DIR env var override + global bun install path fallback. Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern with ClawVisor credential gateway, validate-as-you-go, agent instructions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add email, X, calendar, and meeting sync recipes Four new integration recipes extracted from production wintermute patterns: - recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector pattern (code pulls emails with baked-in links, agent does judgment), noise filtering, signature detection, digest generation - recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search, deletion detection (diffs previous run, verifies 404), engagement velocity tracking, rate limit awareness - recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical backfill (years of data), daily markdown files with attendees + locations, attendee enrichment for brain pages - recipes/meeting-sync.md: Circleback API, transcript import with speaker labels, attendee detection + filtering, entity propagation to people/ company pages, action item extraction, idempotent by source_id All recipes follow the same format: agent preamble with sequential execution rules, validate-as-you-go credentials, exact URLs for API key setup, stop-on-failure verification, and heartbeat logging. Updated README, SKILLPACK index, and integrations landing page with all 5 recipes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes Both recipes now offer two auth options: - Option A: ClawVisor (recommended, handles OAuth + token refresh) - Option B: Google OAuth2 directly (no extra service, you manage tokens) Option B includes step-by-step instructions for Google Cloud Console: exact URLs, which buttons to click, which scopes to add, how to enable the API, and the OAuth flow for token exchange. This removes ClawVisor as a hard dependency for getting started. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add implementation guides with pseudocode and test suggestions Every recipe now includes an "Implementation Guide" section with: - Production-tested pseudocode the agent can follow to build each collector - Edge cases and failure modes discovered in real deployment - Non-obvious implementation details (why the 48h staleness heuristic, why Gmail links need authuser, why SSE responses need double-parsing) - Test suggestions: what the agent should verify after setup email-to-brain: noise filtering algorithm, signature detection patterns, Gmail link generation (authuser is critical), sent-mail dedup x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness, API verification), engagement velocity thresholds (50 min for 2x, 100 absolute jump), atomic writes, stdout contract, rate limit handling calendar-to-brain: smart chunking (monthly for sparse years, weekly for dense), attendee filtering (rooms, groups, distros), merge-with-existing (only replace ## Calendar section), date/time parsing edge cases meeting-sync: SSE double-JSON parsing, idempotency double-check (grep + filename), auto-tagging from meeting names, git commit after sync Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: 6 new guides from production patterns (wintermute extraction) New guides extracted and generalized from production deployment: - repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge). Strict boundary rules, decision tree, hard rule: never write knowledge to the agent repo. - sub-agent-routing.md: Model routing table by task type. Signal detector pattern (spawn Sonnet on every message). Research pipeline pattern (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization. - skill-development.md: 5-step cycle (concept, prototype, evaluate, codify, cron). MECE discipline (no overlapping skills). Quality bar checklist. "If you ask twice, it should already be a skill." - idea-capture.md: Originality distribution rating (0-100 across 4 populations). Depth test ("could someone unfamiliar understand WHY?"). Deep cross-linking mandate. Notability filtering. - quiet-hours.md: Hold notifications 11pm-8am local time. Held messages directory pattern. Timezone-aware delivery. Morning briefing pickup. - diligence-ingestion.md: 9-step pipeline for data room materials. Detection patterns (PDF filenames, spreadsheet tabs, user language). Index.md template with bull/bear case. Company page enrichment. All PII scrubbed. Patterns generalized for any user. SKILLPACK index updated with 6 new entries. CLAUDE.md references added. All 37 SKILLPACK links verified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: upgrade all guides to operational playbooks with pseudocode Every guide now follows the playbook structure: - Goal: one sentence, what this achieves - What the User Gets: without this / with this - Implementation: pseudocode with actual gbrain commands - Tricky Spots: production-tested gotchas - How to Verify: test steps the agent runs after setup Guides upgraded (15 files): - brain-agent-loop: on_message() loop with read/write/sync pseudocode - brain-first-lookup: 4-step lookup cascade with exact commands - brain-vs-memory: routing algorithm for 3 knowledge layers - compiled-truth: page structure + rewrite vs append rules - content-media: 3 ingest patterns (YouTube, social, PDFs) - cron-schedule: full schedule table + dream cycle pseudocode - enrichment-pipeline: 7-step protocol with tier classification - entity-detection: spawn pattern + detection prompt + notability filter - executive-assistant: 3 workflow algorithms (triage, prep, post-inbox) - meeting-ingestion: 6-step transcript-to-brain flow - operational-disciplines: 5 executable discipline blocks - originals-folder: detection + exact-phrasing capture + cross-linking - search-modes: decision tree for keyword vs hybrid vs direct - source-attribution: citation format + hierarchy + conflict resolution - Plus Goal/What User Gets headers on 6 newer guides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add WebRTC to voice recipe + ngrok Hobby setup guide Voice recipe updates: - Added WebRTC endpoint (POST /session, GET /call, POST /tool) for browser-based calling with RNNoise noise suppression - WebRTC pseudocode with the 4 non-obvious gotchas from production (voice under audio.output.voice, no turn_detection, no session.update on connect, trigger greeting via data channel) - Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier - Fixed domain means URLs never change, Twilio never breaks New guide: docs/mcp/NGROK_SETUP.md - How to set up ngrok Hobby for both MCP and voice agent - Fixed domain setup, watchdog pattern, AI client configuration - Claude Desktop requires Settings > Integrations (not JSON config) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add dependency graph + ngrok-tunnel + credential-gateway recipes Recipes now have real dependencies via the `requires` field: - voice-to-brain requires ngrok-tunnel (needs public URL for Twilio) - email-to-brain requires credential-gateway (needs Gmail access) - calendar-to-brain requires credential-gateway (needs Calendar access) - x-to-brain and meeting-sync are standalone (direct API keys) Two new infrastructure recipes: - ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby ($8/mo) for a domain that never changes. Includes watchdog pattern. - credential-gateway: secure Google service access via ClawVisor (recommended) or direct OAuth2. One setup, all Google recipes use it. Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure, not MCP-specific. README and integrations landing page show dependency chains. When agent installs voice-to-brain, it sets up ngrok-tunnel first. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add infra category, fix dashboard alignment, show dependencies DX audit found two bugs in gbrain integrations dashboard: 1. Column alignment broken — IDs > 18 chars ran into descriptions with no space. Fixed: pad to 22 chars. 2. ngrok-tunnel and credential-gateway showed as SENSES but they're infrastructure. Added 'infra' category. Dashboard now shows three sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES. 3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)" Also added 'requires' field to JSON output for agent consumption. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add frontier model requirement disclaimer to README GBrain's markdown-is-code approach requires models capable of interpreting intent and implementing from architecture descriptions. Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models will struggle with the recipe format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add PGLite → Supabase upgrade path to README Clarify the database progression: start with PGLite (Postgres as WASM, zero infrastructure, pgvector built in, nothing to install). Graduate to Supabase or self-hosted Postgres when you need connection pooling, concurrency, and remote MCP access from Claude Desktop, Cowork, ChatGPT, Perplexity Computer, or any MCP-compatible agent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: revert PGLite mention (coming in next branch) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify) Every guide now has exactly these sections in this order: - ## Goal (one sentence) - ## What the User Gets (without this / with this) - ## Implementation (pseudocode with gbrain commands) - ## Tricky Spots (3-5 numbered gotchas) - ## How to Verify (3-5 numbered test steps) 11 guides restructured from non-standard headings: - deterministic-collectors, live-sync, upgrades-auto-update (full rewrites) - entity-detection, diligence-ingestion, idea-capture, quiet-hours, repo-architecture, skill-development, sub-agent-routing (restructured) 23/23 guides now pass consistency audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: restructure README around the #1 blocker (getting data in) The README was leading with Postgres and database architecture. Most users are stuck at step zero: "I have an agent but it doesn't know anything about my life." New structure: 1. The Problem — your agent doesn't know your life 2. Getting Data In — integration recipes, front and center 3. The Compounding Thesis — why this matters 4. How this happened — credibility, origin story 5. When you need Postgres — scale, not starting point Postgres is de-emphasized from a full section to two paragraphs: "You don't need Postgres to start" and "When you need Postgres" (1,000+ files, remote MCP access, multiple AI clients). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move Install to top of README, remove duplicate section Install now appears right after Getting Data In (line 38), not buried at line 295. The user sees: Problem → Getting Data In → Install. Removed the duplicate Install section (262 lines) that was lower in the README. The agent instructions block, CLI quickstart, and all content is now in the single Install section near the top. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move agent install block to first thing in README "Start here: paste this into your agent" is now the first section, right after the one-line pitch. No scrolling, no context, no preamble. User opens the README, sees the paste block, copies it into OpenClaw or Hermes, and the agent takes over. Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: compress install block from 11 steps to 5 The agent install block was 102 lines and 11 steps. Now it's 40 lines and 5 steps. Same coverage, half the text. Changes: - Merged "prove keyword search" + "embed" + "prove hybrid search" into one SEARCH step (the user doesn't care about the intermediate) - Merged skillpack, sync, auto-update, integrations, verification into one GO LIVE step with sub-items (post-install polish, not install) - Shortened database instructions (one line instead of 5 sub-steps) - Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now just "Do not skip steps. Verify each step.") The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: gitignore all .env files, not just specific ones CSO audit found .gitignore covered .env.testing and .env.production but not bare .env. A user creating .env with database credentials could accidentally commit it. Fix: .env and .env.* are now gitignored. .env.*.example files are explicitly un-ignored so templates remain tracked. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: scrub PII from essay and recipe examples - 510-MY-GARRY phone mnemonic → "Your Phone Number" - "Garry → Authenticated Mode" → "Owner → Authenticated Mode" - "Telegram" → "secure channel" in auth example - @garrytan → @yourhandle in X recipe example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |