mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 11:22:34 +00:00
* feat(ai/types): add resolveAuth + probe + user_provided_models fields
Foundation commit for the embedding-provider fix-wave (5 API-key recipes
+ discoverability pass). Three optional additions to the recipe contract:
- `EmbeddingTouchpoint.user_provided_models?: true` (D8=A): flag for
recipes that ship without a fixed model list. Consumed by the contract
test (permits empty `models[]`), gateway.ts:223 (replaces hardcoded
`recipe.id === 'litellm'` check in a follow-up commit), and
init.ts:resolveAIOptions (refuses implicit "first model" pick for
shorthand `--model <provider>`).
- `Recipe.resolveAuth?(env): {headerName, token}` (D12=A): unified auth
seam across embed / expansion / chat. Default behavior (returns
`Authorization: Bearer <env-key>`) covers the existing 9 recipes
unchanged. Recipes deviating (Azure with `api-key:`; future OAuth
providers) override this single seam instead of adding parallel
mechanisms in 3 places. Codex review caught that auth was triplicated
at gateway.ts:281/728/931; D12=A unifies all three in one follow-up
commit.
- `Recipe.probe?(): Promise<{ready, hint?}>` (D13=A): recipe-owned
readiness check for local-server providers (ollama, llama-server).
Replaces the hardcoded `recipe.id === 'ollama'` special case in
providers.ts. Wrapped in 200ms timeout at the call sites.
Pure type additions — no behavior change. Typecheck green; existing 9
recipes work unchanged because all three fields are optional.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (decisions
D8=A, D11=C, D12=A, D13=A).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai/gateway): unify openai-compatible auth via Recipe.resolveAuth (D12=A)
Pre-v0.32, openai-compatible auth was duplicated 3 times in gateway.ts at
instantiateEmbedding, instantiateExpansion, instantiateChat — with subtle
drift (embedding had a `${recipe.id.toUpperCase()}_API_KEY` fallback the
other two lacked). Codex outside-voice review caught this during /plan-eng-review.
D12=A: unify all three through `Recipe.resolveAuth?(env)` (declared in the
prior commit). Two new module-level helpers:
- `defaultResolveAuth(recipe, env, touchpoint)` — applied when a recipe
doesn't declare its own resolver. Returns Authorization Bearer with
`auth_env.required[0]`, falling back to the first present
`auth_env.optional` env var, or 'unauthenticated' for no-auth recipes
like Ollama. Throws AIConfigError with the recipe's setup_hint when
required env is missing.
- `applyResolveAuth(recipe, cfg, touchpoint)` — returns
`createOpenAICompatible` options. Bearer-via-Authorization paths use
the SDK's native `apiKey` field; custom-header paths (Azure: api-key)
use `headers` and OMIT apiKey to avoid double-auth leaks.
The 3 `case 'openai-compatible':` branches in instantiateEmbedding (line
~281), instantiateExpansion (line ~728), instantiateChat (line ~931) each
collapse from ~10 lines of bespoke auth handling to a single
`applyResolveAuth(recipe, cfg, '<touchpoint>')` call.
Also: the litellm-template hardcode at gateway.ts:223 (`recipe.id ===
'litellm'`) is replaced with a union check for
`EmbeddingTouchpoint.user_provided_models === true` (D8=A wire-through
per Codex finding #3). Pre-v0.32 builds keep working via back-compat
`recipe.id === 'litellm'` clause; new recipes declaring
user_provided_models pick up the same gating automatically.
Existing 9 recipes (openai, anthropic, google, deepseek, groq, ollama,
litellm-proxy, together, voyage) gain zero per-recipe edits — the
default resolver covers their existing behavior. Behavior change for
ollama expansion/chat only: now reads OLLAMA_API_KEY when set (pre-v0.32
silently passed 'unauthenticated' for those touchpoints; embedding
already read it). Ollama servers ignore the header so no real-world
impact; this aligns the 3 touchpoints.
Tests: bun test test/ai/ — 77/77 pass.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (D8=A,
D12=A; addresses Codex findings #3, #4).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai): IRON RULE regression test for v0.32 resolveAuth refactor
Pins the contract that the v0.32 D2/D12=A resolveAuth refactor preserves
auth behavior for the 9 existing recipes (openai, anthropic, google,
deepseek, groq, ollama, litellm-proxy, together, voyage).
10 cases covering:
- the 9 expected recipe ids are still registered
- every recipe with non-empty required[] returns Authorization Bearer <key>
- missing required env throws AIConfigError naming recipe + touchpoint + env-var
- Ollama (empty required, optional set) reads first present optional env
- Ollama (no env) falls back to "Bearer unauthenticated"
- all 3 touchpoints (embedding/expansion/chat) produce identical auth
shape for the same recipe + env (this is the core regression: pre-v0.32,
embedding had a fallback the other two lacked)
- applyResolveAuth converts Authorization Bearer to {apiKey} (SDK-native)
- applyResolveAuth respects a custom-header override (Azure preview; the
recipe ships in commit 8) and emits {headers} WITHOUT apiKey to avoid
double-auth
- native-* recipes (openai, anthropic, google) intentionally have no
resolveAuth declared (they use AI-SDK adapters directly)
- all openai-compatible recipes ship without resolveAuth in v0.32 (default
applies); the first override is Azure in commit 8
Also: export `defaultResolveAuth` and `applyResolveAuth` as @internal
gateway helpers so tests can pin them directly. Mirrors the pattern of
`splitByTokenBudget` and `isTokenLimitError` already exported with the
same @internal annotation.
Tests: bun test test/ai/ — 87/87 pass (10 new + 77 existing).
Typecheck: clean.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (IRON RULE
per Section 3 test review).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add llama-server recipe (#702 reworked)
10th recipe in the registry; first to ship Recipe.probe (D13=A) and the
second user_provided_models recipe (litellm-proxy is the first).
llama.cpp's llama-server exposes an OpenAI-compatible /v1/embeddings
endpoint. Distinct from Ollama: different default port (8080), different
model-management story (you launch it with --model <path>; the server
serves whatever was passed). Recipe ships with `models: []`,
`user_provided_models: true`, `default_dims: 0` so the wizard refuses
implicit defaults and forces explicit --embedding-model + --embedding-dimensions.
Added:
- src/core/ai/recipes/llama-server.ts (61 lines)
- probeLlamaServer() in src/core/ai/probes.ts; reads
LLAMA_SERVER_BASE_URL with default http://localhost:8080/v1
- Registered in src/core/ai/recipes/index.ts (10 recipes total now)
- test/ai/recipe-llama-server.test.ts (8 cases): registered + shape,
user_provided_models flag, probe declared + reachability fail-with-hint,
default-auth covering no-env / API_KEY / URL-shaped-only paths
Hardening: defaultResolveAuth in gateway.ts now skips URL-shaped optional
env entries (names ending in _URL or _BASE_URL) when picking a fallback
auth token. Pre-fix, OLLAMA_BASE_URL=http://my-ollama would have become
the Bearer token; Ollama ignores it but llama-server (and future
local-server recipes) shouldn't depend on the server tolerating garbage
auth. The regression test (recipes-existing-regression) gains one case
pinning this contract.
Per-recipe test file follows D7=B (per-recipe over DRY for readability).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 4
of 11). Reworked from #702 because the original PR didn't model the
recipe-owned probe pattern (D13=A) or user_provided_models (D8=A).
Tests: bun test test/ai/ — 95/95 pass (8 new + 87 existing).
Co-Authored-By: SiyaoZheng <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add MiniMax recipe (#148 reworked)
11th recipe. embo-01 model, 1536 dims, $0.07/1M tokens.
OpenAI-compatible at api.minimax.chat. MiniMax requires a `type:
'db' | 'query'` field for asymmetric retrieval (documents indexed with
type='db', queries embedded with type='query'). gbrain has no
query/document signal at the embed-call site today, so v1 defaults to
type='db' for both indexing and retrieval — same vector space, symmetric
similarity. Asymmetric query support is a follow-up TODO that needs the
embed seam to thread query/document context.
Plumbed via src/core/ai/dims.ts: dimsProviderOptions returns
{openaiCompatible: {type: 'db'}} for modelId === 'embo-01'.
Conservative max_batch_tokens=4096 declared (MiniMax docs don't publish
the limit). Recursive halving in the gateway catches token-limit errors
at runtime.
Tests: bun test test/ai/ — 101/101 (6 new + 95 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 5
of 11). Reworked from #148.
Co-Authored-By: cacity <20351699+cacity@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add Alibaba DashScope recipe (#59 split, part 1/2)
12th recipe. text-embedding-v3 (current) + text-embedding-v2; 1024
default dims with Matryoshka options [64, 128, 256, 512, 768, 1024].
OpenAI-compatible at dashscope-intl.aliyuncs.com. China-region users
override via cfg.base_urls['dashscope']; v0.32 ships with the
international default.
Conservative max_batch_tokens=8192 + chars_per_token=2 declared because
Alibaba doesn't publish a hard batch limit and text-embedding-v3 mixes
English + CJK heavily (CJK density closer to Voyage than OpenAI tiktoken).
Tests: bun test test/ai/ — 106/106 (5 new + 101 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 6
of 11). Reworked from #59 (DashScope+Zhipu split into 2 commits per
the plan; Zhipu lands next).
Co-Authored-By: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add Zhipu AI (BigModel) recipe (#59 split, part 2/2)
13th recipe. embedding-3 (current) + embedding-2; 1024 default dims
with Matryoshka options [256, 512, 1024, 2048].
OpenAI-compatible at open.bigmodel.cn. embedding-3 at 2048 dims exceeds
pgvector's HNSW cap of 2000 — those brains fall back to exact vector
scans via the existing chunkEmbeddingIndexSql policy at
src/core/vector-index.ts. Default stays at 1024 (HNSW-fast); users who
want maximum fidelity opt into 2048 via --embedding-dimensions and
accept the slower retrieval.
Tests pin the HNSW boundary: 1024 returns the index SQL, 2048 returns
the skip-index/exact-scan SQL.
Tests: bun test test/ai/ — 112/112 (6 new + 106 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 7
of 11). Reworked from #59. Together with DashScope (commit 6), closes
the China-region embedding gap users repeatedly reported (DashScope
covers Alibaba, Zhipu covers BigModel; both ship with international
endpoints by default).
Co-Authored-By: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): add Azure OpenAI recipe (#459 reworked)
14th recipe and the first to exercise both v0.32 architectural seams:
- resolveAuth (D12=A) returns `{headerName: 'api-key', token: <key>}`
instead of the default Authorization Bearer. Azure rejects double-auth,
so applyResolveAuth puts the key in `headers` and OMITS apiKey.
- A new `Recipe.resolveOpenAICompatConfig?(env)` seam (Recipe.ts) lets
the recipe template the baseURL from env (Azure: ENDPOINT + DEPLOYMENT
combine into a non-/v1 path) and inject a custom fetch wrapper that
splices ?api-version= onto every request URL.
The fetch wrapper is type-safe via `as unknown as typeof fetch`; AI SDK
never calls TS's strict `preconnect()` method on the wrapper so the cast
is sound. `applyOpenAICompatConfig` (new gateway helper) routes through
the recipe override or falls back to the pre-v0.32 base_urls/base_url_default
behavior — existing 13 recipes get zero behavior change.
API version defaults to `2024-10-21` (current stable as of 2026-05);
override via AZURE_OPENAI_API_VERSION env. Endpoint trailing slash gets
stripped during URL construction so users can copy-paste from the Azure
portal.
Tests (12 cases in test/ai/recipe-azure-openai.test.ts):
- resolveAuth returns api-key NOT Authorization Bearer
- applyResolveAuth puts key in headers, NOT apiKey (no double-auth)
- baseURL templating from endpoint + deployment, with trailing-slash strip
- AIConfigError on missing endpoint OR deployment
- fetch wrapper splices api-version (default + AZURE_OPENAI_API_VERSION override)
- fetch wrapper does NOT double-add api-version when caller already set it
- applyOpenAICompatConfig honors recipe override
IRON RULE regression test updated: now asserts azure-openai is the
documented exception that overrides resolveAuth; any future override
needs review.
Tests: bun test test/ai/ — 124/124 (12 new + 112 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 8
of 11, plus the resolveOpenAICompatConfig seam discovered during fold-in).
Reworked from #459. The original PR proposed a hardcoded AzureOpenAI
client switch; this implementation routes through the unified seams so
future Azure-shaped providers (other custom-URL services) can reuse them.
Co-Authored-By: JamesJZhang <32652444+JamesJZhang@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(ai): adjacent fixes — no_batch_cap (#779) + config-key fallbacks (#121)
Two small ergonomics fixes folded together (#765 deferred — see TODOS.md
follow-up; the CJK PGLite extraction was bigger than the plan estimated).
#779 reworked (alexandreroumieu-codeapprentice): silence the
missing-max_batch_tokens startup warning for recipes with genuinely
dynamic batch capacity. New `EmbeddingTouchpoint.no_batch_cap?: true`
field. Set on ollama (capacity depends on locally loaded model +
OLLAMA_NUM_PARALLEL), litellm-proxy (depends on backend), llama-server
(set by --ctx-size at server launch). Three less stderr warnings on
every gateway configure; google still warns (it's a real fixed-cap
provider that ought to ship a max_batch_tokens declaration).
Bonus: litellm-proxy now declares `user_provided_models: true`, removing
the last consumer of the legacy `recipe.id === 'litellm'` hardcode in
gateway.ts:223 (D8=A wire-through completion).
#121 reworked (vinsew): self-contained API keys. Two parts:
1. config.ts: ANTHROPIC_API_KEY env merge was silently missing.
loadConfig() merged OPENAI_API_KEY but not ANTHROPIC_API_KEY into
the file-config-shape result. One-line addition.
2. cli.ts:buildGatewayConfig: when ~/.gbrain/config.json declares
openai_api_key / anthropic_api_key but the process env doesn't
have those env vars set (common for launchd-spawned daemons,
agent subprocess tools, containers that don't propagate
~/.zshrc), fold the config-file values into the gateway env
snapshot. Process env still wins (loaded last) so per-process
overrides keep working.
Tests (4 cases in test/ai/no-batch-cap-suppression.test.ts):
- Ollama / LiteLLM / llama-server all declare no_batch_cap: true
- configureGateway does NOT warn for those three
- configureGateway STILL warns for google (regression guard)
- Cross-cutting invariant: empty-models recipes declare user_provided_models
Tests: bun test test/ai/ — 128/128 (4 new + 124 prior).
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 9 of 11).
#765 (Hunyuan PGLite + CJK keyword fallback) deferred to TODOS.md
follow-up; the CJK extraction (~150 lines + scoring logic + tests) is
larger than the wave's adjacent-fix lane should carry. Closes that PR
with a deferral note.
Co-Authored-By: alexandreroumieu-codeapprentice <noreply@github.com>
Co-Authored-By: vinsew <noreply@github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(discoverability): doctor alt-provider advisory + init user_provided_models refusal
Two small but high-leverage changes that address the discoverability
problem the v0.32 wave is trying to fix.
src/commands/doctor.ts: new `alternative_providers` check (8c). After
the existing embedding-provider smoke test, walks listRecipes() and
surfaces any recipe whose required env vars are ALL present in the
process env but is not the currently configured provider. Reports as
status: 'ok' with an informational message — never errors. Helps users
discover that, e.g., `OPENAI_API_KEY=x DASHSCOPE_API_KEY=y` configured
for openai means they have a Chinese-region alternative ready without
extra setup.
src/commands/init.ts: user_provided_models recipes (litellm, llama-server)
now refuse the implicit "first model" pick from shorthand --model with
a structured setup hint pointing the user at the explicit form
`--embedding-model <provider>:<your-model-id> --embedding-dimensions <N>`.
Pre-fix, shorthand --model litellm threw "no embedding models listed"
which was technically correct but unhelpful. The new error includes the
recipe's setup_hint when available.
Tests: bun test test/ai/ — 128/128 pass; typecheck clean.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 10
of 11). The full interactive provider chooser in init.ts (the bigger
piece of the discoverability lane) is deferred to a v0.32.x follow-up;
this commit ships the doctor advisory + cleaner refusal that close the
80% case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(v0.32.0): embedding-providers.md + README callout + CHANGELOG + TODOS.md
Final commit of the v0.32 wave. Closes the discoverability gap that
generated the 17-PR community cluster.
- New docs/integrations/embedding-providers.md: capability matrix, decision
tree, per-recipe one-pagers, OAuth provider notes, "my provider isn't
listed" pointer to LiteLLM proxy. Voice: capability not marketing per
CLAUDE.md voice rules.
- README.md: embedding-providers callout near the top, naming the count
(14 recipes) and pointing at the new doc.
- CHANGELOG.md: v0.32.0 entry following the verdict-headline format from
CLAUDE.md voice rules. Lead-with-numbers ("14 providers, 5 new"), what-this-
means-for-users closer, "to take advantage" upgrade block, itemized
changes, contributor credits, deferred-with-context list.
- VERSION + package.json: 0.31.1 → 0.32.0. Minor bump justified by the
new public Recipe surface (resolveAuth, resolveOpenAICompatConfig, probe,
user_provided_models, no_batch_cap fields), the new OAuth subsystem
scaffold (deferred to v0.32.x but typed in v0.32.0), and the 5 new
recipes.
- TODOS.md: 7 follow-up entries for the v0.32 wave's deferred work
(Vertex ADC, Copilot OAuth, Codex OAuth, CJK PGLite, interactive
wizard, real-credentials CI matrix, MiniMax asymmetric retrieval,
multimodal hardcode un-stuck). Each entry has full context + the
exact file paths + the spike work needed so a future contributor can
pick up cleanly.
Tests: bun test test/ai/ — 128/128 pass; typecheck clean.
Plan: ~/.claude/plans/ok-lets-turn-this-enumerated-sonnet.md (commit 11
of 11). Wave complete: 11 commits, ~1500 net lines, 5 new recipes, full
docs, doctor advisory, IRON RULE regression test, 7 TODOS for the
v0.32.x follow-up wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: regenerate llms.txt + llms-full.txt for v0.32.0
After commit c384fadc added the embedding-providers callout to README.md,
the committed llms-full.txt drifted from the generator output and the
build-llms test failed. Running `bun run build:llms` regenerates both
files. The single line addition is the README callout pointing at
docs/integrations/embedding-providers.md.
Tests: bun test test/build-llms.test.ts — 7/7 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: hermetic GBRAIN_HOME for brain-registry serial flake + withEnv on recipe-llama-server
Two test-isolation cleanups uncovered while shipping v0.32.
test/brain-registry.serial.test.ts (the BrainRegistry "empty/null/undefined
id routes to host" test): pre-existing flake on dev machines that have a
real ~/.gbrain/config.json. The test asserts getBrain(null) REJECTS but
on those machines the host-init path RESOLVES instead (it found the
maintainer's actual brain). The fix pins GBRAIN_HOME to a guaranteed-empty
tempdir for the test's duration so host-init has nothing to find and fails
loudly with a non-UnknownBrainError — exactly what the assertion wants.
File is .serial.test.ts so direct process.env mutation is allowed by the
test-isolation linter (R1 quarantine).
test/ai/recipe-llama-server.test.ts: rewrites the manual beforeEach/afterEach
env save/restore as withEnv() per the canonical pattern in
test/helpers/with-env.ts. The original was correct in behavior but tripped
the test-isolation linter (R1: process.env mutation). withEnv() is exactly
the cross-test-safe save+try/finally+restore the manual code did, just
factored out. No behavior change.
Tests: bun run test — 5217 pass / 0 fail (was 5027 / 1 pre-existing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address 5 codex pre-merge findings (dim passthrough + URL routing + MiniMax host)
Codex adversarial review during /ship caught five real production bugs.
All five fixed with regression test coverage.
1. **dimsProviderOptions on openai-compatible** (src/core/ai/dims.ts):
text-embedding-3-* (Azure), text-embedding-v3 (DashScope), and
embedding-3 (Zhipu) now thread `dimensions` to the wire. Without this,
Azure-default 3072d hard-fails a 1536d brain on first embed; DashScope
and Zhipu Matryoshka requests silently get the provider's default size
instead of what the user asked for. New tests in
recipe-azure-openai/dashscope/zhipu pin the contract.
2. **`gbrain init --embedding-model llama-server:foo` verbose path**
(src/commands/init.ts): now refuses without `--embedding-dimensions`
for user_provided_models recipes. Pre-fix, the shorthand `--model`
path was guarded but the verbose `--embedding-model` path fell through
to configureGateway's 1536d default and silently created the wrong-
width schema; failure surfaced only at first real embed.
3. **MiniMax host correction** (src/core/ai/recipes/minimax.ts):
`api.minimax.chat/v1` → `api.minimaxi.com/v1` matches MiniMax's
current OpenAI-compatible docs. Default-config users would have hit
the wrong endpoint before auth or model selection mattered.
4. **`LLAMA_SERVER_BASE_URL` reaches the gateway** (src/cli.ts:
buildGatewayConfig): env-set local-server URLs (LLAMA_SERVER_BASE_URL,
OLLAMA_BASE_URL, LMSTUDIO_BASE_URL, LITELLM_BASE_URL) now thread into
`cfg.base_urls` so embed traffic hits the configured port. Pre-fix,
the probe would succeed against a custom port while real embed calls
went to localhost:8080. Caller-supplied `cfg.provider_base_urls` still
wins over env.
5. **Recipe.probe(baseURL?) accepts the resolved URL** (src/core/ai/types.ts,
src/core/ai/probes.ts, src/core/ai/recipes/llama-server.ts): when the
user configures `provider_base_urls.llama-server` in config but no env
var is set, the probe and gateway no longer disagree. Callers with cfg
pass the resolved URL; legacy callers fall back to env / recipe default.
CHANGELOG updated; llms-full.txt regenerated.
Tests: bun run test — 5220/5220 pass / 0 fail (was 5217 / 0; +3 new
codex-finding regression tests).
Pre-merge codex adversarial: ran during /ship Step 11 against the v0.32
diff. All 5 findings addressed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): isolate v0.32 no-batch-cap test from mock.module leak (closes 19 CI fails)
Three CI test-isolation fixes uncovered by yesterday's CI run on PR #810:
1. **`scripts/test-shard.sh` excludes `*.serial.test.ts`** (was running them
in parallel shards). Without this, serial files race with non-serial
files in the CI shard process. Mirrors `scripts/run-unit-shard.sh`'s
exclusion set; 1-line `find` filter.
2. **`scripts/run-serial-tests.sh` runs each serial file in its own bun
process**. Pre-fix, all serial files ran in ONE bun process with
`--max-concurrency=1` — that limits intra-file concurrency but does
NOT prevent module-registry leakage across files. When
`eval-takes-quality-runner.serial.test.ts` does
`mock.module('../src/core/ai/gateway.ts', () => ({chat, configureGateway}))`
(a partial mock missing `resetGateway`, `defaultResolveAuth`, etc.),
the next file in the same process gets the partial mock on import and
`import { resetGateway }` fails with "Export named 'resetGateway' not
found." Per-file processes give true isolation; cost is ~100ms × N
files (negligible vs CI walltime).
3. **`test/ai/no-batch-cap-suppression.test.ts` → `.serial.test.ts`**.
The test mutates `console.warn` globally (mock spy). When other tests
in the same shard process load `src/core/ai/gateway.ts` and call
`configureGateway()` first, they populate the module-scoped
`_warnedRecipes` Set; the test's `resetGateway()` clears it but races
if other gateway-touching code runs concurrently in the same process.
Renaming to `.serial.test.ts` quarantines it via fix #1 + #2.
4. **CI workflow gains a serial-tests step on shard 1**. Pre-fix, shard 1
ran `bun run verify` + the parallel shard, but no shard ran
`*.serial.test.ts` files. After fix #1 excludes them from shards, they
need explicit invocation. New step:
`bash scripts/run-serial-tests.sh` (shard 1 only).
Tests: bun run test — 5220 / 0 fail (matches local pre-CI run; was
showing 19 fails on CI for PR #810 due to fixes #1-#3 missing).
Failure analysis from .context/attachments/test__2__75236697976.log:
- 18 multimodal failures: caused by mock.module leak from
eval-takes-quality-runner.serial.test.ts being run alongside
voyage-multimodal.test.ts in the same parallel shard process. After
fix #1 + fix #3, eval-takes-quality only runs in serial pass; after
fix #2, its mock.module doesn't leak to subsequent serial files.
- 1 no-batch-cap failure: same root cause; fix #3 quarantines it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: SiyaoZheng <noreply@github.com>
Co-authored-by: cacity <20351699+cacity@users.noreply.github.com>
Co-authored-by: Magicray1217 <267836857+Magicray1217@users.noreply.github.com>
Co-authored-by: JamesJZhang <32652444+JamesJZhang@users.noreply.github.com>
758 lines
32 KiB
TypeScript
758 lines
32 KiB
TypeScript
import { execSync } from 'child_process';
|
|
import { readdirSync, lstatSync, existsSync, copyFileSync, mkdirSync, readFileSync } from 'fs';
|
|
import { join, dirname } from 'path';
|
|
import { fileURLToPath } from 'url';
|
|
import { homedir } from 'os';
|
|
|
|
const __filename = fileURLToPath(import.meta.url);
|
|
const __dirname = dirname(__filename);
|
|
import { saveConfig, loadConfig, toEngineConfig, gbrainPath, configPath, isThinClient, type GBrainConfig } from '../core/config.ts';
|
|
import { createEngine } from '../core/engine-factory.ts';
|
|
import { discoverOAuth, mintClientCredentialsToken, smokeTestMcp } from '../core/remote-mcp-probe.ts';
|
|
|
|
export async function runInit(args: string[]) {
|
|
const isSupabase = args.includes('--supabase');
|
|
const isPGLite = args.includes('--pglite');
|
|
const isMcpOnly = args.includes('--mcp-only');
|
|
const isForce = args.includes('--force');
|
|
const isNonInteractive = args.includes('--non-interactive');
|
|
const isMigrateOnly = args.includes('--migrate-only');
|
|
const jsonOutput = args.includes('--json');
|
|
const urlIndex = args.indexOf('--url');
|
|
const manualUrl = urlIndex !== -1 ? args[urlIndex + 1] : null;
|
|
const keyIndex = args.indexOf('--key');
|
|
const apiKey = keyIndex !== -1 ? args[keyIndex + 1] : null;
|
|
const pathIndex = args.indexOf('--path');
|
|
const customPath = pathIndex !== -1 ? args[pathIndex + 1] : null;
|
|
|
|
// Multi-topology v1: thin-client init. Skips local engine entirely; writes
|
|
// remote_mcp config that the CLI dispatch guard reads to refuse DB-bound ops.
|
|
if (isMcpOnly) {
|
|
return initRemoteMcp({ args, jsonOutput, isForce, isNonInteractive });
|
|
}
|
|
|
|
// Re-run guard (A8): if thin-client config is already present, refuse to
|
|
// create a local engine without --force. Catches the scripted-setup-loop
|
|
// friction (running setup-gbrain repeatedly on a thin-client machine).
|
|
const existing = loadConfig();
|
|
if (isThinClient(existing) && !isForce && !isMigrateOnly) {
|
|
const url = existing!.remote_mcp!.mcp_url;
|
|
const msg = `Thin-client config already present at ${configPath()} (remote_mcp.mcp_url=${url}).\n` +
|
|
`Re-init would create a local engine and conflict with the remote MCP setup.\n` +
|
|
`Use --force to overwrite, or \`gbrain init --mcp-only --force\` to refresh thin-client config.`;
|
|
if (jsonOutput) {
|
|
console.log(JSON.stringify({ status: 'error', reason: 'thin_client_config_present', mcp_url: url, message: msg }));
|
|
} else {
|
|
console.error(msg);
|
|
}
|
|
process.exit(1);
|
|
}
|
|
|
|
// v0.14: AI provider selection.
|
|
// --embedding-model PROVIDER:MODEL (verbose) or --model PROVIDER (shorthand, picks recipe default)
|
|
const embModelIdx = args.indexOf('--embedding-model');
|
|
const modelShortIdx = args.indexOf('--model');
|
|
const embDimsIdx = args.indexOf('--embedding-dimensions');
|
|
const expModelIdx = args.indexOf('--expansion-model');
|
|
// v0.27: --chat-model PROVIDER:MODEL — default subagent driver.
|
|
const chatModelIdx = args.indexOf('--chat-model');
|
|
const aiOpts = await resolveAIOptions(
|
|
embModelIdx !== -1 ? args[embModelIdx + 1] : null,
|
|
modelShortIdx !== -1 ? args[modelShortIdx + 1] : null,
|
|
embDimsIdx !== -1 ? parseInt(args[embDimsIdx + 1], 10) : null,
|
|
expModelIdx !== -1 ? args[expModelIdx + 1] : null,
|
|
chatModelIdx !== -1 ? args[chatModelIdx + 1] : null,
|
|
);
|
|
|
|
// Schema-only path: apply initSchema against the already-configured engine
|
|
// without ever calling saveConfig. Used by apply-migrations, the stopgap
|
|
// script, and the postinstall hook. Bare `gbrain init` defaults to PGLite
|
|
// and overwrites any existing Postgres config — we must never take that
|
|
// branch from a migration orchestrator.
|
|
if (isMigrateOnly) {
|
|
return initMigrateOnly({ jsonOutput });
|
|
}
|
|
|
|
// Explicit PGLite mode
|
|
if (isPGLite || (!isSupabase && !manualUrl && !isNonInteractive)) {
|
|
// Smart detection: scan for .md files unless --pglite flag forces it
|
|
if (!isPGLite && !isSupabase) {
|
|
const fileCount = countMarkdownFiles(process.cwd());
|
|
if (fileCount >= 1000) {
|
|
console.log(`Found ~${fileCount} .md files. For a brain this size, Supabase gives faster`);
|
|
console.log('search and remote access ($25/mo). PGLite works too but search will be slower at scale.');
|
|
console.log('');
|
|
console.log(' gbrain init --supabase Set up with Supabase (recommended for large brains)');
|
|
console.log(' gbrain init --pglite Use local PGLite anyway');
|
|
console.log('');
|
|
// Default to PGLite, let the user choose Supabase if they want
|
|
}
|
|
}
|
|
|
|
return initPGLite({ jsonOutput, apiKey, customPath, aiOpts });
|
|
}
|
|
|
|
// Supabase/Postgres mode
|
|
let databaseUrl: string;
|
|
if (manualUrl) {
|
|
databaseUrl = manualUrl;
|
|
} else if (isNonInteractive) {
|
|
const envUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL;
|
|
if (envUrl) {
|
|
databaseUrl = envUrl;
|
|
} else {
|
|
console.error('--non-interactive requires --url <connection_string> or GBRAIN_DATABASE_URL env var');
|
|
process.exit(1);
|
|
}
|
|
} else {
|
|
databaseUrl = await supabaseWizard();
|
|
}
|
|
|
|
return initPostgres({ databaseUrl, jsonOutput, apiKey, aiOpts });
|
|
}
|
|
|
|
/**
|
|
* Resolve AI provider options from CLI flags. Verbose form (--embedding-model
|
|
* openai:text-embedding-3-large) overrides shorthand (--model openai which
|
|
* expands to the recipe's first embedding model).
|
|
*/
|
|
async function resolveAIOptions(
|
|
verbose: string | null,
|
|
shorthand: string | null,
|
|
dimsArg: number | null,
|
|
expansion: string | null,
|
|
chat: string | null,
|
|
): Promise<{ embedding_model?: string; embedding_dimensions?: number; expansion_model?: string; chat_model?: string }> {
|
|
const out: { embedding_model?: string; embedding_dimensions?: number; expansion_model?: string; chat_model?: string } = {};
|
|
|
|
if (verbose) {
|
|
out.embedding_model = verbose;
|
|
} else if (shorthand) {
|
|
const { getRecipe } = await import('../core/ai/recipes/index.ts');
|
|
const recipe = getRecipe(shorthand);
|
|
if (!recipe) {
|
|
console.error(`Unknown provider: ${shorthand}. Run \`gbrain providers list\` to see known providers.`);
|
|
process.exit(1);
|
|
}
|
|
// v0.32 D8=A: recipes flagged user_provided_models (litellm, llama-server)
|
|
// refuse implicit "first model" pick with a setup hint pointing the user
|
|
// at the explicit form. The shorthand --model is meaningless for these
|
|
// recipes because there's no canonical first model.
|
|
if (recipe.touchpoints.embedding?.user_provided_models === true) {
|
|
console.error(
|
|
`Provider ${shorthand} requires you to specify the model + dimensions explicitly:\n` +
|
|
` gbrain init --embedding-model ${shorthand}:<your-model-id> --embedding-dimensions <N>\n` +
|
|
(recipe.setup_hint ? `\nSetup: ${recipe.setup_hint}` : '')
|
|
);
|
|
process.exit(1);
|
|
}
|
|
const firstModel = recipe.touchpoints.embedding?.models[0];
|
|
if (!firstModel) {
|
|
console.error(`Provider ${shorthand} has no embedding models listed. Use --embedding-model provider:model.`);
|
|
process.exit(1);
|
|
}
|
|
out.embedding_model = `${shorthand}:${firstModel}`;
|
|
out.embedding_dimensions = recipe.touchpoints.embedding!.default_dims;
|
|
}
|
|
|
|
if (dimsArg !== null && !Number.isNaN(dimsArg) && dimsArg > 0) {
|
|
out.embedding_dimensions = dimsArg;
|
|
} else if (out.embedding_model && out.embedding_dimensions === undefined) {
|
|
// Derive default dims from the resolved recipe when verbose form was used.
|
|
const { getRecipe } = await import('../core/ai/recipes/index.ts');
|
|
const providerId = out.embedding_model.split(':')[0];
|
|
const recipe = getRecipe(providerId);
|
|
// v0.32: user_provided_models recipes (litellm, llama-server) have
|
|
// default_dims=0 and ship with `models: []` — there's no sensible
|
|
// fallback. Refuse explicitly here too. Without this, the verbose path
|
|
// `--embedding-model llama-server:foo` (no --embedding-dimensions) would
|
|
// fall through to configureGateway's default (1536), creating a
|
|
// wrong-width schema that explodes only at first embed.
|
|
if (recipe?.touchpoints.embedding?.user_provided_models === true) {
|
|
console.error(
|
|
`Provider ${providerId} requires --embedding-dimensions <N> when using --embedding-model ${out.embedding_model}.\n` +
|
|
`User-driven-model recipes (litellm, llama-server) have no default dimension.\n` +
|
|
(recipe.setup_hint ? `\nSetup: ${recipe.setup_hint}` : '')
|
|
);
|
|
process.exit(1);
|
|
}
|
|
if (recipe?.touchpoints.embedding?.default_dims) {
|
|
out.embedding_dimensions = recipe.touchpoints.embedding.default_dims;
|
|
}
|
|
}
|
|
|
|
if (expansion) out.expansion_model = expansion;
|
|
if (chat) out.chat_model = chat;
|
|
|
|
return out;
|
|
}
|
|
|
|
/**
|
|
* Apply the schema against the already-configured engine. No saveConfig.
|
|
* No PGLite fallback when no config exists. Used by migration orchestrators
|
|
* to bump an existing brain's schema to the latest version without
|
|
* clobbering the user's chosen engine.
|
|
*/
|
|
async function initMigrateOnly(opts: { jsonOutput: boolean }) {
|
|
const config = loadConfig();
|
|
if (!config) {
|
|
const msg = 'No brain configured. Run `gbrain init` (interactive) or `gbrain init --pglite` / `gbrain init --supabase` first.';
|
|
if (opts.jsonOutput) {
|
|
console.log(JSON.stringify({ status: 'error', reason: 'no_config', message: msg }));
|
|
} else {
|
|
console.error(msg);
|
|
}
|
|
process.exit(1);
|
|
}
|
|
|
|
const engine = await createEngine(toEngineConfig(config));
|
|
try {
|
|
await engine.connect(toEngineConfig(config));
|
|
await engine.initSchema();
|
|
} finally {
|
|
try { await engine.disconnect(); } catch { /* best-effort */ }
|
|
}
|
|
|
|
if (opts.jsonOutput) {
|
|
console.log(JSON.stringify({ status: 'success', engine: config.engine, mode: 'migrate-only' }));
|
|
} else {
|
|
console.log(`Schema up to date (engine: ${config.engine}).`);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* `gbrain init --mcp-only` — thin-client setup. Writes a `remote_mcp` config
|
|
* field, runs three pre-flight smokes (OAuth discovery, token round-trip,
|
|
* MCP initialize), and never creates a local engine.
|
|
*
|
|
* Required flags (or env vars):
|
|
* --issuer-url <url> (or GBRAIN_REMOTE_ISSUER_URL)
|
|
* --mcp-url <url> (or GBRAIN_REMOTE_MCP_URL)
|
|
* --oauth-client-id <id> (or GBRAIN_REMOTE_CLIENT_ID)
|
|
* --oauth-client-secret <s> (or GBRAIN_REMOTE_CLIENT_SECRET; preferred)
|
|
*
|
|
* Re-run semantics: if a thin-client config already exists, --force overwrites;
|
|
* otherwise refuses with a hint pointing at the existing mcp_url.
|
|
*/
|
|
async function initRemoteMcp(opts: {
|
|
args: string[];
|
|
jsonOutput: boolean;
|
|
isForce: boolean;
|
|
isNonInteractive: boolean;
|
|
}) {
|
|
const { args, jsonOutput, isForce } = opts;
|
|
const arg = (flag: string) => {
|
|
const i = args.indexOf(flag);
|
|
return i !== -1 ? args[i + 1] : null;
|
|
};
|
|
const issuerUrl = (arg('--issuer-url') ?? process.env.GBRAIN_REMOTE_ISSUER_URL ?? '').trim();
|
|
const mcpUrl = (arg('--mcp-url') ?? process.env.GBRAIN_REMOTE_MCP_URL ?? '').trim();
|
|
const clientId = (arg('--oauth-client-id') ?? process.env.GBRAIN_REMOTE_CLIENT_ID ?? '').trim();
|
|
const clientSecret = (arg('--oauth-client-secret') ?? process.env.GBRAIN_REMOTE_CLIENT_SECRET ?? '').trim();
|
|
|
|
function fail(reason: string, message: string, extra: Record<string, unknown> = {}): never {
|
|
if (jsonOutput) {
|
|
console.log(JSON.stringify({ status: 'error', reason, message, ...extra }));
|
|
} else {
|
|
console.error(message);
|
|
}
|
|
process.exit(1);
|
|
}
|
|
|
|
if (!issuerUrl) fail('missing_issuer_url', '--issuer-url is required (or set GBRAIN_REMOTE_ISSUER_URL). Example: --issuer-url https://brain-host.local:3001');
|
|
if (!mcpUrl) fail('missing_mcp_url', '--mcp-url is required (or set GBRAIN_REMOTE_MCP_URL). Example: --mcp-url https://brain-host.local:3001/mcp');
|
|
if (!clientId) fail('missing_client_id', '--oauth-client-id is required (or set GBRAIN_REMOTE_CLIENT_ID). Get it from `gbrain auth register-client` on the host.');
|
|
if (!clientSecret) fail('missing_client_secret', '--oauth-client-secret is required (or set GBRAIN_REMOTE_CLIENT_SECRET). Get it from `gbrain auth register-client` on the host.');
|
|
|
|
// Re-run guard for --mcp-only specifically: refuse without --force to
|
|
// avoid silently rotating credentials on a working install.
|
|
const existing = loadConfig();
|
|
if (isThinClient(existing) && !isForce) {
|
|
const prevUrl = existing!.remote_mcp!.mcp_url;
|
|
fail(
|
|
'thin_client_config_present',
|
|
`Thin-client config already present at ${configPath()} (remote_mcp.mcp_url=${prevUrl}).\n` +
|
|
`Re-running --mcp-only would overwrite. Use --force to refresh.`,
|
|
{ mcp_url: prevUrl },
|
|
);
|
|
}
|
|
|
|
if (!jsonOutput) {
|
|
console.log('Thin-client setup — running pre-flight smoke...');
|
|
console.log(` issuer: ${issuerUrl}`);
|
|
console.log(` mcp: ${mcpUrl}`);
|
|
}
|
|
|
|
// 1. OAuth discovery
|
|
const disco = await discoverOAuth(issuerUrl);
|
|
if (!disco.ok) {
|
|
fail(
|
|
`discovery_${disco.reason}`,
|
|
`Pre-flight failed: OAuth discovery on ${issuerUrl} — ${disco.message}\n` +
|
|
`Hint: confirm the issuer_url, that the host is reachable, and that \`gbrain serve --http\` is running there.`,
|
|
{ detail: disco.message, ...(disco.status ? { status: disco.status } : {}) },
|
|
);
|
|
}
|
|
if (!jsonOutput) console.log(` ✓ OAuth discovery (token_endpoint=${disco.metadata.token_endpoint})`);
|
|
|
|
// 2. Token round-trip
|
|
const tokenRes = await mintClientCredentialsToken(disco.metadata.token_endpoint, clientId, clientSecret);
|
|
if (!tokenRes.ok) {
|
|
fail(
|
|
`token_${tokenRes.reason}`,
|
|
`Pre-flight failed: OAuth /token — ${tokenRes.message}\n` +
|
|
`Hint: the host operator can run \`gbrain auth register-client <name> --grant-types client_credentials --scopes read,write,admin\` to mint fresh credentials.`,
|
|
{ detail: tokenRes.message, ...(tokenRes.status ? { status: tokenRes.status } : {}) },
|
|
);
|
|
}
|
|
if (!jsonOutput) console.log(` ✓ OAuth /token (${tokenRes.token.token_type ?? 'bearer'}, scope=${tokenRes.token.scope ?? 'unspecified'})`);
|
|
|
|
// 3. MCP smoke
|
|
const mcpRes = await smokeTestMcp(mcpUrl, tokenRes.token.access_token);
|
|
if (!mcpRes.ok) {
|
|
fail(
|
|
`mcp_smoke_${mcpRes.reason}`,
|
|
`Pre-flight failed: MCP initialize on ${mcpUrl} — ${mcpRes.message}\n` +
|
|
`Hint: confirm \`mcp_url\` matches the path the host serves \`/mcp\` on (default: <issuer_url>/mcp).`,
|
|
{ detail: mcpRes.message, ...(mcpRes.status ? { status: mcpRes.status } : {}) },
|
|
);
|
|
}
|
|
if (!jsonOutput) console.log(` ✓ MCP initialize`);
|
|
|
|
// 4. Persist config. Preserve any existing AI/storage/etc. fields on
|
|
// the existing config — only overwrite remote_mcp + drop engine/database
|
|
// fields if this install is converting from local-engine to thin-client.
|
|
// For first-time setup, write a minimal config.
|
|
const baseConfig: Partial<GBrainConfig> = existing
|
|
? { ...existing, database_url: undefined, database_path: undefined }
|
|
: {};
|
|
// engine field is required on the type; leave it inferred to 'postgres'
|
|
// for default purposes — it's never used because the dispatch guard
|
|
// short-circuits any DB-bound path before connectEngine.
|
|
const config: GBrainConfig = {
|
|
...(baseConfig as GBrainConfig),
|
|
engine: existing?.engine ?? 'postgres',
|
|
remote_mcp: {
|
|
issuer_url: issuerUrl.replace(/\/+$/, ''),
|
|
mcp_url: mcpUrl,
|
|
oauth_client_id: clientId,
|
|
// Only persist the secret to disk if it didn't come from the env var.
|
|
// Env-var-supplied secrets stay in env; on-disk copy is opt-in via
|
|
// the --oauth-client-secret flag (or absent env var).
|
|
...(process.env.GBRAIN_REMOTE_CLIENT_SECRET === clientSecret
|
|
? {}
|
|
: { oauth_client_secret: clientSecret }),
|
|
},
|
|
};
|
|
// database_url / database_path get explicitly removed when converting; the
|
|
// spread above with `undefined` doesn't drop them in JSON, so prune.
|
|
const configRecord = config as unknown as Record<string, unknown>;
|
|
delete configRecord.database_url;
|
|
delete configRecord.database_path;
|
|
saveConfig(config);
|
|
|
|
if (jsonOutput) {
|
|
console.log(JSON.stringify({
|
|
status: 'success',
|
|
mode: 'thin-client',
|
|
issuer_url: config.remote_mcp!.issuer_url,
|
|
mcp_url: config.remote_mcp!.mcp_url,
|
|
oauth_client_id: config.remote_mcp!.oauth_client_id,
|
|
oauth_secret_in_config: 'oauth_client_secret' in config.remote_mcp!,
|
|
}));
|
|
} else {
|
|
console.log('');
|
|
console.log('Thin-client mode configured. No local DB.');
|
|
console.log(` Config: ${configPath()}`);
|
|
console.log(` Talks to: ${config.remote_mcp!.mcp_url}`);
|
|
console.log('');
|
|
console.log('Next steps:');
|
|
console.log(` 1. Configure your agent's MCP client to point at ${config.remote_mcp!.mcp_url} (Claude Desktop / Hermes / openclaw).`);
|
|
console.log(' 2. Run `gbrain doctor` to re-verify connectivity at any time.');
|
|
console.log(' 3. Run `gbrain remote ping` after writing markdown if you want the host to re-index immediately (Tier B).');
|
|
}
|
|
}
|
|
|
|
async function initPGLite(opts: {
|
|
jsonOutput: boolean;
|
|
apiKey: string | null;
|
|
customPath: string | null;
|
|
aiOpts?: { embedding_model?: string; embedding_dimensions?: number; expansion_model?: string; chat_model?: string };
|
|
}) {
|
|
const dbPath = opts.customPath || gbrainPath('brain.pglite');
|
|
console.log(`Setting up local brain with PGLite (no server needed)...`);
|
|
|
|
// Configure AI gateway BEFORE initSchema so the vector column uses the right dim.
|
|
if (opts.aiOpts?.embedding_model || opts.aiOpts?.chat_model) {
|
|
const { configureGateway } = await import('../core/ai/gateway.ts');
|
|
configureGateway({
|
|
embedding_model: opts.aiOpts?.embedding_model,
|
|
embedding_dimensions: opts.aiOpts?.embedding_dimensions,
|
|
expansion_model: opts.aiOpts?.expansion_model,
|
|
chat_model: opts.aiOpts?.chat_model,
|
|
env: { ...process.env },
|
|
});
|
|
if (opts.aiOpts?.embedding_model) console.log(` Embedding: ${opts.aiOpts.embedding_model} (${opts.aiOpts.embedding_dimensions ?? '?'}d)`);
|
|
if (opts.aiOpts?.expansion_model) console.log(` Expansion: ${opts.aiOpts.expansion_model}`);
|
|
if (opts.aiOpts?.chat_model) console.log(` Chat: ${opts.aiOpts.chat_model}`);
|
|
}
|
|
|
|
const engine = await createEngine({ engine: 'pglite' });
|
|
try {
|
|
await engine.connect({ database_path: dbPath, engine: 'pglite' });
|
|
|
|
// v0.28.5 (A4): refuse to silently re-template an existing brain with a
|
|
// mismatched embedding dimension. Loud failure beats the v0.27 silent-
|
|
// corruption pattern that surfaced as #673.
|
|
if (opts.aiOpts?.embedding_dimensions) {
|
|
const { readContentChunksEmbeddingDim, embeddingMismatchMessage } = await import('../core/embedding-dim-check.ts');
|
|
const existing = await readContentChunksEmbeddingDim(engine);
|
|
if (existing.exists && existing.dims !== null && existing.dims !== opts.aiOpts.embedding_dimensions) {
|
|
console.error('\n' + embeddingMismatchMessage({
|
|
currentDims: existing.dims,
|
|
requestedDims: opts.aiOpts.embedding_dimensions,
|
|
requestedModel: opts.aiOpts.embedding_model,
|
|
source: 'init',
|
|
}) + '\n');
|
|
if (opts.jsonOutput) {
|
|
console.log(JSON.stringify({
|
|
status: 'error',
|
|
reason: 'embedding_dim_mismatch',
|
|
current_dims: existing.dims,
|
|
requested_dims: opts.aiOpts.embedding_dimensions,
|
|
}));
|
|
}
|
|
process.exit(1);
|
|
}
|
|
}
|
|
|
|
await engine.initSchema();
|
|
|
|
const config: GBrainConfig = {
|
|
engine: 'pglite',
|
|
database_path: dbPath,
|
|
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
|
|
...(opts.aiOpts?.embedding_model ? { embedding_model: opts.aiOpts.embedding_model } : {}),
|
|
...(opts.aiOpts?.embedding_dimensions ? { embedding_dimensions: opts.aiOpts.embedding_dimensions } : {}),
|
|
...(opts.aiOpts?.expansion_model ? { expansion_model: opts.aiOpts.expansion_model } : {}),
|
|
...(opts.aiOpts?.chat_model ? { chat_model: opts.aiOpts.chat_model } : {}),
|
|
};
|
|
saveConfig(config);
|
|
|
|
const stats = await engine.getStats();
|
|
|
|
if (opts.jsonOutput) {
|
|
console.log(JSON.stringify({ status: 'success', engine: 'pglite', path: dbPath, pages: stats.page_count }));
|
|
} else {
|
|
console.log(`\nBrain ready at ${dbPath}`);
|
|
console.log(`${stats.page_count} pages. Engine: PGLite (local Postgres).`);
|
|
if (stats.page_count > 0) {
|
|
console.log('');
|
|
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
|
|
console.log(' gbrain extract links --source db (typed link backfill)');
|
|
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
|
|
console.log(' gbrain stats (verify links > 0)');
|
|
} else {
|
|
console.log('Next: gbrain import <dir>');
|
|
}
|
|
console.log('');
|
|
console.log('When you outgrow local: gbrain migrate --to supabase');
|
|
reportModStatus();
|
|
const { printAdvisoryIfRecommended } = await import('../core/skillpack/post-install-advisory.ts');
|
|
const { VERSION } = await import('../version.ts');
|
|
printAdvisoryIfRecommended({ version: VERSION, context: 'init' });
|
|
}
|
|
} finally {
|
|
try { await engine.disconnect(); } catch { /* best-effort */ }
|
|
}
|
|
}
|
|
|
|
async function initPostgres(opts: {
|
|
databaseUrl: string;
|
|
jsonOutput: boolean;
|
|
apiKey: string | null;
|
|
aiOpts?: { embedding_model?: string; embedding_dimensions?: number; expansion_model?: string; chat_model?: string };
|
|
}) {
|
|
const { databaseUrl } = opts;
|
|
|
|
// Configure AI gateway BEFORE initSchema so the vector column uses the right dim.
|
|
if (opts.aiOpts?.embedding_model || opts.aiOpts?.chat_model) {
|
|
const { configureGateway } = await import('../core/ai/gateway.ts');
|
|
configureGateway({
|
|
embedding_model: opts.aiOpts?.embedding_model,
|
|
embedding_dimensions: opts.aiOpts?.embedding_dimensions,
|
|
expansion_model: opts.aiOpts?.expansion_model,
|
|
chat_model: opts.aiOpts?.chat_model,
|
|
env: { ...process.env },
|
|
});
|
|
if (opts.aiOpts?.embedding_model) console.log(` Embedding: ${opts.aiOpts.embedding_model} (${opts.aiOpts.embedding_dimensions ?? '?'}d)`);
|
|
if (opts.aiOpts?.expansion_model) console.log(` Expansion: ${opts.aiOpts.expansion_model}`);
|
|
if (opts.aiOpts?.chat_model) console.log(` Chat: ${opts.aiOpts.chat_model}`);
|
|
}
|
|
|
|
// Detect Supabase direct connection URLs and warn about IPv6
|
|
if (databaseUrl.match(/db\.[a-z]+\.supabase\.co/) || databaseUrl.includes('.supabase.co:5432')) {
|
|
console.warn('');
|
|
console.warn('WARNING: You provided a Supabase direct connection URL (db.*.supabase.co:5432).');
|
|
console.warn(' Direct connections are IPv6 only and fail in many environments.');
|
|
console.warn(' Use the Session pooler connection string instead (port 6543):');
|
|
console.warn(' Supabase Dashboard > gear icon (Project Settings) > Database >');
|
|
console.warn(' Connection string > URI tab > change dropdown to "Session pooler"');
|
|
console.warn('');
|
|
}
|
|
|
|
console.log('Connecting to database...');
|
|
const engine = await createEngine({ engine: 'postgres' });
|
|
try {
|
|
try {
|
|
await engine.connect({ database_url: databaseUrl });
|
|
} catch (e: unknown) {
|
|
const msg = e instanceof Error ? e.message : String(e);
|
|
if (databaseUrl.includes('supabase.co') && (msg.includes('ECONNREFUSED') || msg.includes('ETIMEDOUT'))) {
|
|
console.error('Connection failed. Supabase direct connections (db.*.supabase.co:5432) are IPv6 only.');
|
|
console.error('Use the Session pooler connection string instead (port 6543).');
|
|
}
|
|
throw e;
|
|
}
|
|
|
|
// Check and auto-create pgvector extension
|
|
try {
|
|
const conn = (engine as any).sql || (await import('../core/db.ts')).getConnection();
|
|
const ext = await conn`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
|
|
if (ext.length === 0) {
|
|
console.log('pgvector extension not found. Attempting to create...');
|
|
try {
|
|
await conn`CREATE EXTENSION IF NOT EXISTS vector`;
|
|
console.log('pgvector extension created successfully.');
|
|
} catch {
|
|
console.error('Could not auto-create pgvector extension. Run manually in SQL Editor:');
|
|
console.error(' CREATE EXTENSION vector;');
|
|
// Throw so the outer finally runs engine.disconnect() before we die.
|
|
throw new Error('pgvector extension missing');
|
|
}
|
|
}
|
|
} catch {
|
|
// Non-fatal
|
|
}
|
|
|
|
// v0.28.5 (A4): refuse to silently re-template an existing brain with a
|
|
// mismatched embedding dimension (mirror of the PGLite path above).
|
|
if (opts.aiOpts?.embedding_dimensions) {
|
|
const { readContentChunksEmbeddingDim, embeddingMismatchMessage } = await import('../core/embedding-dim-check.ts');
|
|
const existing = await readContentChunksEmbeddingDim(engine);
|
|
if (existing.exists && existing.dims !== null && existing.dims !== opts.aiOpts.embedding_dimensions) {
|
|
console.error('\n' + embeddingMismatchMessage({
|
|
currentDims: existing.dims,
|
|
requestedDims: opts.aiOpts.embedding_dimensions,
|
|
requestedModel: opts.aiOpts.embedding_model,
|
|
source: 'init',
|
|
}) + '\n');
|
|
if (opts.jsonOutput) {
|
|
console.log(JSON.stringify({
|
|
status: 'error',
|
|
reason: 'embedding_dim_mismatch',
|
|
current_dims: existing.dims,
|
|
requested_dims: opts.aiOpts.embedding_dimensions,
|
|
}));
|
|
}
|
|
process.exit(1);
|
|
}
|
|
}
|
|
|
|
console.log('Running schema migration...');
|
|
await engine.initSchema();
|
|
|
|
const config: GBrainConfig = {
|
|
engine: 'postgres',
|
|
database_url: databaseUrl,
|
|
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
|
|
...(opts.aiOpts?.embedding_model ? { embedding_model: opts.aiOpts.embedding_model } : {}),
|
|
...(opts.aiOpts?.embedding_dimensions ? { embedding_dimensions: opts.aiOpts.embedding_dimensions } : {}),
|
|
...(opts.aiOpts?.expansion_model ? { expansion_model: opts.aiOpts.expansion_model } : {}),
|
|
...(opts.aiOpts?.chat_model ? { chat_model: opts.aiOpts.chat_model } : {}),
|
|
};
|
|
saveConfig(config);
|
|
console.log('Config saved to ~/.gbrain/config.json');
|
|
|
|
const stats = await engine.getStats();
|
|
|
|
if (opts.jsonOutput) {
|
|
console.log(JSON.stringify({ status: 'success', engine: 'postgres', pages: stats.page_count }));
|
|
} else {
|
|
console.log(`\nBrain ready. ${stats.page_count} pages. Engine: Postgres (Supabase).`);
|
|
if (stats.page_count > 0) {
|
|
console.log('');
|
|
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
|
|
console.log(' gbrain extract links --source db (typed link backfill)');
|
|
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
|
|
console.log(' gbrain stats (verify links > 0)');
|
|
} else {
|
|
console.log('Next: gbrain import <dir>');
|
|
}
|
|
reportModStatus();
|
|
const { printAdvisoryIfRecommended } = await import('../core/skillpack/post-install-advisory.ts');
|
|
const { VERSION } = await import('../version.ts');
|
|
printAdvisoryIfRecommended({ version: VERSION, context: 'init' });
|
|
}
|
|
} finally {
|
|
try { await engine.disconnect(); } catch { /* best-effort */ }
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Quick count of .md files in a directory (stops early at 1000).
|
|
*/
|
|
function countMarkdownFiles(dir: string, maxScan = 1500): number {
|
|
let count = 0;
|
|
try {
|
|
const scan = (d: string) => {
|
|
if (count >= maxScan) return;
|
|
for (const entry of readdirSync(d)) {
|
|
if (count >= maxScan) return;
|
|
if (entry.startsWith('.') || entry === 'node_modules') continue;
|
|
const full = join(d, entry);
|
|
try {
|
|
let stat;
|
|
try {
|
|
stat = lstatSync(full);
|
|
} catch { continue; }
|
|
if (stat.isSymbolicLink()) continue;
|
|
if (stat.isDirectory()) scan(full);
|
|
else if (entry.endsWith('.md')) count++;
|
|
} catch { /* skip unreadable */ }
|
|
}
|
|
};
|
|
scan(dir);
|
|
} catch { /* skip unreadable root */ }
|
|
return count;
|
|
}
|
|
|
|
async function supabaseWizard(): Promise<string> {
|
|
try {
|
|
execSync('bunx supabase --version', { stdio: 'pipe' });
|
|
console.log('Supabase CLI detected.');
|
|
console.log('To auto-provision, run: bunx supabase login && bunx supabase projects create');
|
|
console.log('Then use: gbrain init --url <your-connection-string>');
|
|
} catch {
|
|
console.log('Supabase CLI not found.');
|
|
}
|
|
|
|
console.log('\nEnter your Supabase/Postgres connection URL:');
|
|
console.log(' Format: postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:6543/postgres'); /* allow-pg-url-literal */
|
|
console.log(' Find it: Supabase Dashboard > Connect (top bar) > Connection String > Session Pooler\n');
|
|
|
|
const url = await readLine('Connection URL: ');
|
|
if (!url) {
|
|
console.error('No URL provided.');
|
|
process.exit(1);
|
|
}
|
|
return url;
|
|
}
|
|
|
|
function readLine(prompt: string): Promise<string> {
|
|
return new Promise((resolve) => {
|
|
process.stdout.write(prompt);
|
|
let data = '';
|
|
process.stdin.setEncoding('utf-8');
|
|
process.stdin.once('data', (chunk) => {
|
|
data = chunk.toString().trim();
|
|
process.stdin.pause();
|
|
resolve(data);
|
|
});
|
|
process.stdin.resume();
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Detect GStack installation across known host paths.
|
|
* Uses gstack-global-discover if available, falls back to path checking.
|
|
*/
|
|
export function detectGStack(): { found: boolean; path: string | null; host: string | null } {
|
|
// Try gstack's own discovery tool first (DRY: don't reimplement host detection)
|
|
try {
|
|
const result = execSync(
|
|
`${join(homedir(), '.claude', 'skills', 'gstack', 'bin', 'gstack-global-discover')} 2>/dev/null`,
|
|
{ encoding: 'utf-8', timeout: 5000 }
|
|
).trim();
|
|
if (result) {
|
|
return { found: true, path: result.split('\n')[0], host: 'auto-detected' };
|
|
}
|
|
} catch { /* binary not available */ }
|
|
|
|
// Fallback: check known host paths
|
|
const hostPaths = [
|
|
{ path: join(homedir(), '.claude', 'skills', 'gstack'), host: 'claude' },
|
|
{ path: join(homedir(), '.openclaw', 'skills', 'gstack'), host: 'openclaw' },
|
|
{ path: join(homedir(), '.codex', 'skills', 'gstack'), host: 'codex' },
|
|
{ path: join(homedir(), '.factory', 'skills', 'gstack'), host: 'factory' },
|
|
{ path: join(homedir(), '.kiro', 'skills', 'gstack'), host: 'kiro' },
|
|
];
|
|
|
|
for (const { path, host } of hostPaths) {
|
|
if (existsSync(join(path, 'SKILL.md')) || existsSync(join(path, 'setup'))) {
|
|
return { found: true, path, host };
|
|
}
|
|
}
|
|
|
|
return { found: false, path: null, host: null };
|
|
}
|
|
|
|
/**
|
|
* Install default identity templates (SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md)
|
|
* into the agent workspace. Uses minimal defaults, not the soul-audit interview.
|
|
*/
|
|
export function installDefaultTemplates(workspaceDir: string): string[] {
|
|
const gbrainRoot = dirname(dirname(__dirname)); // up from src/commands/ to repo root
|
|
const templatesDir = join(gbrainRoot, 'templates');
|
|
const installed: string[] = [];
|
|
|
|
const templates = [
|
|
{ src: 'SOUL.md.template', dest: 'SOUL.md' },
|
|
{ src: 'USER.md.template', dest: 'USER.md' },
|
|
{ src: 'ACCESS_POLICY.md.template', dest: 'ACCESS_POLICY.md' },
|
|
{ src: 'HEARTBEAT.md.template', dest: 'HEARTBEAT.md' },
|
|
];
|
|
|
|
for (const { src, dest } of templates) {
|
|
const srcPath = join(templatesDir, src);
|
|
const destPath = join(workspaceDir, dest);
|
|
if (existsSync(srcPath) && !existsSync(destPath)) {
|
|
mkdirSync(dirname(destPath), { recursive: true });
|
|
copyFileSync(srcPath, destPath);
|
|
installed.push(dest);
|
|
}
|
|
}
|
|
|
|
return installed;
|
|
}
|
|
|
|
/**
|
|
* Report post-init status including GStack detection and skill count.
|
|
*/
|
|
export function reportModStatus(): void {
|
|
const gstack = detectGStack();
|
|
const gbrainRoot = dirname(dirname(__dirname));
|
|
const skillsDir = join(gbrainRoot, 'skills');
|
|
|
|
let skillCount = 0;
|
|
try {
|
|
const manifest = JSON.parse(
|
|
readFileSync(join(skillsDir, 'manifest.json'), 'utf-8')
|
|
);
|
|
skillCount = manifest.skills?.length || 0;
|
|
} catch { /* manifest not found */ }
|
|
|
|
console.log('');
|
|
console.log('--- GBrain Mod Status ---');
|
|
console.log(`Skills: ${skillCount} loaded`);
|
|
console.log(`GStack: ${gstack.found ? `found (${gstack.host})` : 'not found'}`);
|
|
if (!gstack.found) {
|
|
console.log(' Install GStack for coding skills:');
|
|
console.log(' git clone https://github.com/garrytan/gstack.git ~/.claude/skills/gstack');
|
|
console.log(' cd ~/.claude/skills/gstack && ./setup');
|
|
}
|
|
console.log('Resolver: skills/RESOLVER.md');
|
|
console.log('Soul audit: run `gbrain soul-audit` to customize agent identity');
|
|
console.log('');
|
|
}
|