mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.42.14.0 fix(zero-config): code-* readiness signal + init embedding-key validation + lock self-heal (#1780) (#1804)
* feat(code-intel): readiness signal on code-def/refs/callers/callees (#1780 Gap 1) New src/core/code-graph-readiness.ts: resolveCodeReadiness() returns a typed status (not_built | indexing | ready | unknown) + ready boolean so callers can tell "graph not built / still indexing" apart from "genuinely no match" when count===0. EXISTS-based (cheap), chunk-grain, resolver-version-matching pending predicate, fail-open. Wired into the 4 CLI envelopes (+ human hint) and the 4 MCP op handlers. def/refs are 2-state brain-wide; callers/callees 3-state scoped. * feat(db-lock): automatic same-host dead-pid cycle-lock takeover (#1780 Gap 3) tryAcquireDbLock now reclaims a held, not-TTL-expired lock when the same-host holder is provably dead (process.kill ESRCH) past a 60s grace, via guarded DELETE + one normal-upsert retry returning the normal handle. New shared injectable classifyHolderLiveness/isHolderDeadLocally (EPERM treated as ALIVE — never steals a live lock). runBreakLock's safe path consumes the shared predicate, fixing its prior EPERM-as-dead bug. Cross-host stays TTL-only. * feat(init): validate the embedding key at gbrain init (#1780 Gap 2) New src/core/init-embed-check.ts: config-only diagnoseEmbedding (missing key, all providers) + best-effort 1-token live test-embed (invalid/expired key, 5s timeout, never blocks). Loud warning to stderr, init still exits 0; skipped by --no-embedding / --skip-embed-check / GBRAIN_INIT_SKIP_EMBED_CHECK=1. Builds the effective env (process.env + file-plane keys + --key) via buildGatewayConfig, extracted to src/core/ai/build-gateway-config.ts (cli.ts re-exports) so the check sees the same keys + provider base URLs as runtime. embedding_check added to --json. * chore: bump version and changelog (v0.42.14.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document the #1780 zero-config gaps for v0.42.14.0 CLAUDE.md Key Files: add src/core/code-graph-readiness.ts, init-embed-check.ts, ai/build-gateway-config.ts, and the db-lock auto-takeover + code-* readiness field behaviors. Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
bea2d3e6c9
commit
1036f8f752
@@ -2,6 +2,53 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.14.0] - 2026-06-02
|
||||
|
||||
**Two zero-config gaps closed: code-* queries now tell you whether the graph is built, and `gbrain init` tells you up front when your embedding key is missing.**
|
||||
|
||||
Ask `gbrain code-callers foo` and get nothing back, and until now you had no way to know whether that meant "this symbol genuinely has no callers" or "the code graph isn't built yet." Same for `code-def`, `code-refs`, `code-callees`. An agent would see `count: 0` and confidently conclude "no callers" while the source was still indexing or had never been synced. This release adds a typed readiness field so the empty answer is honest.
|
||||
|
||||
```
|
||||
gbrain code-callers parseMarkdown --json
|
||||
# count: 0, status: "not_built", ready: false → no code indexed; run `gbrain sync`
|
||||
# count: 0, status: "indexing", ready: false → edges still resolving; retry after `gbrain dream`
|
||||
# count: 0, status: "ready", ready: true → genuinely no callers, trust it
|
||||
```
|
||||
|
||||
`count: 0 + ready: true` means "genuinely none." `ready: false` means "ask later." Both the CLI and the MCP tools (`code_def`, `code_refs`, `code_callers`, `code_callees`) carry the field; human output prints a one-line hint telling you exactly what to run. `code-def`/`code-refs` are ready as soon as code is synced (their data is set at chunk time); `code-callers`/`code-callees` also report `indexing` until the call graph is resolved.
|
||||
|
||||
**`gbrain init` now checks your embedding key before first sync.** Before, init happily saved `--embedding-model openai:...` without ever checking the key was set; then your first `gbrain sync` imported every page but embedded zero of them, and search came back empty. Now init runs a free config check (is the key present, for any provider?) plus a tiny test embed (does the key actually work?) and warns loudly if either fails:
|
||||
|
||||
```
|
||||
Heads up: embedding is configured but not ready.
|
||||
Model "openai:text-embedding-3-large" needs OPENAI_API_KEY — not set in your shell or ~/.gbrain/config.json.
|
||||
```
|
||||
|
||||
Init still exits 0 so deferred setup keeps working. The check correctly sees keys set in `~/.gbrain/config.json` (not just the shell), and `--no-embedding` or the new `--skip-embed-check` skip it.
|
||||
|
||||
**Crashed cycle locks self-heal faster.** If a `gbrain dream`/sync process crashed while holding the cycle lock, the next run waited out the full 30-minute TTL. Now, when the dead holder is on the same machine and provably gone, the lock is reclaimed automatically after a 60-second grace (a guard against PID reuse). Cross-host locks stay TTL-only. `gbrain sync --break-lock` got the same liveness fix — it no longer treats a permission-denied probe (a live process you don't own) as dead.
|
||||
|
||||
## To take advantage of v0.42.14.0
|
||||
|
||||
`gbrain upgrade` handles everything — no schema migration in this release.
|
||||
|
||||
1. **Readiness:** `gbrain code-callers <symbol> --json` and read the new `status` / `ready` fields. `ready: false` means wait and retry; `ready: true` with `count: 0` means genuinely none.
|
||||
2. **Init check:** next time you run `gbrain init` with an embedding model, a missing or invalid key warns immediately. Set the key and re-run `gbrain sync`, or pass `--no-embedding` to defer.
|
||||
3. **Lock self-heal is automatic** — nothing to configure.
|
||||
|
||||
If anything looks off, file an issue with the output of `gbrain doctor`: https://github.com/garrytan/gbrain/issues
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **`src/core/code-graph-readiness.ts` (new)** — `resolveCodeReadiness(engine, {kind, count, sourceId?, allSources?})` returns `{status: 'not_built' | 'indexing' | 'ready' | 'unknown', ready, has_code, pending_edges}`. `count > 0` short-circuits to `ready` with no query; on empty it runs `EXISTS` probes (no `page_kind` index needed; the pending probe rides the partial `idx_content_chunks_edges_backfill`). `kind: 'symbol'` (code-def/refs) is 2-state and brain-wide; `kind: 'edge'` (callers/callees) is 3-state and source-scoped, with the pending predicate mirroring the resolver (`edges_backfilled_at IS NULL OR < EDGE_EXTRACTOR_VERSION_TS`) so a resolver-version bump doesn't falsely report `ready`. Scope matches the result query's `deleted_at` posture; any DB error returns `unknown` (fail-open). `readinessHint()` renders the human one-liner.
|
||||
- **`src/commands/code-def.ts`, `code-refs.ts`, `code-callers.ts`, `code-callees.ts`** — each JSON envelope gains `status` + `ready`; human output prints the hint when not ready. callers/callees pass their resolved `sourceId` / `allSources`; def/refs query brain-wide.
|
||||
- **`src/core/operations.ts`** — the four `code_*` MCP op handlers stamp `status` + `ready` on their result envelopes.
|
||||
- **`src/core/init-embed-check.ts` (new)** — `runInitEmbedCheck()` builds the effective env (process.env + file-plane `openai/anthropic/zeroentropy_api_key` + `--key`), reconfigures the gateway via `buildGatewayConfig`, runs `diagnoseEmbedding` (config-only), then a best-effort `liveTestEmbed` (1 token, 5s `AbortController` timeout, never throws). Init-specific warning names `--no-embedding` / `--skip-embed-check`.
|
||||
- **`src/core/ai/build-gateway-config.ts` (new)** — `buildGatewayConfig` extracted from `src/cli.ts` (which now re-exports it) so core modules reuse it without importing the CLI entrypoint. Folds file-plane API keys + provider base URLs into the gateway config; `process.env` wins.
|
||||
- **`src/commands/init.ts`** — new `--skip-embed-check` flag (also `GBRAIN_INIT_SKIP_EMBED_CHECK=1`); replaces the prior ZeroEntropy-only warning in both the PGLite and Postgres paths with the generalized check; `embedding_check {ok, reason?, live_ok?}` added to the `--json` success envelope; help text updated.
|
||||
- **`src/core/db-lock.ts`** — `tryAcquireDbLock` adds same-host dead-pid auto-takeover (guarded `DELETE WHERE id=$1 AND holder_pid=$2` + one normal-upsert retry returning the standard handle). New exported `classifyHolderLiveness` / `isHolderDeadLocally` (injectable `process.kill` seam; `HOLDER_TAKEOVER_GRACE_MS = 60_000`; EPERM classified as `alive`, never reclaimed). TTL-expired locks stay the upsert's job; cross-host stays TTL-only.
|
||||
- **`src/commands/sync.ts`** — `runBreakLock`'s safe path consumes the shared `classifyHolderLiveness` predicate, fixing the prior bug where any `process.kill` throw (including EPERM) counted the holder as dead.
|
||||
- **Tests** — `test/code-graph-readiness.test.ts` (11), `test/db-lock-auto-takeover.test.ts` (11), `test/init-embed-check.test.ts` (9, hermetic via the gateway embed-transport seam + `withEnv`), plus readiness-envelope cases added to `test/e2e/code-intel-mcp-ops-pglite.test.ts`. Closes #1780.
|
||||
## [0.42.13.0] - 2026-06-02
|
||||
|
||||
**Pages under `archive/` are findable again.** If you committed a note to your brain under `archive/` (old conversation exports, prior-system logs, notes you filed away), GBrain was embedding it and graphing it but then hiding it from every search. You would search for an exact phrase you knew was in the page, get nothing back, and conclude the page did not exist. It did. A hardcoded list was quietly dropping the whole `archive/` subtree from results unless you knew to pass a special flag.
|
||||
|
||||
@@ -4006,3 +4006,27 @@ Start at `probeChatModel` in `src/core/ai/gateway.ts` and the explicit gate in
|
||||
|
||||
**Depends on:** a config-independent provider-general key probe (new gateway
|
||||
helper) so the `isAvailable` unconfigured-gateway false-reject footgun is avoided.
|
||||
|
||||
## v0.42.14.0 follow-ups (#1780)
|
||||
|
||||
### Unify the init live-test-embed with the models-doctor reachability probe
|
||||
**Priority:** P3
|
||||
|
||||
**What:** `src/core/init-embed-check.ts:liveTestEmbed` and
|
||||
`src/commands/models.ts:probeEmbeddingReachability` both do the same thing —
|
||||
a 1-token `gateway.embed(['probe'], {inputType:'query', abortSignal})` with a 5s
|
||||
timeout + error classification. They were left as two small implementations
|
||||
because `probeEmbeddingReachability` is private and returns the doctor-shaped
|
||||
`ProbeResult`, while the init path wants `{ok, reason, message}`.
|
||||
|
||||
**Why:** rule-of-three is met (init check + models doctor + the classifyError
|
||||
duplication). One shared embed-probe core would prevent the two from drifting
|
||||
on timeout/classification behavior.
|
||||
|
||||
**How to start:** extract the embed + AbortController-timeout + error-classify
|
||||
core into a shared helper (e.g. `src/core/ai/embed-probe.ts`), have both
|
||||
`liveTestEmbed` and `probeEmbeddingReachability` adapt its result to their
|
||||
respective shapes. Small, mechanical; pinned by `test/init-embed-check.test.ts`
|
||||
+ the models-doctor tests.
|
||||
|
||||
**Depends on:** nothing.
|
||||
|
||||
@@ -47,7 +47,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/chunkers/` — 3-tier chunking (recursive, semantic, LLM-guided). `code.ts` is a tree-sitter-based semantic chunker for 30 languages (plus SQL via DerekStride/tree-sitter-sql) with embedded-asset WASMs (`src/assets/wasm/`), `@dqbd/tiktoken` cl100k_base tokenizer, small-sibling merging. `CHUNKER_VERSION` is folded into `importCodeFile`'s `content_hash` so chunker shape changes force clean re-chunks across releases. `extractSymbolName` has an inline SQL branch (`extractSqlSymbolName`) diving through DerekStride's `statement` wrapper into the inner DDL child (`create_table`/`create_function`/`create_view`/`create_index`/`create_procedure`/`create_type`/`create_schema`/`create_database`/`create_trigger`/`alter_table`/`alter_view`) and extracting the target identifier via the `name` field with identifier-shaped fallback; DML kinds (`select`/`insert`/`update`/`delete`/`merge`/`with`) deliberately return null so chunks emit unnamed (code-def is a DDL signal). `normalizeSymbolType` has parallel SQL branches mapping `create_table → 'table'`, `create_view → 'view'`, etc. `src/commands/code-def.ts:DEF_TYPES` is extended with `'table' | 'view' | 'index' | 'procedure' | 'schema' | 'database' | 'trigger'` so the new chunks surface in `gbrain code-def <name>` queries.
|
||||
- `src/core/errors.ts` — `StructuredAgentError` + `buildError` + `serializeError`. Every agent-facing surface (code-def, code-refs, usage errors) uses this envelope; matches the `CycleReport.PhaseResult.error` shape.
|
||||
- `src/assets/wasm/` — 37 tree-sitter grammar WASMs + tree-sitter runtime. Committed to the repo so `bun --compile` embeds them deterministically via `import path from ... with { type: 'file' }`. The CI guard `scripts/check-wasm-embedded.sh` fails the build if the compiled binary ever silently falls through to recursive chunks. `tree-sitter-sql.wasm` (DerekStride/tree-sitter-sql @ c2e1e08db1ea20dc23bdb8d228a81a8756e9c450, built with tree-sitter-cli@v0.26.3 --abi 14) adds SQL coverage at 11 MB — larger than peers because the grammar covers PostgreSQL + MySQL + SQLite + T-SQL basics (40 MB generated parser.c); the compiled binary grows ~6%.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface.
|
||||
- `src/commands/code-def.ts` + `src/commands/code-refs.ts` — symbol definition + references lookup. Query `content_chunks.symbol_name` or chunk_text ILIKE with `page_kind='code'` filter. Auto-JSON when stdout is not a TTY (gh-CLI convention). Bypass the standard `searchKeyword` `DISTINCT ON (slug)` collapse so multiple call-sites from the same file surface. The JSON envelope (CLI + the `code_def`/`code_refs` MCP ops) carries `status` + `ready` from `src/core/code-graph-readiness.ts` so a `count:0` result is distinguishable as `not_built` (no code indexed) vs `ready` (genuinely no match); human output prints a one-line hint when not ready.
|
||||
- `src/core/code-graph-readiness.ts` — typed readiness signal shared by the four code-* surfaces (`code-def`/`code-refs`/`code-callers`/`code-callees`). `resolveCodeReadiness(engine, {kind:'symbol'|'edge', count, sourceId?, allSources?})` returns `{status:'not_built'|'indexing'|'ready'|'unknown', ready, has_code, pending_edges}`. `count>0` short-circuits to `ready` with no query; on empty it runs `EXISTS` probes against `content_chunks` JOIN `pages` (`page_kind='code'`) — no `page_kind` index needed, and the pending probe rides the partial `idx_content_chunks_edges_backfill`. `kind:'symbol'` (code-def/refs) is 2-state + brain-wide because symbol metadata is set at chunk time; `kind:'edge'` (code-callers/callees) is 3-state + source-scoped, with the pending predicate mirroring the resolver (`edges_backfilled_at IS NULL OR < EDGE_EXTRACTOR_VERSION_TS` from `src/core/chunkers/symbol-resolver.ts`) so a resolver-version bump never falsely reports `ready`. Probe scope matches each command's result-query `deleted_at` posture (def/refs don't filter `deleted_at`, so neither do the probes). Any DB error returns `status:'unknown'` (fail-open; never breaks the command). `readinessHint(r)` renders the human one-liner. Wired into `code-def.ts`/`code-refs.ts` (brain-wide), `code-callers.ts`/`code-callees.ts` (resolved `sourceId`/`allSources`), and all four `code_*` MCP op handlers in `src/core/operations.ts`. Pinned by `test/code-graph-readiness.test.ts` + readiness-envelope cases in `test/e2e/code-intel-mcp-ops-pglite.test.ts`.
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `<fork>/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally.
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level).
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator.
|
||||
@@ -116,6 +117,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/ai/gateway.ts` extension — module-scoped `_extendedModels: Map<providerId, Set<modelId>>` registry feeds `assertTouchpoint`'s 4th-arg path. `reconfigureGatewayWithEngine(engine)` (async, called from `cli.ts` after `engine.connect()`, before every command except `CLI_ONLY` no-DB commands) re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to both. `DEFAULT_CHAT_MODEL` is `anthropic:claude-sonnet-4-6`. `__setChatTransportForTests` seam mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport.
|
||||
- `src/core/minions/queue.ts` extension — `MinionQueue.add()` rejects `subagent` jobs whose `data.model` resolves via `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (layers 2+3: `model-config.ts:enforceSubagentAnthropic` runtime fallback + `src/commands/doctor.ts` `subagent_provider` check). Pinned by `test/agent-cli.test.ts`.
|
||||
- `src/commands/models.ts` — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain), every per-task override (11 `PER_TASK_KEYS`: `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map, and a source-of-truth column (`default` / `config: <key>` / `env: <VAR>`). `gbrain models doctor [--skip=<provider>] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}`. Wired into `cli.ts` dispatch + `CLI_ONLY` set. A zero-token `embedding_config` probe runs FIRST, before any chat/expansion probes spend money: `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. `ProbeStatus` variant `'config'` + optional `fix?: string` on `ProbeResult` surface a paste-ready `gbrain config set ...` line in human + JSON output; touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'`.
|
||||
- `src/core/init-embed-check.ts` — embedding-key validation at `gbrain init`. `runInitEmbedCheck(opts)` runs a config-only `diagnoseEmbedding` (catches a missing key for ANY provider) plus a best-effort `liveTestEmbed` (1-token `gateway.embed(['probe'], {inputType:'query', abortSignal})`, 5s `AbortController` timeout, never throws — catches an invalid/expired key). Loud warning to stderr; init still exits 0 (`--no-embedding` is the deferred-setup escape; `--skip-embed-check` / `GBRAIN_INIT_SKIP_EMBED_CHECK=1` skip the check). Builds the effective env (`process.env` + file-plane `openai/anthropic/zeroentropy_api_key` from `loadConfigFileOnly()` + `opts.apiKey`) and configures the gateway via `buildGatewayConfig` before diagnose/probe, so the check sees the same keys AND provider base URLs runtime will (no false "missing key" for config.json-keyed users; the probe hits the right endpoint). Init-specific warning text names `--no-embedding` / `--skip-embed-check`, not the sync-flavored `--no-embed`. Wired into `initPGLite` + `initPostgres` in `src/commands/init.ts`, with the result added to the `--json` envelope as `embedding_check {ok, reason?, live_ok?}`. Pinned by `test/init-embed-check.test.ts` (hermetic via the gateway embed-transport seam + `withEnv`).
|
||||
- `src/core/ai/build-gateway-config.ts` — `buildGatewayConfig(c: GBrainConfig): AIGatewayConfig`, extracted from `src/cli.ts` (which re-exports it for back-compat). Lets core modules (`init-embed-check.ts`) reuse it without importing the CLI entrypoint. Single owner of folding file-plane API keys (openai/anthropic/zeroentropy) into the gateway env and threading local-server `*_BASE_URL` env vars into base_urls; `process.env` wins. Pinned by `test/ai/build-gateway-config.test.ts`.
|
||||
- `src/commands/doctor.ts` extension — `subagent_provider` check (layer 3 of 3). Warns when `models.tier.subagent` is explicitly set non-Anthropic (message names the bad value + paste-ready fix `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK when subagent tier resolves to Anthropic. Tests in `test/doctor.test.ts`.
|
||||
- `src/core/skill-trigger-index.ts` — Shared loader that unions per-skill SKILL.md frontmatter `triggers:` with curated RESOLVER.md / AGENTS.md rows from `skillsDir` AND the parent dir (preserves the OpenClaw workspace-root layout). UNION semantics: explicit RESOLVER.md rows ADD to frontmatter triggers (don't replace). Dedup keyed on `(skillPath, trigger.trim().toLowerCase())`. Three consumers fold through this primitive — `checkResolvable`, `runRoutingEvalCli`, `mounts-cache.composeResolvers` — so fixing frontmatter reaches all of them. Exports `loadSkillTriggerIndex(skillsDir): SkillTriggerEntry[]`, `entriesToResolverContent(entries): string` (synthesizes a markdown-table resolver string for `runRoutingEval`'s string-content API), `findPrimaryResolverPath(skillsDir): string | null`, the `FRONTMATTER_SECTION` constant, and `_resetWarnedSkillsForTests`. Skip rules: non-directory entries, `_*`/`.*` prefixes, `conventions/`+`migrations/` subdirs, skills with no `SKILL.md` (deprecated `install/` graceful-skipped), no `triggers:` array, or malformed YAML (warn-once + skip). Reuses `parseSkillFrontmatter` from `src/core/skill-frontmatter.ts` (regex-based, not full YAML). Pinned by `test/skill-trigger-index.test.ts` (18 hermetic cases). CI gate `bun run check:resolver` (= `bun src/cli.ts check-resolvable --strict --skills-dir skills/`) wired into `bun run verify`.
|
||||
- `src/core/skill-catalog.ts` — host-repo skill catalog backing the MCP `list_skills` / `get_skill` ops. Lets a thin MCP client (Codex desktop, Claude Code, Claude Cowork, Perplexity) DISCOVER + FOLLOW the agent repo's fat-markdown skills over `gbrain serve` — a skill is prose, so "using" one = fetching its body then calling the gbrain MCP tools the server already exposes. Read-scope, NOT localOnly (defensible only via the full mitigation stack): (1) **publish gate** — `assertPublishEnabled(ctx, publishSkills)`; remote callers require `mcp.publish_skills === true`, default-OFF so an upgrade never silently grants existing read tokens host-skill read; local callers (`ctx.remote === false`) always pass. (2) **path confinement** — `assertSkillNameShape` rejects separators/`..`/null/space before any FS access; the client `name` is a manifest LOOKUP KEY (via `loadOrDeriveManifest`), never a raw path segment; `confineManifestPath` does realpath + relative-containment + `SKILL.md`-regular-file check on EVERY entry (defeats poisoned manifest.json `path`, symlink/`..` escape). (3) **frontmatter allowlist** — `GetSkillResult.frontmatter` projects a safe subset; private `writes_to` + `sources` dropped. (4) **prose-only + 256KB cap** (`MAX_SKILL_MD_BYTES`, env `GBRAIN_MAX_SKILL_MD_BYTES`), size-checked twice (statSync + UTF-8 byte length). (5) **no install_path serve for remote** — remote callers use `autoDetectSkillsDir` (no install-path tier) so a hosted gbrain with no agent repo returns `storage_error`; local callers use `autoDetectSkillsDirReadOnly`. (6) MCP rate-limiter caps call rate. Config reads honor BOTH planes: `readMcpPublishSkills` / `readMcpSkillsDir` prefer the DB plane (`engine.getConfig`) over the file plane (`ctx.config.mcp`). Tool-honesty: `crossReferenceTools(declared, ctx)` splits a skill's declared `tools:` into `usable_tools` vs `unavailable_tools`; `buildSkillCatalog`'s `instructions` envelope (`SKILL_CATALOG_INSTRUCTIONS`) carries the "these are prose, follow-then-call-tools" protocol. Skills are host-filesystem repo-global — `sourceScopeOpts(ctx)` / `ctx.brainId` deliberately do NOT apply. `buildSkillCatalog` is resilient (one malformed/escaping skill is skipped, never throws). Config keys in `src/core/config.ts`: `GBrainConfig.mcp?: { publish_skills?, skills_dir? }` + `KNOWN_CONFIG_KEYS` entries `mcp.publish_skills`/`mcp.publish_skills_prompted`/`mcp.skills_dir` + `mcp.` prefix in `KNOWN_CONFIG_KEY_PREFIXES`. `src/commands/init.ts` writes `config.mcp = { publish_skills: true, ... }` for new installs (existing config wins on re-init). `src/commands/upgrade.ts:runPostUpgrade` adds a one-time consent prompt (gated by `mcp.publish_skills_prompted`; existing installs stay OFF until owner opts in). Two ops register in `src/core/operations.ts` (`list_skills` with optional `section` filter + `cliHints:{name:'skills'}`; `get_skill` taking `name` + `cliHints:{name:'skill', positional:['name']}`) and dynamically import this module to avoid the import cycle (skill-catalog statically imports the `operations` array). Descriptions in `src/core/operations-descriptions.ts` (`LIST_SKILLS_DESCRIPTION`, `GET_SKILL_DESCRIPTION`, `SKILL_CATALOG_INSTRUCTIONS`, `SKILL_CLIENT_GUIDANCE`), pinned by `test/operations-descriptions.test.ts`. CLI: `gbrain skills` / `gbrain skill <name>`. Pinned by `test/skill-catalog.test.ts`, `test/skill-catalog-security.test.ts` (path-confinement / poisoned-manifest / symlink-escape), `test/skill-catalog-transports.test.ts` (publish-gate + remote-vs-local) over `test/fixtures/skill-catalog/`.
|
||||
@@ -261,7 +264,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY `\r`-rewriting; non-TTY plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. `emitHumanLine` is prefix-aware — inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts` it prepends `[id] ` (and TTY-rewrite mode `\r\x1b[2K` carries the prefix inside the clear-to-EOL escape); `emitJson` is intentionally NOT prefixed so NDJSON consumers don't choke on a `[id] {...}` shape.
|
||||
- `src/core/console-prefix.ts` — `AsyncLocalStorage<string>`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as active prefix; nested wraps replace then restore), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log`/`console.error`). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog`/`serr` fall through to bare `console.log`/`console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form) to defeat log-injection through newline/control-character names. Coverage: `src/commands/sync.ts` performSync + callees, `src/commands/embed.ts` runEmbedCore + helpers, `src/core/progress.ts` emitHumanLine.
|
||||
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
|
||||
- `src/core/db-lock.ts` — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the `gbrain_cycle_locks` table. Parameterized lock id so scopes nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID`) for `performSync`'s narrower writer window. UPSERT-with-TTL semantics survive PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
|
||||
- `src/core/db-lock.ts` — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the `gbrain_cycle_locks` table. Parameterized lock id so scopes nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID`) for `performSync`'s narrower writer window. UPSERT-with-TTL semantics survive PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. It also does automatic same-host dead-pid takeover: when the upsert finds a held, NOT-TTL-expired lock whose holder is on this host and provably dead, it reclaims via a guarded `DELETE WHERE id=$1 AND holder_pid=$2` + one normal-upsert retry returning the standard handle (refresh/release intact). The liveness check is the exported `classifyHolderLiveness(pid, host, ageMs, opts?)` / `isHolderDeadLocally(...)` (injectable `process.kill` seam; `HOLDER_TAKEOVER_GRACE_MS = 60_000` PID-reuse guard; EPERM classified as `alive` so a live process you don't own is never stolen). TTL-expired locks stay the upsert's job; cross-host stays TTL-only. `runBreakLock` (`src/commands/sync.ts`) consumes the same predicate. Pinned by `test/db-lock-auto-takeover.test.ts`.
|
||||
- `src/core/sync-concurrency.ts` — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the sites can't drift. `DEFAULT_PARALLEL_SOURCES = 4` is a SEPARATE constant for the per-source fan-out under `gbrain sync --all` — kept distinct from `DEFAULT_PARALLEL_WORKERS` because total live Postgres connections per wave ≈ `DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2 (per-file pool)` = 32 at both defaults (each per-file worker opens its own `PostgresEngine` with `poolSize = min(2, resolvePoolSize(2))`); `sync.ts` warns when `parallel × workers × 2 > 16`. `resolveWorkersWithClamp(engine, override, commandName, fileCount)` wraps `autoConcurrency` with a per-command stderr clamp warning on PGLite (per-(command, requested) dedup via module-scoped warned-once set with `_resetWorkersClampWarningsForTest()` seam) and is the canonical surface for every bulk-command `--workers N` flag (extract-conversation-facts, extract, edges-backfill, reindex-multimodal, reindex, reindex-code); embed.ts deliberately bypasses it and keeps `GBRAIN_EMBED_CONCURRENCY || 20`. Pinned by `test/pglite-workers-clamp.test.ts`.
|
||||
- `src/core/worker-pool.ts` — Canonical sliding-pool + bounded-semaphore primitive (extracted from `src/commands/embed.ts` sliding-pool sites and `src/commands/eval-cross-modal.ts` `runWithLimit` semaphore). Two exports: `runSlidingPool<T>({items, workers, onItem, signal?, onError?, failureLabel?, onProgress?})` + `runWithLimit<TIn, TOut>({items, limit, fn, signal?})`. Atomicity invariant: `const idx = nextIdx++` is one synchronous JS statement (no `await` between read and write — guaranteed by the single-threaded event loop), documented in the module header AND enforced by `scripts/check-worker-pool-atomicity.sh` (wired into `bun run verify`), which rejects importing `worker_threads` in any consuming file and inserting `await` between the `nextIdx` read and write. `MUST_ABORT_ERROR_TAGS` set is seeded with `BUDGET_EXHAUSTED` from `src/core/budget/budget-tracker.ts`; tagged errors (matched via `err.tag === 'BUDGET_EXHAUSTED'` to avoid cross-module import) bypass `onError` and hard-abort the pool via `AbortController.abort()` to in-flight `onItem` — the budget cap is a structural ceiling under concurrency. `failures[]` shape is `{idx, label, error}` records (NOT full items; callers supply `failureLabel(item) => string`) for bounded memory under huge brains. Pinned by `test/worker-pool.test.ts` + `test/scripts/check-worker-pool-atomicity.test.ts`. Drives every `--workers N` bulk command.
|
||||
- `src/commands/embed.ts` extension — both inline sliding-pool sites (`embedAll` simple at `:458-467` and `embedAllStale` paginated + AbortSignal at `:586-632`) call `runSlidingPool` from the shared worker-pool helper. Invariant-level contract preserved (counts + cost + AbortSignal propagation + per-batch rate-limit retry via `embedBatchWithBackoff`); byte-equality on progress-event ORDERING is NOT promised. The `GBRAIN_EMBED_CONCURRENCY || 20` default is preserved and embed bypasses `resolveWorkersWithClamp` because the 20-worker default would otherwise silently change every brain's embed hot path. Pinned by `test/embed-helper-migration.test.ts` (asserts the helper is wired in AND the pre-migration `let nextIdx = 0` + `Promise.all(Array.from({length: numWorkers}, ...))` shapes are gone).
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.13.0"
|
||||
"version": "0.42.14.0"
|
||||
}
|
||||
|
||||
+8
-50
@@ -1905,56 +1905,14 @@ async function dispatchReadOnlyCommand(engine: BrainEngine, command: string, arg
|
||||
|
||||
// Build the AIGatewayConfig payload from a GBrainConfig. Both configureGateway
|
||||
// sites in connectEngine() pass through this helper so adding a new field
|
||||
// touches one place. Adding a field to one site but not the other previously
|
||||
// required remembering to mirror the change; the helper makes that structural.
|
||||
// v0.37.6.0: exported so `test/ai/build-gateway-config.test.ts` can pin the
|
||||
// env-baseURL passthrough contract for every `_BASE_URL` env var the CLI
|
||||
// reads (LLAMA_SERVER, OLLAMA, LMSTUDIO, LITELLM, OPENROUTER).
|
||||
export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
|
||||
// v0.32 (#121 reworked): when ~/.gbrain/config.json declares
|
||||
// openai_api_key / anthropic_api_key, fold them into the gateway env so
|
||||
// recipes that read OPENAI_API_KEY / ANTHROPIC_API_KEY find them. Process
|
||||
// env still wins (it's loaded last) — this is a fallback for daemons /
|
||||
// launchd-spawned subprocesses that don't propagate ~/.zshrc-sourced keys.
|
||||
const envFromConfig: Record<string, string> = {};
|
||||
if (c.openai_api_key) envFromConfig.OPENAI_API_KEY = c.openai_api_key;
|
||||
if (c.anthropic_api_key) envFromConfig.ANTHROPIC_API_KEY = c.anthropic_api_key;
|
||||
// v0.37 fix wave (CDX2-5+6): ZE became the default provider in v0.36 but
|
||||
// the env-mapping at this seam never picked it up. `gbrain config set
|
||||
// zeroentropy_api_key X` wrote DB plane (ignored by gateway). The file-
|
||||
// plane field now exists (GBrainConfig type) and gets mapped here, so
|
||||
// setting it via `~/.gbrain/config.json` propagates into the gateway.
|
||||
if (c.zeroentropy_api_key) envFromConfig.ZEROENTROPY_API_KEY = c.zeroentropy_api_key;
|
||||
|
||||
// v0.32 codex finding #4+#5 fix: thread local-server _BASE_URL env vars
|
||||
// into base_urls so the gateway hits the user's configured port. Without
|
||||
// this, `LLAMA_SERVER_BASE_URL=http://localhost:9000` would let the probe
|
||||
// succeed against :9000 but the actual embed call would still go to the
|
||||
// recipe's base_url_default (localhost:8080). Same fix applies to
|
||||
// OLLAMA_BASE_URL. Caller-provided cfg.provider_base_urls wins.
|
||||
const envBaseUrls: Record<string, string> = {};
|
||||
if (process.env.LLAMA_SERVER_BASE_URL) envBaseUrls['llama-server'] = process.env.LLAMA_SERVER_BASE_URL;
|
||||
// v0.40.6.1: sibling recipe for llama-server in reranking mode. Separate
|
||||
// env var because --reranking and --embeddings are mutually exclusive at
|
||||
// server launch — users running both will have two llama-server processes
|
||||
// on different ports.
|
||||
if (process.env.LLAMA_SERVER_RERANKER_BASE_URL) envBaseUrls['llama-server-reranker'] = process.env.LLAMA_SERVER_RERANKER_BASE_URL;
|
||||
if (process.env.OLLAMA_BASE_URL) envBaseUrls['ollama'] = process.env.OLLAMA_BASE_URL;
|
||||
if (process.env.LMSTUDIO_BASE_URL) envBaseUrls['lmstudio'] = process.env.LMSTUDIO_BASE_URL;
|
||||
if (process.env.LITELLM_BASE_URL) envBaseUrls['litellm'] = process.env.LITELLM_BASE_URL;
|
||||
if (process.env.OPENROUTER_BASE_URL) envBaseUrls['openrouter'] = process.env.OPENROUTER_BASE_URL;
|
||||
|
||||
return {
|
||||
embedding_model: c.embedding_model,
|
||||
embedding_dimensions: c.embedding_dimensions,
|
||||
embedding_multimodal_model: c.embedding_multimodal_model,
|
||||
expansion_model: c.expansion_model,
|
||||
chat_model: c.chat_model,
|
||||
chat_fallback_chain: c.chat_fallback_chain,
|
||||
base_urls: { ...envBaseUrls, ...(c.provider_base_urls ?? {}) }, // config wins over env
|
||||
env: { ...envFromConfig, ...process.env }, // process.env wins
|
||||
};
|
||||
}
|
||||
// touches one place.
|
||||
// v0.42 (#1780): moved to src/core/ai/build-gateway-config.ts so core modules
|
||||
// (init-embed-check) can reuse it without importing the CLI entrypoint. Still
|
||||
// re-exported here for back-compat with `test/ai/build-gateway-config.test.ts`
|
||||
// and other callers that import it from `../../src/cli.ts`. Imported (not just
|
||||
// re-exported) so cli.ts's own connectEngine() call sites bind it locally.
|
||||
import { buildGatewayConfig } from './core/ai/build-gateway-config.ts';
|
||||
export { buildGatewayConfig };
|
||||
|
||||
async function connectEngine(opts?: { probeOnly?: boolean }): Promise<BrainEngine> {
|
||||
const config = loadConfig();
|
||||
|
||||
@@ -18,6 +18,7 @@ import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import { resolveScopedSourceOrThrow, SourceResolutionError } from '../core/sources-ops.ts';
|
||||
import { formatSoleNonDefaultNudge } from '../core/source-resolver.ts';
|
||||
import { resolveCodeReadiness, readinessHint } from '../core/code-graph-readiness.ts';
|
||||
|
||||
/** A bad/invalid `.gbrain-source` pin or GBRAIN_SOURCE value surfaces from
|
||||
* `resolveSourceWithTier`'s `assertSourceExists` as a plain Error with one of
|
||||
@@ -115,9 +116,16 @@ export async function runCodeCallees(engine: BrainEngine, args: string[]): Promi
|
||||
const scope = allSources ? 'all' : 'single';
|
||||
const envelopeSourceId = allSources ? null : (sourceId ?? null);
|
||||
|
||||
// Call-graph readiness ('edge' grain): distinguishes "graph not built / still
|
||||
// indexing" from "genuinely no callees" when count === 0.
|
||||
const readiness = await resolveCodeReadiness(engine, {
|
||||
kind: 'edge', count: edges.length, sourceId: sourceId ?? undefined, allSources,
|
||||
});
|
||||
|
||||
if (shouldEmitJson(args)) {
|
||||
const out: Record<string, unknown> = {
|
||||
symbol: sym, source_id: envelopeSourceId, scope, count: edges.length, callees: edges,
|
||||
symbol: sym, source_id: envelopeSourceId, scope, count: edges.length,
|
||||
status: readiness.status, ready: readiness.ready, callees: edges,
|
||||
};
|
||||
if (edges.length === 0 && !allSources && sourceId) {
|
||||
out.hint = `No callees in source '${sourceId}'. Try --all-sources to search every source.`;
|
||||
@@ -129,6 +137,8 @@ export async function runCodeCallees(engine: BrainEngine, args: string[]): Promi
|
||||
} else {
|
||||
console.log(`No callees found for "${sym}".`);
|
||||
}
|
||||
const hint = readinessHint(readiness);
|
||||
if (hint) console.log(hint);
|
||||
} else {
|
||||
console.log(`${edges.length} callee(s) for "${sym}":`);
|
||||
for (const e of edges) {
|
||||
|
||||
@@ -30,6 +30,7 @@ import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import { resolveScopedSourceOrThrow, SourceResolutionError } from '../core/sources-ops.ts';
|
||||
import { formatSoleNonDefaultNudge } from '../core/source-resolver.ts';
|
||||
import { resolveCodeReadiness, readinessHint } from '../core/code-graph-readiness.ts';
|
||||
|
||||
/** A bad/invalid `.gbrain-source` pin or GBRAIN_SOURCE value surfaces from
|
||||
* `resolveSourceWithTier`'s `assertSourceExists` as a plain Error with one of
|
||||
@@ -134,9 +135,16 @@ export async function runCodeCallers(engine: BrainEngine, args: string[]): Promi
|
||||
const scope = allSources ? 'all' : 'single';
|
||||
const envelopeSourceId = allSources ? null : (sourceId ?? null);
|
||||
|
||||
// Call-graph readiness ('edge' grain): distinguishes "graph not built / still
|
||||
// indexing" from "genuinely no callers" when count === 0.
|
||||
const readiness = await resolveCodeReadiness(engine, {
|
||||
kind: 'edge', count: edges.length, sourceId: sourceId ?? undefined, allSources,
|
||||
});
|
||||
|
||||
if (shouldEmitJson(args)) {
|
||||
const out: Record<string, unknown> = {
|
||||
symbol: sym, source_id: envelopeSourceId, scope, count: edges.length, callers: edges,
|
||||
symbol: sym, source_id: envelopeSourceId, scope, count: edges.length,
|
||||
status: readiness.status, ready: readiness.ready, callers: edges,
|
||||
};
|
||||
if (edges.length === 0 && !allSources && sourceId) {
|
||||
out.hint = `No callers in source '${sourceId}'. Try --all-sources to search every source.`;
|
||||
@@ -148,6 +156,8 @@ export async function runCodeCallers(engine: BrainEngine, args: string[]): Promi
|
||||
} else {
|
||||
console.log(`No callers found for "${sym}".`);
|
||||
}
|
||||
const hint = readinessHint(readiness);
|
||||
if (hint) console.log(hint);
|
||||
} else {
|
||||
console.log(`${edges.length} caller(s) for "${sym}":`);
|
||||
for (const e of edges) {
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import { resolveCodeReadiness, readinessHint } from '../core/code-graph-readiness.ts';
|
||||
|
||||
export interface CodeDefResult {
|
||||
slug: string;
|
||||
@@ -118,11 +119,21 @@ export async function runCodeDef(engine: BrainEngine, args: string[]): Promise<v
|
||||
const language = parseFlag(args, '--lang');
|
||||
try {
|
||||
const results = await findCodeDef(engine, sym, { limit, language });
|
||||
// code-def is brain-wide (not source-scoped); readiness is 'symbol' grain.
|
||||
const readiness = await resolveCodeReadiness(engine, { kind: 'symbol', count: results.length });
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ symbol: sym, count: results.length, results }, null, 2));
|
||||
console.log(JSON.stringify({
|
||||
symbol: sym,
|
||||
count: results.length,
|
||||
status: readiness.status,
|
||||
ready: readiness.ready,
|
||||
results,
|
||||
}, null, 2));
|
||||
} else {
|
||||
if (results.length === 0) {
|
||||
console.log(`No definitions found for "${sym}"`);
|
||||
const hint = readinessHint(readiness);
|
||||
if (hint) console.log(hint);
|
||||
} else {
|
||||
console.log(`Found ${results.length} definition(s) for "${sym}":`);
|
||||
for (const r of results) {
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import { resolveCodeReadiness, readinessHint } from '../core/code-graph-readiness.ts';
|
||||
|
||||
export interface CodeRefResult {
|
||||
slug: string;
|
||||
@@ -107,11 +108,21 @@ export async function runCodeRefs(engine: BrainEngine, args: string[]): Promise<
|
||||
const language = parseFlag(args, '--lang');
|
||||
try {
|
||||
const results = await findCodeRefs(engine, sym, { limit, language });
|
||||
// code-refs is brain-wide (not source-scoped); readiness is 'symbol' grain.
|
||||
const readiness = await resolveCodeReadiness(engine, { kind: 'symbol', count: results.length });
|
||||
if (shouldEmitJson(args)) {
|
||||
console.log(JSON.stringify({ symbol: sym, count: results.length, results }, null, 2));
|
||||
console.log(JSON.stringify({
|
||||
symbol: sym,
|
||||
count: results.length,
|
||||
status: readiness.status,
|
||||
ready: readiness.ready,
|
||||
results,
|
||||
}, null, 2));
|
||||
} else {
|
||||
if (results.length === 0) {
|
||||
console.log(`No references found for "${sym}"`);
|
||||
const hint = readinessHint(readiness);
|
||||
if (hint) console.log(hint);
|
||||
} else {
|
||||
console.log(`Found ${results.length} reference(s) to "${sym}":`);
|
||||
for (const r of results) {
|
||||
|
||||
+42
-36
@@ -9,6 +9,7 @@ const __dirname = dirname(__filename);
|
||||
import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, type GBrainConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import { discoverOAuth, mintClientCredentialsToken, smokeTestMcp } from '../core/remote-mcp-probe.ts';
|
||||
import { runInitEmbedCheck } from '../core/init-embed-check.ts';
|
||||
|
||||
export async function runInit(args: string[]) {
|
||||
// Help guard: cli.ts only routes --help to printOpHelp() for shared-op
|
||||
@@ -95,6 +96,9 @@ export async function runInit(args: string[]) {
|
||||
const chatModelIdx = args.indexOf('--chat-model');
|
||||
// v0.37 (D9): --no-embedding opts into deferred-setup mode (D9 escape hatch).
|
||||
const noEmbedding = args.includes('--no-embedding');
|
||||
// v0.42 (#1780 Gap 2): --skip-embed-check bypasses the init-time embedding
|
||||
// key validation (also honored via GBRAIN_INIT_SKIP_EMBED_CHECK=1).
|
||||
const skipEmbedCheck = args.includes('--skip-embed-check');
|
||||
const aiOpts = await resolveAIOptions({
|
||||
verbose: embModelIdx !== -1 ? args[embModelIdx + 1] : null,
|
||||
shorthand: modelShortIdx !== -1 ? args[modelShortIdx + 1] : null,
|
||||
@@ -121,7 +125,7 @@ export async function runInit(args: string[]) {
|
||||
}
|
||||
}
|
||||
|
||||
return initPGLite({ jsonOutput, apiKey, customPath, aiOpts, schemaPack });
|
||||
return initPGLite({ jsonOutput, apiKey, customPath, aiOpts, schemaPack, skipEmbedCheck });
|
||||
}
|
||||
|
||||
// Supabase/Postgres mode
|
||||
@@ -140,7 +144,7 @@ export async function runInit(args: string[]) {
|
||||
databaseUrl = await supabaseWizard();
|
||||
}
|
||||
|
||||
return initPostgres({ databaseUrl, jsonOutput, apiKey, aiOpts, schemaPack });
|
||||
return initPostgres({ databaseUrl, jsonOutput, apiKey, aiOpts, schemaPack, skipEmbedCheck });
|
||||
}
|
||||
|
||||
interface ResolveAIOptionsArgs {
|
||||
@@ -780,6 +784,8 @@ async function initPGLite(opts: {
|
||||
/** v0.42 (T17): schema pack to default. Stored as config.schema_pack
|
||||
* so loadActivePack's homeConfig tier resolves it. */
|
||||
schemaPack?: string;
|
||||
/** v0.42 (#1780 Gap 2): skip the init-time embedding-key validation. */
|
||||
skipEmbedCheck?: boolean;
|
||||
}) {
|
||||
const dbPath = opts.customPath || gbrainPath('brain.pglite');
|
||||
console.log(`Setting up local brain with PGLite (no server needed)...`);
|
||||
@@ -832,22 +838,20 @@ async function initPGLite(opts: {
|
||||
if (opts.aiOpts?.expansion_model) console.log(` Expansion: ${opts.aiOpts.expansion_model}`);
|
||||
if (opts.aiOpts?.chat_model) console.log(` Chat: ${opts.aiOpts.chat_model}`);
|
||||
|
||||
// v0.37.11.0 Lane C.3: surface ZE setup gap inline at init time when the
|
||||
// resolved provider is ZeroEntropy and neither env nor file-plane key is
|
||||
// set. Beats "first embed call blows up four minutes later" UX.
|
||||
if (resolvedModel?.startsWith('zeroentropyai:')) {
|
||||
const fileCfg = loadConfigFileOnly();
|
||||
if (!process.env.ZEROENTROPY_API_KEY && !fileCfg?.zeroentropy_api_key) {
|
||||
console.warn('');
|
||||
console.warn(' Heads up: ZEROENTROPY_API_KEY is not set.');
|
||||
console.warn(' Set it before first embed:');
|
||||
console.warn(' export ZEROENTROPY_API_KEY=...');
|
||||
console.warn(' Or add to ~/.gbrain/config.json:');
|
||||
console.warn(' "zeroentropy_api_key": "..."');
|
||||
console.warn(' Or pick a different provider:');
|
||||
console.warn(' gbrain init --pglite --embedding-model openai:text-embedding-3-large --embedding-dimensions 1536');
|
||||
}
|
||||
}
|
||||
// v0.42 (#1780 Gap 2): validate the embedding key at init for ALL providers
|
||||
// (generalizes the prior ZeroEntropy-only warning). Config-only diagnose
|
||||
// catches a missing key; a best-effort live test-embed catches an
|
||||
// invalid/expired key. Loud warning to stderr, init still succeeds.
|
||||
// Skipped by --no-embedding / --skip-embed-check / GBRAIN_INIT_SKIP_EMBED_CHECK=1.
|
||||
const embedCheck = await runInitEmbedCheck({
|
||||
resolvedModel,
|
||||
resolvedDim,
|
||||
expansionModel: opts.aiOpts?.expansion_model,
|
||||
chatModel: opts.aiOpts?.chat_model,
|
||||
apiKey: opts.apiKey ?? undefined,
|
||||
noEmbedding: opts.aiOpts?.noEmbedding,
|
||||
skipFlag: opts.skipEmbedCheck,
|
||||
});
|
||||
|
||||
const engine = await createEngine({ engine: 'pglite' });
|
||||
try {
|
||||
@@ -965,7 +969,7 @@ async function initPGLite(opts: {
|
||||
const stats = await engine.getStats();
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
console.log(JSON.stringify({ status: 'success', engine: 'pglite', path: dbPath, pages: stats.page_count }));
|
||||
console.log(JSON.stringify({ status: 'success', engine: 'pglite', path: dbPath, pages: stats.page_count, embedding_check: embedCheck }));
|
||||
} else {
|
||||
console.log(`\nBrain ready at ${dbPath}`);
|
||||
console.log(`${stats.page_count} pages. Engine: PGLite (local Postgres).`);
|
||||
@@ -1002,6 +1006,8 @@ async function initPostgres(opts: {
|
||||
aiOpts?: ResolvedAIOptions;
|
||||
/** v0.42 (T17): schema pack to default. */
|
||||
schemaPack?: string;
|
||||
/** v0.42 (#1780 Gap 2): skip the init-time embedding-key validation. */
|
||||
skipEmbedCheck?: boolean;
|
||||
}) {
|
||||
const { databaseUrl } = opts;
|
||||
|
||||
@@ -1047,22 +1053,19 @@ async function initPostgres(opts: {
|
||||
if (opts.aiOpts?.expansion_model) console.log(` Expansion: ${opts.aiOpts.expansion_model}`);
|
||||
if (opts.aiOpts?.chat_model) console.log(` Chat: ${opts.aiOpts.chat_model}`);
|
||||
|
||||
// v0.37.11.0 Lane C.3: surface ZE setup gap inline at init time when the
|
||||
// resolved provider is ZeroEntropy and neither env nor file-plane key is
|
||||
// set. Beats "first embed call blows up four minutes later" UX.
|
||||
if (resolvedModel?.startsWith('zeroentropyai:')) {
|
||||
const fileCfg = loadConfigFileOnly();
|
||||
if (!process.env.ZEROENTROPY_API_KEY && !fileCfg?.zeroentropy_api_key) {
|
||||
console.warn('');
|
||||
console.warn(' Heads up: ZEROENTROPY_API_KEY is not set.');
|
||||
console.warn(' Set it before first embed:');
|
||||
console.warn(' export ZEROENTROPY_API_KEY=...');
|
||||
console.warn(' Or add to ~/.gbrain/config.json:');
|
||||
console.warn(' "zeroentropy_api_key": "..."');
|
||||
console.warn(' Or pick a different provider:');
|
||||
console.warn(' gbrain init --pglite --embedding-model openai:text-embedding-3-large --embedding-dimensions 1536');
|
||||
}
|
||||
}
|
||||
// v0.42 (#1780 Gap 2): validate the embedding key at init for ALL providers
|
||||
// (generalizes the prior ZeroEntropy-only warning). Same contract as the
|
||||
// PGLite path: loud warning to stderr, init still succeeds; skipped by
|
||||
// --no-embedding / --skip-embed-check / GBRAIN_INIT_SKIP_EMBED_CHECK=1.
|
||||
const embedCheck = await runInitEmbedCheck({
|
||||
resolvedModel,
|
||||
resolvedDim,
|
||||
expansionModel: opts.aiOpts?.expansion_model,
|
||||
chatModel: opts.aiOpts?.chat_model,
|
||||
apiKey: opts.apiKey ?? undefined,
|
||||
noEmbedding: opts.aiOpts?.noEmbedding,
|
||||
skipFlag: opts.skipEmbedCheck,
|
||||
});
|
||||
|
||||
// Detect Supabase direct connection URLs and warn about IPv6
|
||||
if (databaseUrl.match(/db\.[a-z]+\.supabase\.co/) || databaseUrl.includes('.supabase.co:5432')) {
|
||||
@@ -1207,7 +1210,7 @@ async function initPostgres(opts: {
|
||||
const stats = await engine.getStats();
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
console.log(JSON.stringify({ status: 'success', engine: 'postgres', pages: stats.page_count }));
|
||||
console.log(JSON.stringify({ status: 'success', engine: 'postgres', pages: stats.page_count, embedding_check: embedCheck }));
|
||||
} else {
|
||||
console.log(`\nBrain ready. ${stats.page_count} pages. Engine: Postgres (Supabase).`);
|
||||
if (stats.page_count > 0) {
|
||||
@@ -1483,6 +1486,9 @@ OPTIONS
|
||||
Model for query expansion (default: anthropic:claude-haiku)
|
||||
--chat-model <PROVIDER:MODEL>
|
||||
Default subagent driver (v0.27+)
|
||||
--no-embedding Defer embedding setup (skips the embedding-key check)
|
||||
--skip-embed-check Skip the init-time embedding-key validation (config +
|
||||
live test-embed). Also via GBRAIN_INIT_SKIP_EMBED_CHECK=1
|
||||
|
||||
EXAMPLES
|
||||
gbrain init --pglite # Local-only, no API keys
|
||||
|
||||
+10
-9
@@ -751,7 +751,7 @@ async function runBreakLock(
|
||||
sourceId: string,
|
||||
opts: { force: boolean; json: boolean; maxAgeSeconds?: number },
|
||||
): Promise<number> {
|
||||
const { inspectLock, deleteLockRow, deleteLockRowIfStale } = await import('../core/db-lock.ts');
|
||||
const { inspectLock, deleteLockRow, deleteLockRowIfStale, classifyHolderLiveness } = await import('../core/db-lock.ts');
|
||||
const { hostname } = await import('os');
|
||||
const localHost = hostname();
|
||||
let snap;
|
||||
@@ -863,18 +863,19 @@ async function runBreakLock(
|
||||
safe = true;
|
||||
reason = 'ttl_expired';
|
||||
} else {
|
||||
// PID liveness check on local host. process.kill(pid, 0) throws ESRCH
|
||||
// when the PID is dead. Combined with 60s age guard (per outside-voice F7).
|
||||
let alive = true;
|
||||
try { process.kill(snap.holder_pid, 0); }
|
||||
catch { alive = false; }
|
||||
const oldEnough = snap.age_ms >= 60_000;
|
||||
if (!alive && oldEnough) {
|
||||
// PID liveness on local host, via the shared predicate (v0.42 #1780 Gap 3).
|
||||
// Same gate as tryAcquireDbLock's auto-takeover: same-host + provably-dead
|
||||
// (ESRCH) + age >= 60s. EPERM is treated as ALIVE (the PID exists but isn't
|
||||
// ours) — never break a live lock. host is already == localHost here (the
|
||||
// cross-host branch returned above), so classify never yields 'cross_host'.
|
||||
const liveness = classifyHolderLiveness(snap.holder_pid, snap.holder_host, snap.age_ms);
|
||||
if (liveness === 'dead_eligible') {
|
||||
safe = true;
|
||||
reason = 'pid_dead_age_60s';
|
||||
} else if (!alive && !oldEnough) {
|
||||
} else if (liveness === 'too_young') {
|
||||
reason = 'pid_dead_but_lock_too_young';
|
||||
} else {
|
||||
// 'alive' | 'unknown' | 'cross_host' (the latter unreachable here).
|
||||
reason = 'pid_alive';
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* buildGatewayConfig — translate a stored GBrainConfig into the gateway's
|
||||
* AIGatewayConfig (env dict + base_urls + model strings).
|
||||
*
|
||||
* v0.42 (#1780 Gap 2): extracted from src/cli.ts into a core module so
|
||||
* `src/core/init-embed-check.ts` can reuse it without importing the CLI
|
||||
* entrypoint (which would create a load-time cycle). cli.ts re-exports
|
||||
* `buildGatewayConfig` for back-compat with existing callers + tests that
|
||||
* import it from `../../src/cli.ts`.
|
||||
*
|
||||
* The single ownership site for: (a) folding file-plane API keys
|
||||
* (openai/anthropic/zeroentropy) into the gateway env, and (b) threading
|
||||
* local-server `*_BASE_URL` env vars into base_urls. Both matter for the
|
||||
* init-time embedding-key probe — without (a) it would false-warn on
|
||||
* config.json-keyed users, and without (b) a live probe could hit the wrong
|
||||
* endpoint (custom OpenAI base URL, llama-server, etc.).
|
||||
*/
|
||||
|
||||
import type { GBrainConfig } from '../config.ts';
|
||||
import type { AIGatewayConfig } from './types.ts';
|
||||
|
||||
export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
|
||||
// v0.32 (#121 reworked): when ~/.gbrain/config.json declares
|
||||
// openai_api_key / anthropic_api_key, fold them into the gateway env so
|
||||
// recipes that read OPENAI_API_KEY / ANTHROPIC_API_KEY find them. Process
|
||||
// env still wins (it's loaded last) — this is a fallback for daemons /
|
||||
// launchd-spawned subprocesses that don't propagate ~/.zshrc-sourced keys.
|
||||
const envFromConfig: Record<string, string> = {};
|
||||
if (c.openai_api_key) envFromConfig.OPENAI_API_KEY = c.openai_api_key;
|
||||
if (c.anthropic_api_key) envFromConfig.ANTHROPIC_API_KEY = c.anthropic_api_key;
|
||||
// v0.37 fix wave (CDX2-5+6): ZE became the default provider in v0.36 but
|
||||
// the env-mapping at this seam never picked it up. `gbrain config set
|
||||
// zeroentropy_api_key X` wrote DB plane (ignored by gateway). The file-
|
||||
// plane field now exists (GBrainConfig type) and gets mapped here, so
|
||||
// setting it via `~/.gbrain/config.json` propagates into the gateway.
|
||||
if (c.zeroentropy_api_key) envFromConfig.ZEROENTROPY_API_KEY = c.zeroentropy_api_key;
|
||||
|
||||
// v0.32 codex finding #4+#5 fix: thread local-server _BASE_URL env vars
|
||||
// into base_urls so the gateway hits the user's configured port. Without
|
||||
// this, `LLAMA_SERVER_BASE_URL=http://localhost:9000` would let the probe
|
||||
// succeed against :9000 but the actual embed call would still go to the
|
||||
// recipe's base_url_default (localhost:8080). Same fix applies to
|
||||
// OLLAMA_BASE_URL. Caller-provided cfg.provider_base_urls wins.
|
||||
const envBaseUrls: Record<string, string> = {};
|
||||
if (process.env.LLAMA_SERVER_BASE_URL) envBaseUrls['llama-server'] = process.env.LLAMA_SERVER_BASE_URL;
|
||||
// v0.40.6.1: sibling recipe for llama-server in reranking mode. Separate
|
||||
// env var because --reranking and --embeddings are mutually exclusive at
|
||||
// server launch — users running both will have two llama-server processes
|
||||
// on different ports.
|
||||
if (process.env.LLAMA_SERVER_RERANKER_BASE_URL) envBaseUrls['llama-server-reranker'] = process.env.LLAMA_SERVER_RERANKER_BASE_URL;
|
||||
if (process.env.OLLAMA_BASE_URL) envBaseUrls['ollama'] = process.env.OLLAMA_BASE_URL;
|
||||
if (process.env.LMSTUDIO_BASE_URL) envBaseUrls['lmstudio'] = process.env.LMSTUDIO_BASE_URL;
|
||||
if (process.env.LITELLM_BASE_URL) envBaseUrls['litellm'] = process.env.LITELLM_BASE_URL;
|
||||
if (process.env.OPENROUTER_BASE_URL) envBaseUrls['openrouter'] = process.env.OPENROUTER_BASE_URL;
|
||||
|
||||
return {
|
||||
embedding_model: c.embedding_model,
|
||||
embedding_dimensions: c.embedding_dimensions,
|
||||
embedding_multimodal_model: c.embedding_multimodal_model,
|
||||
expansion_model: c.expansion_model,
|
||||
chat_model: c.chat_model,
|
||||
chat_fallback_chain: c.chat_fallback_chain,
|
||||
base_urls: { ...envBaseUrls, ...(c.provider_base_urls ?? {}) }, // config wins over env
|
||||
env: { ...envFromConfig, ...process.env }, // process.env wins
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
/**
|
||||
* Code-graph readiness signal (issue #1780 Gap 1).
|
||||
*
|
||||
* `code-def` / `code-refs` / `code-callers` / `code-callees` historically
|
||||
* returned `count: 0` in three indistinguishable situations:
|
||||
* 1. the symbol graph isn't built yet for the scope (code never synced /
|
||||
* chunked, or edges not yet resolved),
|
||||
* 2. the source was never synced,
|
||||
* 3. the graph IS built and the symbol genuinely has no match.
|
||||
*
|
||||
* An agent that gets `count: 0` can't tell "wait and retry" from "trust this
|
||||
* empty result." This module adds a typed readiness signal so the envelope
|
||||
* carries `status` + `ready`, letting the caller distinguish those cases.
|
||||
*
|
||||
* Two grains, because the four commands read different data:
|
||||
* - `code-def` / `code-refs` read `content_chunks.symbol_name` /
|
||||
* `chunk_text`, which are populated at CHUNK time (during sync/import),
|
||||
* independent of edge resolution. Their readiness is 2-state: code chunks
|
||||
* exist → `ready`, else `not_built`. They never report `indexing` (edge
|
||||
* resolution is irrelevant to them).
|
||||
* - `code-callers` / `code-callees` read the call graph (`code_edges_*`).
|
||||
* Their readiness is 3-state: no code chunks → `not_built`; code chunks
|
||||
* but edges not yet resolved → `indexing`; all resolved → `ready`.
|
||||
*
|
||||
* The "pending edges" predicate MUST mirror the resolver
|
||||
* (`symbol-resolver.ts:resolveSymbolEdgesIncremental`): a chunk is pending
|
||||
* when `edges_backfilled_at IS NULL OR edges_backfilled_at <
|
||||
* EDGE_EXTRACTOR_VERSION_TS`. Counting only `IS NULL` would falsely report
|
||||
* `ready` after a resolver-version bump (the graph is stale, not done).
|
||||
*
|
||||
* Cost: callers run this ONLY when `count === 0` (see `resolveCodeReadiness`);
|
||||
* a non-empty result short-circuits to `ready: true` with no query. Probes use
|
||||
* `EXISTS` (short-circuits on first row) rather than `COUNT(*)` because the
|
||||
* bootstrap schema has no `page_kind` index; the pending probe rides the
|
||||
* partial `idx_content_chunks_edges_backfill` index. Fail-open: any DB error
|
||||
* yields `status: 'unknown'` so a supplementary signal never breaks the command.
|
||||
*
|
||||
* Scope must match the result query exactly: `code-def` / `code-refs` do NOT
|
||||
* filter `deleted_at`, so neither do these probes (else readiness could say
|
||||
* `not_built` while results came from soft-deleted code pages).
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { EDGE_EXTRACTOR_VERSION_TS } from './chunkers/symbol-resolver.ts';
|
||||
|
||||
export type CodeGraphStatus = 'not_built' | 'indexing' | 'ready' | 'unknown';
|
||||
|
||||
export interface CodeGraphReadiness {
|
||||
/** Coarse machine-readable state. */
|
||||
status: CodeGraphStatus;
|
||||
/** Convenience: `status === 'ready'`. */
|
||||
ready: boolean;
|
||||
/** Whether any code chunk exists in scope. */
|
||||
has_code: boolean;
|
||||
/** Whether unresolved/stale edge chunks remain in scope (edge kind only). */
|
||||
pending_edges: boolean;
|
||||
}
|
||||
|
||||
/** Scope for a readiness probe. Omit `sourceId` (or set `allSources`) for brain-wide. */
|
||||
export interface ReadinessScope {
|
||||
sourceId?: string;
|
||||
allSources?: boolean;
|
||||
}
|
||||
|
||||
function effectiveSourceId(scope: ReadinessScope): string | undefined {
|
||||
return scope.allSources ? undefined : scope.sourceId;
|
||||
}
|
||||
|
||||
/** EXISTS probe: does any code chunk exist in scope? Matches the def/refs result query. */
|
||||
async function codeChunksExist(engine: BrainEngine, sourceId: string | undefined): Promise<boolean> {
|
||||
const params: unknown[] = [];
|
||||
let scopeClause = '';
|
||||
if (sourceId) {
|
||||
params.push(sourceId);
|
||||
scopeClause = `AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const rows = await engine.executeRaw<{ e: boolean }>(
|
||||
`SELECT EXISTS(
|
||||
SELECT 1 FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.page_kind = 'code' ${scopeClause}
|
||||
) AS e`,
|
||||
params,
|
||||
);
|
||||
return Boolean(rows[0]?.e);
|
||||
}
|
||||
|
||||
/** EXISTS probe: does any code chunk have unresolved/stale edges (resolver predicate)? */
|
||||
async function pendingEdgeChunksExist(engine: BrainEngine, sourceId: string | undefined): Promise<boolean> {
|
||||
const params: unknown[] = [EDGE_EXTRACTOR_VERSION_TS];
|
||||
let scopeClause = '';
|
||||
if (sourceId) {
|
||||
params.push(sourceId);
|
||||
scopeClause = `AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const rows = await engine.executeRaw<{ e: boolean }>(
|
||||
`SELECT EXISTS(
|
||||
SELECT 1 FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.page_kind = 'code'
|
||||
AND (cc.edges_backfilled_at IS NULL
|
||||
OR cc.edges_backfilled_at < $1::timestamptz)
|
||||
${scopeClause}
|
||||
) AS e`,
|
||||
params,
|
||||
);
|
||||
return Boolean(rows[0]?.e);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the readiness signal for a code-* command.
|
||||
*
|
||||
* `kind: 'symbol'` for code-def/code-refs (2-state); `kind: 'edge'` for
|
||||
* code-callers/code-callees (3-state). When `count > 0` the result is
|
||||
* trivially `ready` and no query runs. Fail-open: any DB error → `unknown`.
|
||||
*/
|
||||
export async function resolveCodeReadiness(
|
||||
engine: BrainEngine,
|
||||
opts: { kind: 'symbol' | 'edge'; count: number } & ReadinessScope,
|
||||
): Promise<CodeGraphReadiness> {
|
||||
if (opts.count > 0) {
|
||||
return { status: 'ready', ready: true, has_code: true, pending_edges: false };
|
||||
}
|
||||
const sourceId = effectiveSourceId(opts);
|
||||
try {
|
||||
const hasCode = await codeChunksExist(engine, sourceId);
|
||||
if (!hasCode) {
|
||||
return { status: 'not_built', ready: false, has_code: false, pending_edges: false };
|
||||
}
|
||||
if (opts.kind === 'symbol') {
|
||||
// Symbol metadata is set at chunk time; code chunks exist ⇒ genuinely none.
|
||||
return { status: 'ready', ready: true, has_code: true, pending_edges: false };
|
||||
}
|
||||
const pending = await pendingEdgeChunksExist(engine, sourceId);
|
||||
return pending
|
||||
? { status: 'indexing', ready: false, has_code: true, pending_edges: true }
|
||||
: { status: 'ready', ready: true, has_code: true, pending_edges: false };
|
||||
} catch {
|
||||
// Supplementary signal: never fail the command on a readiness DB error.
|
||||
return { status: 'unknown', ready: false, has_code: false, pending_edges: false };
|
||||
}
|
||||
}
|
||||
|
||||
/** Human-facing one-liner for non-TTY-less output, or null when ready. */
|
||||
export function readinessHint(r: CodeGraphReadiness): string | null {
|
||||
switch (r.status) {
|
||||
case 'not_built':
|
||||
return 'Symbol graph not built (no code indexed in scope). Run `gbrain sync` to index code.';
|
||||
case 'indexing':
|
||||
return 'Symbol graph still building (edges pending resolution). Re-run after the next `gbrain dream` cycle / autopilot tick.';
|
||||
case 'unknown':
|
||||
return 'Readiness check unavailable (DB error). Treat the empty result as best-effort.';
|
||||
case 'ready':
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -33,6 +33,81 @@ export interface DbLockHandle {
|
||||
/** Default TTL: 30 minutes, same as cycle lock. */
|
||||
const DEFAULT_TTL_MINUTES = 30;
|
||||
|
||||
/**
|
||||
* v0.42 (#1780 Gap 3): grace window before a same-host dead-pid lock is
|
||||
* eligible for automatic takeover. Matches `runBreakLock`'s `age >= 60_000`
|
||||
* gate so the two paths agree. Defends against PID reuse: the OS can recycle
|
||||
* a crashed holder's PID, so we refuse takeover until the lock is older than
|
||||
* this window.
|
||||
*/
|
||||
export const HOLDER_TAKEOVER_GRACE_MS = 60_000;
|
||||
|
||||
/**
|
||||
* Liveness classification of a lock holder, from the perspective of the
|
||||
* current host. Shared by `isHolderDeadLocally` (auto-takeover in
|
||||
* `tryAcquireDbLock`) and `gbrain sync --break-lock`'s safe path so the two
|
||||
* never drift.
|
||||
*
|
||||
* - `cross_host` — holder is on a different host; `process.kill` is
|
||||
* meaningless remotely, never take over.
|
||||
* - `alive` — the PID exists (probe succeeded) OR the probe got
|
||||
* EPERM (the PID exists but isn't ours). EPERM-as-ALIVE
|
||||
* is load-bearing: stealing a live lock is the worst case.
|
||||
* - `too_young` — PID is provably dead (ESRCH) but the lock is younger
|
||||
* than the grace window (possible PID reuse).
|
||||
* - `dead_eligible` — PID is provably dead AND the lock is old enough.
|
||||
* - `unknown` — the probe threw something other than ESRCH/EPERM;
|
||||
* conservative, treat as NOT eligible.
|
||||
*/
|
||||
export type HolderLiveness = 'cross_host' | 'alive' | 'too_young' | 'dead_eligible' | 'unknown';
|
||||
|
||||
export interface HolderLivenessOpts {
|
||||
/** Grace window in ms (default HOLDER_TAKEOVER_GRACE_MS). */
|
||||
graceMs?: number;
|
||||
/** Override the local hostname (test seam; default `os.hostname()`). */
|
||||
localHost?: string;
|
||||
/** Override the liveness probe (test seam; default `process.kill`). */
|
||||
processKill?: (pid: number, signal: number) => void;
|
||||
}
|
||||
|
||||
export function classifyHolderLiveness(
|
||||
holderPid: number,
|
||||
holderHost: string,
|
||||
ageMs: number,
|
||||
opts: HolderLivenessOpts = {},
|
||||
): HolderLiveness {
|
||||
const localHost = opts.localHost ?? hostname();
|
||||
if (holderHost !== localHost) return 'cross_host';
|
||||
|
||||
const probe = opts.processKill ?? ((p: number, s: number) => process.kill(p, s));
|
||||
let probeResult: 'alive' | 'dead' | 'eperm' | 'unknown';
|
||||
try {
|
||||
probe(holderPid, 0);
|
||||
probeResult = 'alive';
|
||||
} catch (e) {
|
||||
const code = (e as NodeJS.ErrnoException)?.code;
|
||||
probeResult = code === 'ESRCH' ? 'dead' : code === 'EPERM' ? 'eperm' : 'unknown';
|
||||
}
|
||||
|
||||
// EPERM → the PID exists but isn't ours: treat as ALIVE, never steal.
|
||||
if (probeResult === 'alive' || probeResult === 'eperm') return 'alive';
|
||||
if (probeResult === 'unknown') return 'unknown';
|
||||
|
||||
// Provably dead (ESRCH). Gate on the grace window to defend against PID reuse.
|
||||
const grace = opts.graceMs ?? HOLDER_TAKEOVER_GRACE_MS;
|
||||
return ageMs < grace ? 'too_young' : 'dead_eligible';
|
||||
}
|
||||
|
||||
/** Convenience boolean: is the holder provably dead, same-host, and past the grace window? */
|
||||
export function isHolderDeadLocally(
|
||||
holderPid: number,
|
||||
holderHost: string,
|
||||
ageMs: number,
|
||||
opts: HolderLivenessOpts = {},
|
||||
): boolean {
|
||||
return classifyHolderLiveness(holderPid, holderHost, ageMs, opts) === 'dead_eligible';
|
||||
}
|
||||
|
||||
/**
|
||||
* Try to acquire a named DB lock.
|
||||
*
|
||||
@@ -71,6 +146,7 @@ export async function tryAcquireDbLock(
|
||||
// registration for free (single ownership site per outside-voice F11).
|
||||
const { registerCleanup } = await import('./process-cleanup.ts');
|
||||
|
||||
const acquireOnce = async (): Promise<DbLockHandle | null> => {
|
||||
if (engine.kind === 'postgres' && maybePG.sql) {
|
||||
const sql = maybePG.sql as any;
|
||||
const ttl = `${ttlMinutes} minutes`;
|
||||
@@ -167,6 +243,32 @@ export async function tryAcquireDbLock(
|
||||
}
|
||||
|
||||
throw new Error(`Unknown engine kind for db-lock: ${engine.kind}`);
|
||||
};
|
||||
|
||||
const first = await acquireOnce();
|
||||
if (first) return first;
|
||||
|
||||
// v0.42 (#1780 Gap 3): the lock is held and its TTL hasn't expired (the
|
||||
// upsert's ON CONFLICT ... WHERE ttl_expires_at < NOW() returned no row).
|
||||
// If the holder is on THIS host, provably dead, and past the grace window,
|
||||
// reclaim it: guarded DELETE then retry the normal upsert ONCE. The retry
|
||||
// returns the normal DbLockHandle (refresh/release intact) — no hand-rolled
|
||||
// handle. TTL-expired holders are NOT handled here (the upsert already takes
|
||||
// them); cross-host holders stay TTL-only. Best-effort: any error falls
|
||||
// through to `return null` (busy), exactly as the pre-takeover behavior.
|
||||
try {
|
||||
const snap = await inspectLock(engine, lockId);
|
||||
if (snap && !snap.ttl_expired && isHolderDeadLocally(snap.holder_pid, snap.holder_host, snap.age_ms)) {
|
||||
const { deleted } = await deleteLockRow(engine, lockId, snap.holder_pid);
|
||||
if (deleted) {
|
||||
const second = await acquireOnce();
|
||||
if (second) return second;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Auto-takeover is best-effort; never throw from the acquire path.
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
/**
|
||||
* Embedding-key validation at `gbrain init` (issue #1780 Gap 2).
|
||||
*
|
||||
* Before this, `gbrain init` persisted `--embedding-model` to config.json but
|
||||
* never checked the provider key was present/working. The failure surfaced only
|
||||
* at first sync (`embedBatch` throws, pages import but `embedded=0`), and
|
||||
* combined with Gap 1 the call graph silently never built.
|
||||
*
|
||||
* This runs two checks at init time, both non-fatal (loud warning, init still
|
||||
* exits 0 — `--no-embedding` is the deferred-setup escape hatch):
|
||||
* 1. `diagnoseEmbedding()` — config-only, zero-network. Catches a missing key
|
||||
* for ANY provider.
|
||||
* 2. `liveTestEmbed()` — a best-effort 1-token embed (5s timeout) when a key
|
||||
* IS present. Catches invalid/expired keys. Network/timeout/offline →
|
||||
* warn only, never blocks.
|
||||
*
|
||||
* Both run against the EFFECTIVE gateway config — process.env overlaid with
|
||||
* file-plane keys (openai/anthropic/zeroentropy from config.json) and
|
||||
* `opts.apiKey`, plus provider base URLs — built via the same
|
||||
* `buildGatewayConfig` runtime uses. Without that, the config-only check would
|
||||
* false-warn on config.json-keyed users, and the live probe could hit the
|
||||
* wrong endpoint (custom OpenAI base URL, llama-server, etc.).
|
||||
*
|
||||
* Skips entirely on `--no-embedding`, `--skip-embed-check`, or
|
||||
* `GBRAIN_INIT_SKIP_EMBED_CHECK=1`. Warnings go to stderr; the caller folds the
|
||||
* returned `InitEmbedCheckResult` into init's `--json` envelope as
|
||||
* `embedding_check`.
|
||||
*/
|
||||
|
||||
import type { GBrainConfig } from './config.ts';
|
||||
import { loadConfigFileOnly } from './config.ts';
|
||||
import { buildGatewayConfig } from './ai/build-gateway-config.ts';
|
||||
import type { EmbeddingDiagnosis } from './ai/gateway.ts';
|
||||
|
||||
export interface InitEmbedCheckResult {
|
||||
/** config-level ok: provider key present + recipe valid. */
|
||||
ok: boolean;
|
||||
/** when the whole check was skipped, why. */
|
||||
skipped?: 'no_embedding' | 'flag' | 'env' | 'no_model';
|
||||
/** diagnosis reason when `ok === false`. */
|
||||
reason?: string;
|
||||
/** live test-embed result, undefined when not run. */
|
||||
live_ok?: boolean;
|
||||
/** live test-embed failure reason. */
|
||||
live_reason?: string;
|
||||
}
|
||||
|
||||
export interface RunInitEmbedCheckOpts {
|
||||
resolvedModel?: string;
|
||||
resolvedDim?: number;
|
||||
expansionModel?: string;
|
||||
chatModel?: string;
|
||||
/** opts.apiKey from init (maps to openai_api_key). */
|
||||
apiKey?: string;
|
||||
noEmbedding?: boolean;
|
||||
/** --skip-embed-check flag. */
|
||||
skipFlag?: boolean;
|
||||
// ── test seams ──
|
||||
loadFileConfig?: () => GBrainConfig | null;
|
||||
/** default: console.error (stderr). */
|
||||
warn?: (msg: string) => void;
|
||||
/** skip the network probe (config-only); tests for the diagnose path. */
|
||||
skipLiveProbe?: boolean;
|
||||
liveTimeoutMs?: number;
|
||||
}
|
||||
|
||||
/** Classify a live-probe error into a coarse, stable reason. */
|
||||
function classifyLiveReason(err: unknown): string {
|
||||
const msg = (err instanceof Error ? err.message : String(err)).toLowerCase();
|
||||
if (/timed out|timeout|abort/.test(msg)) return 'timeout';
|
||||
if (/auth|unauthor|401|403|api[_-]?key|credential/.test(msg)) return 'auth';
|
||||
if (/rate.?limit|429|too many/.test(msg)) return 'rate_limit';
|
||||
if (/network|econn|fetch failed|enotfound|dns/.test(msg)) return 'network';
|
||||
return 'unknown';
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort live test-embed against the currently-configured gateway.
|
||||
* 1 token, 5s timeout. Never throws — returns a tagged result.
|
||||
*
|
||||
* (Purpose-built rather than reusing `models.ts:probeEmbeddingReachability`,
|
||||
* which is private and returns the doctor-shaped `ProbeResult`. v0.42+ TODO:
|
||||
* unify the two onto one shared embed-probe core.)
|
||||
*/
|
||||
export async function liveTestEmbed(
|
||||
opts?: { timeoutMs?: number },
|
||||
): Promise<{ ok: true } | { ok: false; reason: string; message: string }> {
|
||||
const { embed } = await import('./ai/gateway.ts');
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(new Error('embed probe timed out')), opts?.timeoutMs ?? 5000);
|
||||
try {
|
||||
await embed(['probe'], { inputType: 'query', abortSignal: controller.signal });
|
||||
return { ok: true };
|
||||
} catch (err) {
|
||||
return { ok: false, reason: classifyLiveReason(err), message: err instanceof Error ? err.message : String(err) };
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
/** Init-specific warning for a non-ok diagnosis. Names `--no-embedding` +
|
||||
* `--skip-embed-check` (NOT `--no-embed`, which is the sync/embed flag). */
|
||||
function formatInitEmbedWarning(d: Exclude<EmbeddingDiagnosis, { ok: true }>): string {
|
||||
const lines: string[] = ['', ' Heads up: embedding is configured but not ready.'];
|
||||
switch (d.reason) {
|
||||
case 'missing_env':
|
||||
lines.push(` Model "${d.model}" needs ${d.missingEnvVars.join(', ')} — not set in your shell or ~/.gbrain/config.json.`);
|
||||
lines.push(' Set it before first sync:');
|
||||
lines.push(` export ${d.missingEnvVars[0]}=...`);
|
||||
break;
|
||||
case 'unknown_provider':
|
||||
lines.push(` Model "${d.model}" uses unknown provider "${d.provider}".`);
|
||||
lines.push(` ${d.message}`);
|
||||
break;
|
||||
case 'no_touchpoint':
|
||||
lines.push(` Provider "${d.provider}" has no embedding touchpoint.`);
|
||||
break;
|
||||
case 'user_provided_model_unset':
|
||||
lines.push(` Provider "${d.provider}" needs an explicit model id (provider:model).`);
|
||||
break;
|
||||
case 'no_model_configured':
|
||||
lines.push(' No embedding model is configured.');
|
||||
break;
|
||||
case 'no_gateway_config':
|
||||
lines.push(' Embedding gateway is not configured (startup-order bug — please file an issue).');
|
||||
break;
|
||||
}
|
||||
lines.push(' Without it, `gbrain sync` imports pages but embeds 0 (search + code graph stay empty).');
|
||||
lines.push(' Fixes:');
|
||||
lines.push(' • Set the key above, then run `gbrain sync`.');
|
||||
lines.push(' • Or defer embedding entirely: re-run init with --no-embedding.');
|
||||
lines.push(' • Or skip this check: --skip-embed-check (or GBRAIN_INIT_SKIP_EMBED_CHECK=1).');
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
function formatLiveProbeWarning(p: { reason: string; message: string }, model: string): string {
|
||||
return [
|
||||
'',
|
||||
` Heads up: an embedding key is set but a test embed failed (${p.reason}).`,
|
||||
` Model: ${model}`,
|
||||
` Error: ${p.message}`,
|
||||
' `gbrain sync` may fail to embed. Verify the key/endpoint, or re-run init',
|
||||
' with --skip-embed-check to bypass this probe.',
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the init-time embedding validation. Configures the gateway with the
|
||||
* effective env, diagnoses config, then (if config ok and a key is present)
|
||||
* runs a best-effort live probe. Warns to stderr; never throws; init proceeds
|
||||
* regardless. Returns the result for the `--json` envelope.
|
||||
*/
|
||||
export async function runInitEmbedCheck(opts: RunInitEmbedCheckOpts): Promise<InitEmbedCheckResult> {
|
||||
const warn = opts.warn ?? ((m: string) => console.error(m));
|
||||
|
||||
if (opts.noEmbedding) return { ok: true, skipped: 'no_embedding' };
|
||||
if (opts.skipFlag) return { ok: true, skipped: 'flag' };
|
||||
if (process.env.GBRAIN_INIT_SKIP_EMBED_CHECK === '1') return { ok: true, skipped: 'env' };
|
||||
// No model resolved means resolveAIOptions already fail-loud'd (or deferred);
|
||||
// nothing to validate here.
|
||||
if (!opts.resolvedModel) return { ok: true, skipped: 'no_model' };
|
||||
|
||||
// Build the effective gateway config the SAME way runtime does so the check
|
||||
// sees the same keys AND provider base URLs (D1A + D7A).
|
||||
const loadFile = opts.loadFileConfig ?? loadConfigFileOnly;
|
||||
const fileCfg = loadFile() ?? ({} as GBrainConfig);
|
||||
const effective: GBrainConfig = {
|
||||
...fileCfg,
|
||||
embedding_model: opts.resolvedModel,
|
||||
embedding_dimensions: opts.resolvedDim,
|
||||
expansion_model: opts.expansionModel ?? fileCfg.expansion_model,
|
||||
chat_model: opts.chatModel ?? fileCfg.chat_model,
|
||||
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
|
||||
};
|
||||
|
||||
const { configureGateway, diagnoseEmbedding } = await import('./ai/gateway.ts');
|
||||
configureGateway(buildGatewayConfig(effective));
|
||||
|
||||
const diag = diagnoseEmbedding();
|
||||
if (!diag.ok) {
|
||||
warn(formatInitEmbedWarning(diag));
|
||||
return { ok: false, reason: diag.reason };
|
||||
}
|
||||
|
||||
if (opts.skipLiveProbe) return { ok: true };
|
||||
|
||||
const probe = await liveTestEmbed({ timeoutMs: opts.liveTimeoutMs });
|
||||
if (!probe.ok) {
|
||||
warn(formatLiveProbeWarning(probe, opts.resolvedModel));
|
||||
return { ok: true, live_ok: false, live_reason: probe.reason };
|
||||
}
|
||||
return { ok: true, live_ok: true };
|
||||
}
|
||||
+18
-4
@@ -3751,7 +3751,11 @@ const code_callers: Operation = {
|
||||
allSources,
|
||||
sourceId,
|
||||
});
|
||||
return { symbol, count: edges.length, callers: edges };
|
||||
const { resolveCodeReadiness } = await import('./code-graph-readiness.ts');
|
||||
const readiness = await resolveCodeReadiness(ctx.engine, {
|
||||
kind: 'edge', count: edges.length, sourceId, allSources,
|
||||
});
|
||||
return { symbol, count: edges.length, status: readiness.status, ready: readiness.ready, callers: edges };
|
||||
},
|
||||
cliHints: { name: 'code_callers', hidden: true },
|
||||
};
|
||||
@@ -3782,7 +3786,11 @@ const code_callees: Operation = {
|
||||
allSources,
|
||||
sourceId,
|
||||
});
|
||||
return { symbol, count: edges.length, callees: edges };
|
||||
const { resolveCodeReadiness } = await import('./code-graph-readiness.ts');
|
||||
const readiness = await resolveCodeReadiness(ctx.engine, {
|
||||
kind: 'edge', count: edges.length, sourceId, allSources,
|
||||
});
|
||||
return { symbol, count: edges.length, status: readiness.status, ready: readiness.ready, callees: edges };
|
||||
},
|
||||
cliHints: { name: 'code_callees', hidden: true },
|
||||
};
|
||||
@@ -3802,7 +3810,10 @@ const code_def: Operation = {
|
||||
limit: (p.limit as number) ?? 20,
|
||||
language: (p.lang as string) || undefined,
|
||||
});
|
||||
return { symbol: p.symbol as string, count: defs.length, defs };
|
||||
// code_def is brain-wide (not source-scoped); readiness is 'symbol' grain.
|
||||
const { resolveCodeReadiness } = await import('./code-graph-readiness.ts');
|
||||
const readiness = await resolveCodeReadiness(ctx.engine, { kind: 'symbol', count: defs.length });
|
||||
return { symbol: p.symbol as string, count: defs.length, status: readiness.status, ready: readiness.ready, defs };
|
||||
},
|
||||
cliHints: { name: 'code_def', hidden: true },
|
||||
};
|
||||
@@ -3822,7 +3833,10 @@ const code_refs: Operation = {
|
||||
limit: (p.limit as number) ?? 50,
|
||||
language: (p.lang as string) || undefined,
|
||||
});
|
||||
return { symbol: p.symbol as string, count: refs.length, refs };
|
||||
// code_refs is brain-wide (not source-scoped); readiness is 'symbol' grain.
|
||||
const { resolveCodeReadiness } = await import('./code-graph-readiness.ts');
|
||||
const readiness = await resolveCodeReadiness(ctx.engine, { kind: 'symbol', count: refs.length });
|
||||
return { symbol: p.symbol as string, count: refs.length, status: readiness.status, ready: readiness.ready, refs };
|
||||
},
|
||||
cliHints: { name: 'code_refs', hidden: true },
|
||||
};
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
/**
|
||||
* #1780 Gap 1 — code-graph readiness signal.
|
||||
*
|
||||
* Verifies the typed readiness contract that lets code-* callers tell
|
||||
* "graph not built / still indexing" apart from "genuinely no match" when
|
||||
* count === 0:
|
||||
* - empty brain → not_built (both grains)
|
||||
* - code synced, edges not resolved → symbol grain ready, edge grain indexing
|
||||
* - edges resolved → edge grain ready
|
||||
* - count > 0 → ready short-circuit (no query)
|
||||
* - source scoping (scoped miss → not_built; allSources → brain-wide)
|
||||
* - DB error → unknown, fail-open (CRITICAL regression)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { importCodeFile } from '../src/core/import-file.ts';
|
||||
import { resolveCodeReadiness, readinessHint } from '../src/core/code-graph-readiness.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clean slate per test: remove all chunks + pages so empty-brain cases hold.
|
||||
await engine.executeRaw('DELETE FROM content_chunks');
|
||||
await engine.executeRaw('DELETE FROM pages');
|
||||
});
|
||||
|
||||
const SAMPLE = `export function alpha(x: number): number {
|
||||
return beta(x) + 1;
|
||||
}
|
||||
|
||||
export function beta(y: number): number {
|
||||
return y * 2;
|
||||
}
|
||||
`;
|
||||
|
||||
describe('resolveCodeReadiness — empty brain', () => {
|
||||
test('symbol grain → not_built when no code exists', async () => {
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'symbol', count: 0 });
|
||||
expect(r.status).toBe('not_built');
|
||||
expect(r.ready).toBe(false);
|
||||
expect(r.has_code).toBe(false);
|
||||
});
|
||||
|
||||
test('edge grain → not_built when no code exists', async () => {
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'edge', count: 0 });
|
||||
expect(r.status).toBe('not_built');
|
||||
expect(r.ready).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveCodeReadiness — code synced, edges unresolved', () => {
|
||||
beforeEach(async () => {
|
||||
// importCodeFile writes code chunks with edges_backfilled_at = NULL
|
||||
// (resolve phase hasn't run), exactly the "graph still building" state.
|
||||
await importCodeFile(engine, 'src/sample.ts', SAMPLE, { noEmbed: true });
|
||||
});
|
||||
|
||||
test('symbol grain → ready (symbol metadata is at chunk time)', async () => {
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'symbol', count: 0 });
|
||||
expect(r.status).toBe('ready');
|
||||
expect(r.ready).toBe(true);
|
||||
expect(r.has_code).toBe(true);
|
||||
});
|
||||
|
||||
test('edge grain → indexing (edges pending resolution)', async () => {
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'edge', count: 0 });
|
||||
expect(r.status).toBe('indexing');
|
||||
expect(r.ready).toBe(false);
|
||||
expect(r.pending_edges).toBe(true);
|
||||
});
|
||||
|
||||
test('edge grain → ready once edges_backfilled_at is stamped fresh', async () => {
|
||||
// Mirror what the resolve_symbol_edges phase does: stamp every code chunk.
|
||||
await engine.executeRaw('UPDATE content_chunks SET edges_backfilled_at = NOW()');
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'edge', count: 0 });
|
||||
expect(r.status).toBe('ready');
|
||||
expect(r.ready).toBe(true);
|
||||
expect(r.pending_edges).toBe(false);
|
||||
});
|
||||
|
||||
test('count > 0 short-circuits to ready with no probe', async () => {
|
||||
// Even with pending edges, a non-empty result is trivially ready.
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'edge', count: 3 });
|
||||
expect(r.status).toBe('ready');
|
||||
expect(r.ready).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveCodeReadiness — source scoping', () => {
|
||||
beforeEach(async () => {
|
||||
await importCodeFile(engine, 'src/sample.ts', SAMPLE, { noEmbed: true });
|
||||
});
|
||||
|
||||
test('scoped to a source with no code → not_built', async () => {
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'symbol', count: 0, sourceId: 'no-such-source' });
|
||||
expect(r.status).toBe('not_built');
|
||||
});
|
||||
|
||||
test('scoped to the default source (where code lives) → ready (symbol)', async () => {
|
||||
const r = await resolveCodeReadiness(engine, { kind: 'symbol', count: 0, sourceId: 'default' });
|
||||
expect(r.status).toBe('ready');
|
||||
});
|
||||
|
||||
test('allSources ignores a non-matching sourceId and goes brain-wide', async () => {
|
||||
const r = await resolveCodeReadiness(engine, {
|
||||
kind: 'symbol', count: 0, sourceId: 'no-such-source', allSources: true,
|
||||
});
|
||||
expect(r.status).toBe('ready');
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveCodeReadiness — fail-open (CRITICAL regression)', () => {
|
||||
test('DB error → status unknown, ready false, never throws', async () => {
|
||||
const broken = {
|
||||
kind: 'pglite',
|
||||
executeRaw: async () => { throw new Error('boom'); },
|
||||
} as unknown as BrainEngine;
|
||||
const r = await resolveCodeReadiness(broken, { kind: 'edge', count: 0 });
|
||||
expect(r.status).toBe('unknown');
|
||||
expect(r.ready).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('readinessHint', () => {
|
||||
test('not_built / indexing / unknown produce a hint; ready does not', () => {
|
||||
expect(readinessHint({ status: 'not_built', ready: false, has_code: false, pending_edges: false })).toContain('not built');
|
||||
expect(readinessHint({ status: 'indexing', ready: false, has_code: true, pending_edges: true })).toContain('still building');
|
||||
expect(readinessHint({ status: 'unknown', ready: false, has_code: false, pending_edges: false })).toContain('unavailable');
|
||||
expect(readinessHint({ status: 'ready', ready: true, has_code: true, pending_edges: false })).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,130 @@
|
||||
/**
|
||||
* #1780 Gap 3 — automatic same-host dead-pid lock takeover.
|
||||
*
|
||||
* Two layers:
|
||||
* - classifyHolderLiveness (pure, injectable process.kill seam): the
|
||||
* decision matrix incl. the CRITICAL EPERM-as-ALIVE rule.
|
||||
* - tryAcquireDbLock auto-takeover (PGLite, real process.kill): a held +
|
||||
* not-TTL-expired lock whose same-host holder is provably dead and past
|
||||
* the 60s grace gets reclaimed; alive / cross-host / young holders don't.
|
||||
* A taken-over lock returns a working handle (refresh + release).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { hostname } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
tryAcquireDbLock,
|
||||
classifyHolderLiveness,
|
||||
isHolderDeadLocally,
|
||||
HOLDER_TAKEOVER_GRACE_MS,
|
||||
} from '../src/core/db-lock.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ database_url: '' });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id LIKE 'test-takeover-%'`);
|
||||
});
|
||||
|
||||
const LOCAL = hostname();
|
||||
const OLD_MS = HOLDER_TAKEOVER_GRACE_MS + 60_000; // comfortably past the grace window
|
||||
const ESRCH = () => { const e = new Error('no such process') as NodeJS.ErrnoException; e.code = 'ESRCH'; throw e; };
|
||||
const EPERM = () => { const e = new Error('operation not permitted') as NodeJS.ErrnoException; e.code = 'EPERM'; throw e; };
|
||||
const EINVAL = () => { const e = new Error('weird') as NodeJS.ErrnoException; e.code = 'EINVAL'; throw e; };
|
||||
const aliveKill = () => { /* no throw → alive */ };
|
||||
|
||||
describe('classifyHolderLiveness', () => {
|
||||
test('same-host + ESRCH + old → dead_eligible', () => {
|
||||
expect(classifyHolderLiveness(123, LOCAL, OLD_MS, { processKill: ESRCH })).toBe('dead_eligible');
|
||||
});
|
||||
|
||||
test('same-host + ESRCH + young → too_young (PID-reuse guard)', () => {
|
||||
expect(classifyHolderLiveness(123, LOCAL, 5_000, { processKill: ESRCH })).toBe('too_young');
|
||||
});
|
||||
|
||||
test('same-host + alive → alive', () => {
|
||||
expect(classifyHolderLiveness(123, LOCAL, OLD_MS, { processKill: aliveKill })).toBe('alive');
|
||||
});
|
||||
|
||||
test('CRITICAL: same-host + EPERM → alive (never steal a live lock)', () => {
|
||||
expect(classifyHolderLiveness(123, LOCAL, OLD_MS, { processKill: EPERM })).toBe('alive');
|
||||
});
|
||||
|
||||
test('same-host + unknown errno → unknown (conservative)', () => {
|
||||
expect(classifyHolderLiveness(123, LOCAL, OLD_MS, { processKill: EINVAL })).toBe('unknown');
|
||||
});
|
||||
|
||||
test('cross-host → cross_host (never probe a remote pid)', () => {
|
||||
expect(classifyHolderLiveness(123, 'some-other-host', OLD_MS, { processKill: ESRCH })).toBe('cross_host');
|
||||
});
|
||||
|
||||
test('isHolderDeadLocally is true only for dead_eligible', () => {
|
||||
expect(isHolderDeadLocally(1, LOCAL, OLD_MS, { processKill: ESRCH })).toBe(true);
|
||||
expect(isHolderDeadLocally(1, LOCAL, 5_000, { processKill: ESRCH })).toBe(false);
|
||||
expect(isHolderDeadLocally(1, LOCAL, OLD_MS, { processKill: EPERM })).toBe(false);
|
||||
expect(isHolderDeadLocally(1, 'other', OLD_MS, { processKill: ESRCH })).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
/** Insert a held, NOT-TTL-expired lock row for the given holder. */
|
||||
async function seedHeldLock(id: string, holderPid: number, holderHost: string, ageSeconds: number) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
||||
VALUES ($1, $2, $3, NOW() - ($4 || ' seconds')::interval, NOW() + INTERVAL '10 minutes', NOW() - ($4 || ' seconds')::interval)`,
|
||||
[id, holderPid, holderHost, String(ageSeconds)],
|
||||
);
|
||||
}
|
||||
|
||||
/** A reliably-dead PID on this host: spawn a process, wait for it to exit. */
|
||||
async function deadPid(): Promise<number> {
|
||||
const proc = Bun.spawn(['sh', '-c', 'exit 0']);
|
||||
await proc.exited;
|
||||
return proc.pid;
|
||||
}
|
||||
|
||||
describe('tryAcquireDbLock auto-takeover', () => {
|
||||
test('reclaims a same-host dead-pid lock past the grace window', async () => {
|
||||
const pid = await deadPid();
|
||||
await seedHeldLock('test-takeover-dead', pid, LOCAL, 120);
|
||||
const handle = await tryAcquireDbLock(engine, 'test-takeover-dead', 30);
|
||||
expect(handle).not.toBeNull();
|
||||
// The reclaimed handle is the normal one: refresh + release work.
|
||||
await handle!.refresh();
|
||||
await handle!.release();
|
||||
// After release, the row is gone → a fresh acquire succeeds immediately.
|
||||
const again = await tryAcquireDbLock(engine, 'test-takeover-dead', 30);
|
||||
expect(again).not.toBeNull();
|
||||
await again!.release();
|
||||
});
|
||||
|
||||
test('does NOT reclaim a live same-host holder', async () => {
|
||||
// process.pid is alive → no takeover; lock stays held.
|
||||
await seedHeldLock('test-takeover-alive', process.pid, LOCAL, 120);
|
||||
const handle = await tryAcquireDbLock(engine, 'test-takeover-alive', 30);
|
||||
expect(handle).toBeNull();
|
||||
});
|
||||
|
||||
test('does NOT reclaim a cross-host holder (TTL-only)', async () => {
|
||||
const pid = await deadPid();
|
||||
await seedHeldLock('test-takeover-xhost', pid, 'a-different-host', 120);
|
||||
const handle = await tryAcquireDbLock(engine, 'test-takeover-xhost', 30);
|
||||
expect(handle).toBeNull();
|
||||
});
|
||||
|
||||
test('does NOT reclaim a dead-pid lock younger than the grace window', async () => {
|
||||
const pid = await deadPid();
|
||||
await seedHeldLock('test-takeover-young', pid, LOCAL, 5); // 5s < 60s grace
|
||||
const handle = await tryAcquireDbLock(engine, 'test-takeover-young', 30);
|
||||
expect(handle).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -169,6 +169,61 @@ describe('v0.34 W3 — code_def finds definition sites', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1780 Gap 1 — readiness envelope on code_* ops', () => {
|
||||
test('code_callers carries status:ready when callers are found', async () => {
|
||||
await seedTwoFileGraph(engine);
|
||||
const ctx = makeCtx(engine, 'source-a');
|
||||
const result = (await operationsByName.code_callers!.handler(ctx, { symbol: 'parseMarkdown' })) as {
|
||||
count: number; status: string; ready: boolean;
|
||||
};
|
||||
expect(result.count).toBeGreaterThanOrEqual(1);
|
||||
expect(result.status).toBe('ready');
|
||||
expect(result.ready).toBe(true);
|
||||
});
|
||||
|
||||
test('code_callers → indexing when code exists but edges unresolved + no callers', async () => {
|
||||
// callerInA has no callers; seeded chunks have edges_backfilled_at = NULL.
|
||||
await seedTwoFileGraph(engine);
|
||||
const ctx = makeCtx(engine, 'source-a');
|
||||
const result = (await operationsByName.code_callers!.handler(ctx, { symbol: 'callerInA' })) as {
|
||||
count: number; status: string; ready: boolean;
|
||||
};
|
||||
expect(result.count).toBe(0);
|
||||
expect(result.status).toBe('indexing');
|
||||
expect(result.ready).toBe(false);
|
||||
});
|
||||
|
||||
test('code_def → not_built on an empty brain', async () => {
|
||||
const ctx = makeCtx(engine, 'source-a');
|
||||
const result = (await operationsByName.code_def!.handler(ctx, { symbol: 'anything' })) as {
|
||||
count: number; status: string; ready: boolean;
|
||||
};
|
||||
expect(result.count).toBe(0);
|
||||
expect(result.status).toBe('not_built');
|
||||
expect(result.ready).toBe(false);
|
||||
});
|
||||
|
||||
test('code_def → ready when a definition exists (brain-wide)', async () => {
|
||||
await seedDefSite(engine);
|
||||
const ctx = makeCtx(engine, 'source-a');
|
||||
const result = (await operationsByName.code_def!.handler(ctx, { symbol: 'parseMarkdown' })) as {
|
||||
count: number; status: string; ready: boolean;
|
||||
};
|
||||
expect(result.count).toBe(1);
|
||||
expect(result.status).toBe('ready');
|
||||
expect(result.ready).toBe(true);
|
||||
});
|
||||
|
||||
test('code_refs → not_built on an empty brain', async () => {
|
||||
const ctx = makeCtx(engine, 'source-a');
|
||||
const result = (await operationsByName.code_refs!.handler(ctx, { symbol: 'anything' })) as {
|
||||
count: number; status: string; ready: boolean;
|
||||
};
|
||||
expect(result.status).toBe('not_built');
|
||||
expect(result.ready).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Fixtures
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
/**
|
||||
* #1780 Gap 2 — init-time embedding-key validation.
|
||||
*
|
||||
* Hermetic: drives `runInitEmbedCheck` with the gateway embed-transport seam
|
||||
* (`__setEmbedTransportForTests`) and `withEnv` — no real network, no
|
||||
* mock.module. Covers:
|
||||
* - skip paths (--no-embedding / --skip-embed-check / env / no model)
|
||||
* - missing key → loud warning, ok:false (config diagnose)
|
||||
* - CRITICAL regression: file-plane key (config.json, not env) → NO false
|
||||
* "missing key" warning (the effective-env merge, D1A)
|
||||
* - live probe failure is best-effort (warns, ok stays true, never throws)
|
||||
* - live probe success → live_ok:true
|
||||
* - init-specific message names --no-embedding / --skip-embed-check (not --no-embed)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import type { GBrainConfig } from '../src/core/config.ts';
|
||||
import { runInitEmbedCheck } from '../src/core/init-embed-check.ts';
|
||||
import { __setEmbedTransportForTests, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
|
||||
const OPENAI = 'openai:text-embedding-3-large';
|
||||
|
||||
beforeEach(() => { resetGateway(); __setEmbedTransportForTests(null); });
|
||||
afterEach(() => { resetGateway(); __setEmbedTransportForTests(null); });
|
||||
|
||||
function capture() {
|
||||
const warned: string[] = [];
|
||||
return { warn: (m: string) => warned.push(m), warned };
|
||||
}
|
||||
|
||||
describe('runInitEmbedCheck — skip paths', () => {
|
||||
test('--no-embedding skips entirely', async () => {
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({ resolvedModel: OPENAI, noEmbedding: true, warn });
|
||||
expect(r.skipped).toBe('no_embedding');
|
||||
expect(warned).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('--skip-embed-check skips entirely', async () => {
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({ resolvedModel: OPENAI, skipFlag: true, warn });
|
||||
expect(r.skipped).toBe('flag');
|
||||
expect(warned).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('GBRAIN_INIT_SKIP_EMBED_CHECK=1 skips entirely', async () => {
|
||||
await withEnv({ GBRAIN_INIT_SKIP_EMBED_CHECK: '1' }, async () => {
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({ resolvedModel: OPENAI, warn });
|
||||
expect(r.skipped).toBe('env');
|
||||
expect(warned).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('no resolved model → skipped no_model', async () => {
|
||||
const r = await runInitEmbedCheck({});
|
||||
expect(r.skipped).toBe('no_model');
|
||||
});
|
||||
});
|
||||
|
||||
describe('runInitEmbedCheck — config diagnose', () => {
|
||||
test('missing key → ok:false + loud warning naming the right flags', async () => {
|
||||
await withEnv({ OPENAI_API_KEY: undefined }, async () => {
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({
|
||||
resolvedModel: OPENAI,
|
||||
resolvedDim: 1536,
|
||||
apiKey: undefined,
|
||||
loadFileConfig: () => ({} as GBrainConfig),
|
||||
warn,
|
||||
});
|
||||
expect(r.ok).toBe(false);
|
||||
expect(r.reason).toBe('missing_env');
|
||||
expect(warned).toHaveLength(1);
|
||||
const msg = warned[0];
|
||||
expect(msg).toContain('OPENAI_API_KEY');
|
||||
// init flag, NOT the sync/embed flag --no-embed
|
||||
expect(msg).toContain('--no-embedding');
|
||||
expect(msg).toContain('--skip-embed-check');
|
||||
expect(msg).not.toMatch(/--no-embed(?!ding)/);
|
||||
});
|
||||
});
|
||||
|
||||
test('CRITICAL: file-plane key (config.json, not env) → no false missing-key warning', async () => {
|
||||
await withEnv({ OPENAI_API_KEY: undefined }, async () => {
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({
|
||||
resolvedModel: OPENAI,
|
||||
resolvedDim: 1536,
|
||||
// key lives in config.json (file plane), not the shell env
|
||||
loadFileConfig: () => ({ openai_api_key: 'sk-from-config-file' } as GBrainConfig),
|
||||
skipLiveProbe: true, // config-only: this is the diagnose regression
|
||||
warn,
|
||||
});
|
||||
expect(r.ok).toBe(true);
|
||||
expect(warned).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('opts.apiKey (--key) satisfies the diagnose like a config-file key', async () => {
|
||||
await withEnv({ OPENAI_API_KEY: undefined }, async () => {
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({
|
||||
resolvedModel: OPENAI,
|
||||
resolvedDim: 1536,
|
||||
apiKey: 'sk-from-flag',
|
||||
loadFileConfig: () => ({} as GBrainConfig),
|
||||
skipLiveProbe: true,
|
||||
warn,
|
||||
});
|
||||
expect(r.ok).toBe(true);
|
||||
expect(warned).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('runInitEmbedCheck — live probe (best-effort)', () => {
|
||||
test('config ok + live probe succeeds → live_ok:true, no warning', async () => {
|
||||
// A present key lets embed() reach the installed transport (the transport
|
||||
// seam bypasses the SDK call, not the auth-resolution step).
|
||||
await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
|
||||
__setEmbedTransportForTests(async (args: any) => ({
|
||||
embeddings: (args.values as string[]).map(() => new Array(1536).fill(0)),
|
||||
usage: { tokens: 1 },
|
||||
}) as any);
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({ resolvedModel: OPENAI, resolvedDim: 1536, loadFileConfig: () => ({} as GBrainConfig), warn });
|
||||
expect(r.ok).toBe(true);
|
||||
expect(r.live_ok).toBe(true);
|
||||
expect(warned).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('config ok + live probe FAILS → ok stays true, warns, never throws', async () => {
|
||||
await withEnv({ OPENAI_API_KEY: 'sk-test' }, async () => {
|
||||
__setEmbedTransportForTests(async () => { throw new Error('401 unauthorized: bad api key'); });
|
||||
const { warn, warned } = capture();
|
||||
const r = await runInitEmbedCheck({ resolvedModel: OPENAI, resolvedDim: 1536, loadFileConfig: () => ({} as GBrainConfig), warn });
|
||||
expect(r.ok).toBe(true);
|
||||
expect(r.live_ok).toBe(false);
|
||||
expect(r.live_reason).toBe('auth');
|
||||
expect(warned).toHaveLength(1);
|
||||
expect(warned[0]).toContain('test embed failed');
|
||||
});
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user