mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* fix(tests): root-cause two master test-infra flakes
gateway.test.ts: add afterAll(resetGateway) hook. The file's tests use
beforeEach(resetGateway) for per-test isolation, but the FINAL test was
leaving configureGateway({env: {OPENAI_API_KEY: 'openai-fake'}}) in the
gateway module state. Sibling files in the same bun shard (e.g.
test/ingestion/ingest-capture.test.ts) then triggered embed() against
the real OpenAI endpoint with the leaked fake key, wedging the shard
with 'Incorrect API key provided: openai-fake'.
header-transport.test.ts → .serial.test.ts: the file mutates RECIPES
(gateway's module-scoped recipe map) plus configureGateway, then asserts
fakeChatFetch was invoked. Under bun's intra-shard parallelism, sibling
files like test/ai/rerank.test.ts race the same state — chat would see
result.text === '[]' instead of 'ok' because another test called
resetGateway between this test's configureGateway and chat. Quarantining
as .serial.test.ts moves the file into the post-parallel serial pass
at --max-concurrency=1 per repo convention.
* refactor(doctor): extract buildChecks seam + behavioral coverage
src/commands/doctor.ts: extract buildChecks(engine, args, dbSource):
Promise<Check[]> from runDoctor. The check-building logic moves into
the new exported function; existing exported computeDoctorReport(checks)
at line 78 stays untouched. runDoctor becomes a thin wrapper:
buildChecks → computeDoctorReport → render + process.exit. All 10
process.exit sites stay in place. The two early-return paths drop
their inline outputResults+process.exit calls and return the partial
check list; the wrapper still produces identical observable output.
test/doctor-behavioral.test.ts (13 cases): pure pure-aggregation cases
pin computeDoctorReport math (3 fails → -60 points, score clamped at 0,
mixed outcome → unhealthy with fail dominating). Orchestrator cases
assert --fast flag honors the skip set, --json doesn't alter the list,
no-engine path returns partial without process.exit, and the snapshot
of load-bearing check names catches accidental drop-outs during
future refactors.
test/doctor-cli-smoke.serial.test.ts (1 case): subprocess smoke spawning
'bun run src/cli.ts doctor --json' against a fresh PGLite tempdir brain.
Catches render-path bugs that buildChecks-only tests miss — the class
the v0.38.2.0 partial-scan wave exposed. Quarantined as .serial because
PGLite write-locks don't play well with parallel runners; skippable via
GBRAIN_SKIP_SUBPROCESS_TESTS=1.
* feat(operations): trust-boundary contract test + filter-bypass shell guard
test/operations-trust-boundary.test.ts (14 cases): hybrid design per
plan D7. Pure assertions over all 74 ops cover the drift-detection
win (every op has a scope; every mutating op has a non-read scope;
hasScope(['read'], op.scope) correctly rejects 'admin' or 'write').
Plus the canonical filter contract: every localOnly: true op is
excluded from operations.filter(op => !op.localOnly). Plus targeted
handler-invocation cases for the two historically-broken HTTP-callable
classes: submit_job(name='shell', ctx.remote=true) MUST reject (F7b
HTTP MCP shell-job RCE class), and search_by_image(image_path,
ctx.remote=true) MUST reject (D18 P0 image-leak class). file_upload
and sync_brain are deliberately omitted from handler-invocation tests
because they're localOnly — calling their handlers directly tests an
impossible production path (codex CMT-3). All 7 localOnly ops are
snapshot-pinned by name to catch future flag-flips.
scripts/check-operations-filter-bypass.sh: greps src/ for any module
that imports the 'operations' value from core/operations.ts outside
the canonical filter site. Three import shapes detected: destructured,
aliased ('as ops'), namespace ('import * as'). Explicit allow-list of
10 known-safe importers with one-line rationale per entry. Plus a
filter-presence check on serve-http.ts that fails if the canonical
filter expression is refactored out. Codex /ship adversarial review
caught the original narrow regex missed aliased + namespace bypasses;
the expanded regex closes that class. Type-only imports of sibling
exports (sourceScopeOpts, OperationContext) are not flagged.
package.json: wires check:operations-filter-bypass into the verify
chain alongside check:jsonb and check:progress.
* refactor(cycle): export runPhaseLint+runPhaseBacklinks; add wrapper tests
src/core/cycle.ts: adds 'export' keyword to two existing phase
functions so behavioral tests can drive them without going through
runCycle's full setup cost. No body changes; no behavior change.
Documented as internal helpers exposed for test-only consumption —
downstream code should NOT take a dependency on them; existing
plan-eng-review D9 explicitly accepted the API-widening tax for
testability.
test/cycle-legacy-phases.test.ts (11 cases): combined file with two
describe blocks per plan D5 (DRY — shared setup, future phase
wrappers land as additional describes). Narrowed to result-mapping
+ error envelope per codex CMT-1 (legacy phases don't extend
BaseCyclePhase and don't take a progress reporter or AbortSignal
directly, so the contract surface is counter → status enum + try/catch
envelope). Cases: clean run → status='ok', partial fix → status='warn'
with dryRun in details, dry-run path doesn't write, throw-from-lib
→ status='fail' with envelope populated (no exception escape).
Verified runLintCore and runBacklinksCore both throw on missing dir,
so the throw-from-lib cases are deterministic.
* chore: bump version and changelog (v0.40.4.1)
E2E + unit test gap coverage wave. Closes 4 audit gaps with new
behavioral coverage (doctor orchestrator + subprocess smoke, operations
trust-boundary contract + filter-bypass guard, cycle phase wrapper
result-mapping). Verifies 3 audit gaps were already covered (ingestion
dedup/daemon/skillpack-load, phantom-redirect, ingestion test-harness).
Root-causes 2 pre-existing master flakes (gateway state leak, header-
transport cross-shard race). Files 5 follow-up TODOs from codex
adversarial-review findings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: note v0.40.4.1 doctor.buildChecks + cycle phase exports in CLAUDE.md
Adds three Key-files entries pinning the v0.40.4.1 test-wave additions:
- doctor.ts extension: buildChecks seam + behavioral tests (13+1 cases)
- cycle.ts extension: runPhaseLint + runPhaseBacklinks exports (11 cases)
- operations-trust-boundary contract + check-operations-filter-bypass.sh
Regenerates llms-full.txt to match (CLAUDE.md edits require build:llms per
project rule, otherwise test/build-llms.test.ts fails in CI shard 1).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(tests): reset gateway in put_page write-through tests to skip embed in CI
CI failure mode: 9 tests in test/ingestion/put-page-write-through.test.ts
failed with `AIConfigError: [embed(zeroentropyai:zembed-1)] Unauthorized`
because put_page's handler at src/core/operations.ts:622 computes
`noEmbed = !isAvailable('embedding')`. When the gateway state has been
configured by a sibling test (or by the cli.ts module-load path reading
.env.testing) with a fake/stale ZEROENTROPY_API_KEY, isAvailable returns
true → put_page tries to embed → the real ZeroEntropy API returns 401.
Local dev passes because real ZE keys are present; CI doesn't have them.
Fix: call resetGateway() in beforeEach so isAvailable('embedding')
returns false → put_page's noEmbed path activates → no network call.
Also reset in afterAll to avoid leaking the cleared state to sibling
files in the same bun shard (the v0.40.4.1 gateway state-leak class
that motivated the earlier gateway.test.ts fix).
The test exercises write-through behavior, not embedding. No need for
a real or fake embed transport — bypass entirely.
* fix(tests): widen brain-writer partial-scan deadlines to absorb CI timing variance
CI failure: scanBrainSources partial-scan state > "hanging COUNT does not
exceed deadline — Promise.race timeout fires" failed once on a GitHub
Actions runner. The test asserted a 100ms deadline budget with a 500ms
bound; observed test duration was 187ms on CI (passes locally 20/20
runs at the original budget).
Root cause: Node.js timer drift under shard parallelism. The deadline
check at src/core/brain-writer.ts:503 uses strict `Date.now() > deadline`,
so when the setTimeout in Promise.race fires exactly at the boundary
(e.g. start+100ms when deadline is start+100ms), the post-await check
sees equality and skips the markRemainingSkipped branch. The test
also asserts elapsed < 500ms; CI overhead can push elapsed past that
bound when setTimeout drifts.
Fix: widen the deadline budget on both deadline-race tests proportionally
(keeps the same 2x ratio that proves "query exceeds deadline"). No
src/ changes — this is purely a test robustness widening.
- "hanging COUNT" test: 100ms → 500ms deadline, 500ms → 2500ms bound
- "slow COUNT" test: 50ms → 250ms deadline, 100ms → 500ms query delay
Verified locally: 20/20 stress runs at the widened budgets, no fails.
* fix(brain-writer): deadline check is >= not > (closes CI flake at boundary)
CI failure recurred: same "hanging COUNT does not exceed deadline" test
failed again at 588ms (past my previous 500ms deadline + 2500ms bound
widening). The root cause isn't test timing — it's an off-by-one in
the source.
src/core/brain-writer.ts had two deadline checks using strict `>`:
- line 445 (between-source abort)
- line 503 (post-COUNT-await re-check)
The Promise.race setTimeout resolves null at exactly `remainingMs` from
now, so post-await Date.now() OFTEN equals the deadline within
integer-ms precision. With `>`, the check skipped → scanOneSource ran
on the source whose budget had just been eaten → that source got
status='scanned' instead of 'skipped'. The test's `expect(firstSource
.status).toBe('skipped')` failed.
Fix: both checks now use `>=`. When Date.now() equals deadline exactly,
the budget IS exhausted — proceeding would let the next source eat its
own budget on top of what's already spent. Matches the boundary the
Promise.race's remainingMs <= 0 immediate-null path uses (line 481).
This is the real fix for the v0.40.x CI flakes; my earlier test-budget
widening papered over the symptom without closing the boundary. Kept
the wider 500ms deadline for headroom but added a comment pointing at
the operator fix as the load-bearing change.
Verified: 20/20 stress runs green locally after the operator fix.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
266 lines
9.8 KiB
TypeScript
266 lines
9.8 KiB
TypeScript
/**
|
|
* Transport-level header sweep (v0.37.2.0).
|
|
*
|
|
* Codex correctly noted that the IRON RULE contract tests only prove the
|
|
* return shape of `applyResolveAuth` — they DO NOT prove that the AI SDK
|
|
* (createOpenAICompatible) actually applies our default_headers on outgoing
|
|
* requests. This file closes that gap by injecting a custom fetch wrapper
|
|
* via `resolveOpenAICompatConfig` and asserting every assembled header
|
|
* (Authorization + default_headers) reaches the wire.
|
|
*
|
|
* Three cases:
|
|
* 1. embed — SDK textEmbeddingModel path through createOpenAICompatible
|
|
* 2. chat — SDK languageModel path through createOpenAICompatible
|
|
* 3. rerank — manual HTTP path (no SDK adapter) — uses __setRerankTransportForTests
|
|
*
|
|
* Synthetic recipes are registered in RECIPES at beforeAll, removed at
|
|
* afterAll — same Map-mutation pattern any future recipe-shape test can reuse.
|
|
*/
|
|
|
|
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
|
|
import {
|
|
configureGateway,
|
|
resetGateway,
|
|
embed,
|
|
chat,
|
|
rerank,
|
|
__setRerankTransportForTests,
|
|
} from '../../src/core/ai/gateway.ts';
|
|
import { RECIPES } from '../../src/core/ai/recipes/index.ts';
|
|
import type { Recipe } from '../../src/core/ai/types.ts';
|
|
|
|
// --- Synthetic embed recipe ---------------------------------------------------
|
|
// Bearer auth + default_headers (HTTP-Referer + X-OpenRouter-Title + X-Title).
|
|
// resolveOpenAICompatConfig injects a fetch wrapper that captures the outgoing
|
|
// request's headers + body for assertions.
|
|
|
|
let lastEmbedRequest: { url: string; headers: Record<string, string>; body: string | null } | null = null;
|
|
const fakeEmbedFetch: (input: any, init?: any) => Promise<Response> = async (input, init) => {
|
|
const url = typeof input === 'string' ? input : input instanceof Request ? input.url : (input as URL).toString();
|
|
const headers: Record<string, string> = {};
|
|
const h = new Headers(init?.headers ?? (input instanceof Request ? input.headers : undefined));
|
|
h.forEach((v, k) => { headers[k.toLowerCase()] = v; });
|
|
const body = init?.body
|
|
? (typeof init.body === 'string' ? init.body : null)
|
|
: (input instanceof Request ? await input.text() : null);
|
|
lastEmbedRequest = { url, headers, body };
|
|
// OpenAI-compatible /embeddings response shape — one embedding for "hello".
|
|
const json = {
|
|
object: 'list',
|
|
data: [{ object: 'embedding', index: 0, embedding: Array.from({ length: 8 }, () => 0.1) }],
|
|
model: 'fake-embed-model',
|
|
usage: { prompt_tokens: 1, total_tokens: 1 },
|
|
};
|
|
return new Response(JSON.stringify(json), {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/json' },
|
|
});
|
|
};
|
|
|
|
const SYNTHETIC_EMBED_RECIPE: Recipe = {
|
|
id: 'syntethic-embed-headers',
|
|
name: 'Synthetic Embed Headers',
|
|
tier: 'openai-compat',
|
|
implementation: 'openai-compatible',
|
|
base_url_default: 'https://synthetic.test/v1',
|
|
auth_env: { required: ['SYNTHETIC_EMBED_KEY'] },
|
|
touchpoints: {
|
|
embedding: {
|
|
models: ['fake-embed-model'],
|
|
default_dims: 8,
|
|
max_batch_tokens: 8192,
|
|
},
|
|
},
|
|
default_headers: {
|
|
'HTTP-Referer': 'https://gbrain.ai',
|
|
'X-OpenRouter-Title': 'gbrain',
|
|
'X-Title': 'gbrain',
|
|
},
|
|
resolveOpenAICompatConfig() {
|
|
return {
|
|
baseURL: 'https://synthetic.test/v1',
|
|
fetch: fakeEmbedFetch as unknown as typeof fetch,
|
|
};
|
|
},
|
|
};
|
|
|
|
// --- Synthetic chat recipe ---------------------------------------------------
|
|
|
|
let lastChatRequest: { url: string; headers: Record<string, string>; body: string | null } | null = null;
|
|
const fakeChatFetch: (input: any, init?: any) => Promise<Response> = async (input, init) => {
|
|
const url = typeof input === 'string' ? input : input instanceof Request ? input.url : (input as URL).toString();
|
|
const headers: Record<string, string> = {};
|
|
const h = new Headers(init?.headers ?? (input instanceof Request ? input.headers : undefined));
|
|
h.forEach((v, k) => { headers[k.toLowerCase()] = v; });
|
|
const body = init?.body
|
|
? (typeof init.body === 'string' ? init.body : null)
|
|
: (input instanceof Request ? await input.text() : null);
|
|
lastChatRequest = { url, headers, body };
|
|
// OpenAI-compatible /chat/completions response shape — one assistant message.
|
|
const json = {
|
|
id: 'fake-chat-1',
|
|
object: 'chat.completion',
|
|
created: 0,
|
|
model: 'fake-chat-model',
|
|
choices: [
|
|
{
|
|
index: 0,
|
|
message: { role: 'assistant', content: 'ok' },
|
|
finish_reason: 'stop',
|
|
},
|
|
],
|
|
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
|
|
};
|
|
return new Response(JSON.stringify(json), {
|
|
status: 200,
|
|
headers: { 'content-type': 'application/json' },
|
|
});
|
|
};
|
|
|
|
const SYNTHETIC_CHAT_RECIPE: Recipe = {
|
|
id: 'syntethic-chat-headers',
|
|
name: 'Synthetic Chat Headers',
|
|
tier: 'openai-compat',
|
|
implementation: 'openai-compatible',
|
|
base_url_default: 'https://synthetic.test/v1',
|
|
auth_env: { required: ['SYNTHETIC_CHAT_KEY'] },
|
|
touchpoints: {
|
|
chat: {
|
|
models: ['fake-chat-model'],
|
|
supports_tools: false,
|
|
supports_subagent_loop: false,
|
|
},
|
|
},
|
|
default_headers: {
|
|
'HTTP-Referer': 'https://gbrain.ai',
|
|
'X-OpenRouter-Title': 'gbrain',
|
|
'X-Title': 'gbrain',
|
|
},
|
|
resolveOpenAICompatConfig() {
|
|
return {
|
|
baseURL: 'https://synthetic.test/v1',
|
|
fetch: fakeChatFetch as unknown as typeof fetch,
|
|
};
|
|
},
|
|
};
|
|
|
|
// --- Synthetic reranker recipe -----------------------------------------------
|
|
|
|
const SYNTHETIC_RERANK_RECIPE: Recipe = {
|
|
id: 'syntethic-rerank-headers',
|
|
name: 'Synthetic Rerank Headers',
|
|
tier: 'openai-compat',
|
|
implementation: 'openai-compatible',
|
|
base_url_default: 'https://synthetic.test/v1',
|
|
auth_env: { required: ['SYNTHETIC_RERANK_KEY'] },
|
|
touchpoints: {
|
|
reranker: {
|
|
models: ['fake-rerank-model'],
|
|
default_model: 'fake-rerank-model',
|
|
max_payload_bytes: 5_000_000,
|
|
},
|
|
},
|
|
default_headers: {
|
|
'HTTP-Referer': 'https://gbrain.ai',
|
|
'X-OpenRouter-Title': 'gbrain',
|
|
'X-Title': 'gbrain',
|
|
},
|
|
};
|
|
|
|
beforeAll(() => {
|
|
// Register synthetic recipes for the lifetime of this test file. RECIPES is
|
|
// a Map; .set/.delete is the natural test seam. No production-code changes
|
|
// are required to enable test-only recipe registration.
|
|
RECIPES.set(SYNTHETIC_EMBED_RECIPE.id, SYNTHETIC_EMBED_RECIPE);
|
|
RECIPES.set(SYNTHETIC_CHAT_RECIPE.id, SYNTHETIC_CHAT_RECIPE);
|
|
RECIPES.set(SYNTHETIC_RERANK_RECIPE.id, SYNTHETIC_RERANK_RECIPE);
|
|
});
|
|
|
|
afterAll(() => {
|
|
RECIPES.delete(SYNTHETIC_EMBED_RECIPE.id);
|
|
RECIPES.delete(SYNTHETIC_CHAT_RECIPE.id);
|
|
RECIPES.delete(SYNTHETIC_RERANK_RECIPE.id);
|
|
__setRerankTransportForTests(null);
|
|
resetGateway();
|
|
});
|
|
|
|
describe('transport-level header sweep (v0.37.2.0)', () => {
|
|
test('1. embed — SDK applies Authorization + default_headers on every request', async () => {
|
|
lastEmbedRequest = null;
|
|
configureGateway({
|
|
embedding_model: `${SYNTHETIC_EMBED_RECIPE.id}:fake-embed-model`,
|
|
embedding_dimensions: 8,
|
|
env: { SYNTHETIC_EMBED_KEY: 'sk-embed-fake' },
|
|
});
|
|
|
|
const result = await embed(['hello']);
|
|
expect(result.length).toBe(1);
|
|
expect(lastEmbedRequest, 'fakeEmbedFetch should have been invoked').not.toBeNull();
|
|
|
|
const h = lastEmbedRequest!.headers;
|
|
expect(h['authorization'], 'Authorization Bearer must be present').toBe('Bearer sk-embed-fake');
|
|
expect(h['http-referer'], 'HTTP-Referer must reach the wire').toBe('https://gbrain.ai');
|
|
expect(h['x-openrouter-title'], 'X-OpenRouter-Title must reach the wire').toBe('gbrain');
|
|
expect(h['x-title'], 'X-Title (back-compat) must reach the wire').toBe('gbrain');
|
|
});
|
|
|
|
test('2. chat — SDK applies Authorization + default_headers on every request', async () => {
|
|
lastChatRequest = null;
|
|
configureGateway({
|
|
chat_model: `${SYNTHETIC_CHAT_RECIPE.id}:fake-chat-model`,
|
|
env: { SYNTHETIC_CHAT_KEY: 'sk-chat-fake' },
|
|
});
|
|
|
|
const result = await chat({
|
|
model: `${SYNTHETIC_CHAT_RECIPE.id}:fake-chat-model`,
|
|
messages: [{ role: 'user', content: 'hello' }],
|
|
});
|
|
expect(result.text).toBe('ok');
|
|
expect(lastChatRequest, 'fakeChatFetch should have been invoked').not.toBeNull();
|
|
|
|
const h = lastChatRequest!.headers;
|
|
expect(h['authorization']).toBe('Bearer sk-chat-fake');
|
|
expect(h['http-referer']).toBe('https://gbrain.ai');
|
|
expect(h['x-openrouter-title']).toBe('gbrain');
|
|
expect(h['x-title']).toBe('gbrain');
|
|
});
|
|
|
|
test('3. rerank — manual HTTP path applies Authorization + default_headers', async () => {
|
|
let capturedHeaders: Record<string, string> | null = null;
|
|
__setRerankTransportForTests(async (_url, init) => {
|
|
const hdrs: Record<string, string> = {};
|
|
new Headers(init.headers).forEach((v, k) => { hdrs[k.toLowerCase()] = v; });
|
|
capturedHeaders = hdrs;
|
|
return new Response(
|
|
JSON.stringify({
|
|
results: [
|
|
{ index: 0, relevance_score: 0.9 },
|
|
{ index: 1, relevance_score: 0.7 },
|
|
],
|
|
}),
|
|
{ status: 200, headers: { 'content-type': 'application/json' } },
|
|
);
|
|
});
|
|
|
|
configureGateway({
|
|
reranker_model: `${SYNTHETIC_RERANK_RECIPE.id}:fake-rerank-model`,
|
|
env: { SYNTHETIC_RERANK_KEY: 'sk-rerank-fake' },
|
|
});
|
|
|
|
const results = await rerank({
|
|
query: 'find relevant docs',
|
|
documents: ['doc a', 'doc b'],
|
|
model: `${SYNTHETIC_RERANK_RECIPE.id}:fake-rerank-model`,
|
|
});
|
|
expect(results.length).toBe(2);
|
|
expect(capturedHeaders, 'rerank transport stub should have been invoked').not.toBeNull();
|
|
|
|
const h = capturedHeaders!;
|
|
expect(h['authorization'], 'rerank: Authorization Bearer must be present').toBe('Bearer sk-rerank-fake');
|
|
expect(h['http-referer'], 'rerank: HTTP-Referer must reach the wire').toBe('https://gbrain.ai');
|
|
expect(h['x-openrouter-title']).toBe('gbrain');
|
|
expect(h['x-title']).toBe('gbrain');
|
|
expect(h['content-type']).toBe('application/json');
|
|
});
|
|
});
|