mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* fix: lightweight /health endpoint — SELECT 1 instead of getStats() On large brains (96K+ pages), getStats() runs 6× count(*) queries that routinely exceed the 3s HEALTH_TIMEOUT_MS through PgBouncer. This produces false 503s that cause external health monitors (cron, Fly.io, k8s) to restart otherwise-healthy servers — which in turn creates advisory lock pile-ups when multiple serve instances compete for the migration lock. Changes: - /health now runs `SELECT 1` for liveness (sub-millisecond) - ?full=true opt-in preserves the old getStats() behavior - /admin/api/health-indicators still returns full stats - probeHealth() retained for callers that need it * refactor(health): extract probeLiveness, move full stats to /admin/api/full-stats Addresses outside-voice review of PR #701. The original ?full=true query-param escape hatch was withdrawn because the loopback IP gate's correctness depended on app.set('trust proxy', 'loopback') semantics holding under proxy/XFF misconfiguration, and the PR's own comment misidentified /admin/api/health-indicators as a full-stats endpoint when it actually returns only {expiring_soon, error_rate}. Changes: - src/commands/serve-http.ts: new probeLiveness(sql, engineName, version, timeoutMs) helper next to probeHealth. Same shape, same return type, same finally-block clearTimeout discipline. /health is now a 2-line dispatch through probeLiveness. Removes ?full=true entirely. Adds new admin route /admin/api/full-stats behind the existing requireAdmin middleware that returns probeHealth(engine, ...) — same body shape /health used to expose (status, version, engine, page_count, chunk_count, embedded_count, link_count, tag_count, timeline_entry_count). - test/serve-http-health.test.ts: 4 new probeLiveness cases (success-shape regression with exact-keys assertion, timeout, db-error, timer-cleanup under 100 concurrent probes). - test/e2e/serve-http-oauth.test.ts: existing /health body-shape assertion rewritten to the liveness-only contract (page_count must NOT be present); 2 new admin-stats cases (401 without cookie, 200 with magic-link-derived admin cookie returns getStats() body). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: bump version and changelog (v0.28.10) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: update CLAUDE.md serve-http.ts annotation for v0.28.10 split Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(claude): explicit "run E2E without asking" + schema-bootstrap step The previous wording ("Always run E2E tests when they exist") was easy to read as a soft preference; in practice agents kept proposing the run instead of just doing it. Make the policy unmistakable: if there's a relevant E2E and you want to verify behavior, just spin up the DB and run. Also documents the schema-bootstrap step that bit a fresh container today — `oauth_clients` doesn't exist on a virgin pgvector image until `gbrain doctor` (or any engine-connecting command) triggers `initSchema()`. `apply-migrations` alone runs ALTER-style migrations on top of an already-bootstrapped schema; it does not seed base tables. Tests that bypass the engine via execSync against `gbrain auth register-client` hit the DB directly and need bootstrap first. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(serve-http): persist mcp_request_log on every JSON-RPC method + admin-scope F7 tests Closes the 4 pre-existing E2E failures in test/e2e/serve-http-oauth.test.ts that surfaced when DATABASE_URL was set on the v0.28.10 branch. The branch isn't the cause — these were broken on master too (verified by checking out origin/master's serve-http.ts + test file: 0/4 pass). Owning them here as a bisectable commit. Two root causes, both in serve-http.ts's /mcp logging + scope discipline. 1. mcp_request_log was only INSERTed inside the tools/call success/error paths. tools/list, the unknown-op early-return, and the insufficient-scope early-return all returned without logging. The v0.26.3 persistence regression test calls tools/list + tools/call non-existent and expects >= 2 rows; on the prior implementation it got 0. The agent_name resolution test (single tools/list, expects the row) had the same shape. Fix: log every JSON-RPC method exit point. tools/list logs operation = 'tools/list' with status='success' (lists never fail). Unknown-op logs operation = the attempted name with error_message starting 'unknown_operation:'. Insufficient-scope logs operation = the attempted name with error_message 'insufficient_scope: requires <scope>'. Admin agents auditing /admin/api/requests now see the full attempt log, not just successful valid-op calls. 2. The F7 RCE-regression tests minted 'read write' tokens to assert submit_job for protected names ('shell', 'subagent') gets rejected. But submit_job's required scope is 'admin' (set by hasScope-aware v0.28 enforcement), so a 'read write' token gets rejected with insufficient_scope BEFORE reaching the F7 protected-name guard at operations.ts:1527. The test's assertion checked for 'permission_denied' / 'cannot be submitted over MCP' — neither appears in an insufficient_scope response — so 'rejected' computed to false even though the call was actually rejected. Worse, if someone removed the F7 guard, the test would still pass because scope check would catch it: regression-test integrity failure. Fix: register the e2e-oauth-test client with admin in its allowed scopes (was 'read write', now 'read write admin'), and have F7 tests mint admin-scoped tokens explicitly. Adding admin to the client's allowed ceiling does not auto-grant it to subset-mint calls — other tests minting 'read' / 'read write' still get the subset they ask for. The persistence test's assertion 'rows.find(r => r.operation === "tools/call")' was also updated to match the actual logging convention (operation = inner tool name on call paths, JSON-RPC method on list/scope/unknown paths). E2E result: 29/29 pass on a fresh pgvector container (fixed 4, kept the 25 that were passing). Unit suite: 4191 pass, 0 fail, unchanged. Typecheck: clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: regenerate llms-full.txt after CLAUDE.md update Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
153 lines
6.5 KiB
TypeScript
153 lines
6.5 KiB
TypeScript
/**
|
|
* Tests for probeHealth(), probeLiveness(), and HEALTH_TIMEOUT_MS in
|
|
* src/commands/serve-http.ts.
|
|
*
|
|
* v0.28.10 split: /health now calls probeLiveness (sql`SELECT 1`); the heavier
|
|
* probeHealth (engine.getStats()) moved behind requireAdmin at
|
|
* /admin/api/full-stats. Both share ProbeHealthResult so the route handlers
|
|
* stay 2-line dispatches.
|
|
*
|
|
* Calls each probe directly with a mock — no Express test client, no module
|
|
* mocking. Each probe gets happy / timeout / db-error coverage.
|
|
*
|
|
* Express-layer wiring (timeout actually propagates through the route, body
|
|
* shape after JSON serialization) is covered by /health + /admin/api/full-stats
|
|
* cases in test/e2e/serve-http-oauth.test.ts.
|
|
*/
|
|
|
|
import { describe, test, expect } from 'bun:test';
|
|
import { HEALTH_TIMEOUT_MS, probeHealth, probeLiveness } from '../src/commands/serve-http.ts';
|
|
import type { BrainEngine } from '../src/core/engine.ts';
|
|
import type { SqlQuery } from '../src/core/oauth-provider.ts';
|
|
|
|
/**
|
|
* Minimal mock engine: only `getStats()` is exercised by probeHealth.
|
|
* Cast to BrainEngine is safe — probeHealth doesn't touch other methods.
|
|
*/
|
|
function makeMockEngine(getStats: () => Promise<unknown>): BrainEngine {
|
|
return { getStats } as unknown as BrainEngine;
|
|
}
|
|
|
|
/**
|
|
* Minimal mock sql tag: probeLiveness only awaits the result of `sql\`SELECT 1\``
|
|
* — the tag function's return value is what's raced, success/throw is what
|
|
* matters. We ignore the template strings and simulate a connection by calling
|
|
* the supplied factory.
|
|
*/
|
|
function makeMockSql(fn: () => Promise<unknown>): SqlQuery {
|
|
const tag: any = (_strings: TemplateStringsArray, ..._values: unknown[]) => fn();
|
|
return tag as SqlQuery;
|
|
}
|
|
|
|
describe('HEALTH_TIMEOUT_MS', () => {
|
|
test('exported as 3000 (Fly.io headroom over the 5s default)', () => {
|
|
expect(HEALTH_TIMEOUT_MS).toBe(3000);
|
|
});
|
|
});
|
|
|
|
describe('probeHealth', () => {
|
|
test('happy path: returns 200 + status:ok + spread stats', async () => {
|
|
const engine = makeMockEngine(async () => ({ pages: 42, links: 10 }));
|
|
const result = await probeHealth(engine, 'pglite', '0.27.1', 100);
|
|
expect(result.ok).toBe(true);
|
|
expect(result.status).toBe(200);
|
|
if (result.ok) {
|
|
expect(result.body.status).toBe('ok');
|
|
expect(result.body.version).toBe('0.27.1');
|
|
expect(result.body.engine).toBe('pglite');
|
|
expect(result.body.pages).toBe(42);
|
|
expect(result.body.links).toBe(10);
|
|
}
|
|
});
|
|
|
|
test('timeout path: getStats() hangs forever → 503 with health_timeout description within 1s', async () => {
|
|
const engine = makeMockEngine(() => new Promise(() => { /* never resolves */ }));
|
|
const start = Date.now();
|
|
const result = await probeHealth(engine, 'pglite', '0.27.1', 100);
|
|
const elapsed = Date.now() - start;
|
|
expect(elapsed).toBeLessThan(1000);
|
|
expect(result.ok).toBe(false);
|
|
expect(result.status).toBe(503);
|
|
if (!result.ok) {
|
|
expect(result.body.error).toBe('service_unavailable');
|
|
expect(result.body.error_description).toBe(
|
|
'Health check timed out (database pool may be saturated)',
|
|
);
|
|
}
|
|
});
|
|
|
|
test('db-error path: getStats() rejects → 503 with database_failed description', async () => {
|
|
const engine = makeMockEngine(() => Promise.reject(new Error('ECONNREFUSED')));
|
|
const result = await probeHealth(engine, 'postgres', '0.27.1', 100);
|
|
expect(result.ok).toBe(false);
|
|
expect(result.status).toBe(503);
|
|
if (!result.ok) {
|
|
expect(result.body.error).toBe('service_unavailable');
|
|
expect(result.body.error_description).toBe('Database connection failed');
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('probeLiveness (v0.28.10)', () => {
|
|
test('happy path: returns 200 + status:ok with NO engine-stats fields', async () => {
|
|
const sql = makeMockSql(async () => [{ '?column?': 1 }]);
|
|
const result = await probeLiveness(sql, 'postgres', '0.28.10', 100);
|
|
expect(result.ok).toBe(true);
|
|
expect(result.status).toBe(200);
|
|
if (result.ok) {
|
|
expect(result.body.status).toBe('ok');
|
|
expect(result.body.version).toBe('0.28.10');
|
|
expect(result.body.engine).toBe('postgres');
|
|
// Regression: the lightweight body must NOT spread getStats() fields.
|
|
// The original PR's pre-refactor /health leaked page_count etc.;
|
|
// tightening this assertion is the iron-rule regression test.
|
|
expect(Object.keys(result.body).sort()).toEqual(['engine', 'status', 'version']);
|
|
expect((result.body as Record<string, unknown>).page_count).toBeUndefined();
|
|
expect((result.body as Record<string, unknown>).chunk_count).toBeUndefined();
|
|
}
|
|
});
|
|
|
|
test('timeout path: sql hangs → 503 with health_timeout description within 1s', async () => {
|
|
const sql = makeMockSql(() => new Promise(() => { /* never resolves */ }));
|
|
const start = Date.now();
|
|
const result = await probeLiveness(sql, 'postgres', '0.28.10', 100);
|
|
const elapsed = Date.now() - start;
|
|
expect(elapsed).toBeLessThan(1000);
|
|
expect(result.ok).toBe(false);
|
|
expect(result.status).toBe(503);
|
|
if (!result.ok) {
|
|
expect(result.body.error).toBe('service_unavailable');
|
|
expect(result.body.error_description).toBe(
|
|
'Health check timed out (database pool may be saturated)',
|
|
);
|
|
}
|
|
});
|
|
|
|
test('db-error path: sql throws → 503 with database_failed description', async () => {
|
|
const sql = makeMockSql(() => Promise.reject(new Error('ECONNREFUSED')));
|
|
const result = await probeLiveness(sql, 'postgres', '0.28.10', 100);
|
|
expect(result.ok).toBe(false);
|
|
expect(result.status).toBe(503);
|
|
if (!result.ok) {
|
|
expect(result.body.error).toBe('service_unavailable');
|
|
expect(result.body.error_description).toBe('Database connection failed');
|
|
}
|
|
});
|
|
|
|
test('timer-cleanup: 100 fast successful probes do not leak pending timers', async () => {
|
|
const sql = makeMockSql(async () => [{ '?column?': 1 }]);
|
|
// Snapshot active handles before; same after. If the finally-block
|
|
// clearTimeout regressed, every probe would leak a 100ms-pending timer.
|
|
const beforeHandles = (process as any)._getActiveHandles?.()?.length ?? 0;
|
|
await Promise.all(
|
|
Array.from({ length: 100 }, () => probeLiveness(sql, 'postgres', '0.28.10', 100)),
|
|
);
|
|
// Allow microtask + process tick drain to let any leaked timers settle.
|
|
await new Promise(r => setImmediate(r));
|
|
const afterHandles = (process as any)._getActiveHandles?.()?.length ?? 0;
|
|
// Loose bound: bun's internal handles can drift by a small amount across
|
|
// many fetches; we only care that we don't ramp by ~100 leaked timers.
|
|
expect(afterHandles - beforeHandles).toBeLessThan(20);
|
|
});
|
|
});
|