Files
gbrain/test/serve-http-health.test.ts
T
f7c129407a v0.28.10 fix: lightweight /health endpoint — SELECT 1 instead of getStats() (#701)
* fix: lightweight /health endpoint — SELECT 1 instead of getStats()

On large brains (96K+ pages), getStats() runs 6× count(*) queries that
routinely exceed the 3s HEALTH_TIMEOUT_MS through PgBouncer. This
produces false 503s that cause external health monitors (cron, Fly.io,
k8s) to restart otherwise-healthy servers — which in turn creates
advisory lock pile-ups when multiple serve instances compete for the
migration lock.

Changes:
- /health now runs `SELECT 1` for liveness (sub-millisecond)
- ?full=true opt-in preserves the old getStats() behavior
- /admin/api/health-indicators still returns full stats
- probeHealth() retained for callers that need it

* refactor(health): extract probeLiveness, move full stats to /admin/api/full-stats

Addresses outside-voice review of PR #701. The original ?full=true query-param
escape hatch was withdrawn because the loopback IP gate's correctness depended
on app.set('trust proxy', 'loopback') semantics holding under proxy/XFF
misconfiguration, and the PR's own comment misidentified
/admin/api/health-indicators as a full-stats endpoint when it actually returns
only {expiring_soon, error_rate}.

Changes:
- src/commands/serve-http.ts: new probeLiveness(sql, engineName, version,
  timeoutMs) helper next to probeHealth. Same shape, same return type, same
  finally-block clearTimeout discipline. /health is now a 2-line dispatch
  through probeLiveness. Removes ?full=true entirely. Adds new admin route
  /admin/api/full-stats behind the existing requireAdmin middleware that
  returns probeHealth(engine, ...) — same body shape /health used to expose
  (status, version, engine, page_count, chunk_count, embedded_count,
  link_count, tag_count, timeline_entry_count).
- test/serve-http-health.test.ts: 4 new probeLiveness cases (success-shape
  regression with exact-keys assertion, timeout, db-error, timer-cleanup
  under 100 concurrent probes).
- test/e2e/serve-http-oauth.test.ts: existing /health body-shape assertion
  rewritten to the liveness-only contract (page_count must NOT be present);
  2 new admin-stats cases (401 without cookie, 200 with magic-link-derived
  admin cookie returns getStats() body).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: bump version and changelog (v0.28.10)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: update CLAUDE.md serve-http.ts annotation for v0.28.10 split

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs(claude): explicit "run E2E without asking" + schema-bootstrap step

The previous wording ("Always run E2E tests when they exist") was easy to read
as a soft preference; in practice agents kept proposing the run instead of just
doing it. Make the policy unmistakable: if there's a relevant E2E and you want
to verify behavior, just spin up the DB and run.

Also documents the schema-bootstrap step that bit a fresh container today —
`oauth_clients` doesn't exist on a virgin pgvector image until `gbrain doctor`
(or any engine-connecting command) triggers `initSchema()`. `apply-migrations`
alone runs ALTER-style migrations on top of an already-bootstrapped schema; it
does not seed base tables. Tests that bypass the engine via execSync against
`gbrain auth register-client` hit the DB directly and need bootstrap first.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(serve-http): persist mcp_request_log on every JSON-RPC method + admin-scope F7 tests

Closes the 4 pre-existing E2E failures in test/e2e/serve-http-oauth.test.ts
that surfaced when DATABASE_URL was set on the v0.28.10 branch. The branch
isn't the cause — these were broken on master too (verified by checking
out origin/master's serve-http.ts + test file: 0/4 pass). Owning them
here as a bisectable commit.

Two root causes, both in serve-http.ts's /mcp logging + scope discipline.

1. mcp_request_log was only INSERTed inside the tools/call success/error
   paths. tools/list, the unknown-op early-return, and the
   insufficient-scope early-return all returned without logging. The
   v0.26.3 persistence regression test calls tools/list + tools/call
   non-existent and expects >= 2 rows; on the prior implementation it
   got 0. The agent_name resolution test (single tools/list, expects
   the row) had the same shape.

   Fix: log every JSON-RPC method exit point. tools/list logs operation
   = 'tools/list' with status='success' (lists never fail). Unknown-op
   logs operation = the attempted name with error_message starting
   'unknown_operation:'. Insufficient-scope logs operation = the
   attempted name with error_message 'insufficient_scope: requires
   <scope>'. Admin agents auditing /admin/api/requests now see the
   full attempt log, not just successful valid-op calls.

2. The F7 RCE-regression tests minted 'read write' tokens to assert
   submit_job for protected names ('shell', 'subagent') gets rejected.
   But submit_job's required scope is 'admin' (set by hasScope-aware
   v0.28 enforcement), so a 'read write' token gets rejected with
   insufficient_scope BEFORE reaching the F7 protected-name guard at
   operations.ts:1527. The test's assertion checked for
   'permission_denied' / 'cannot be submitted over MCP' — neither
   appears in an insufficient_scope response — so 'rejected' computed
   to false even though the call was actually rejected. Worse, if
   someone removed the F7 guard, the test would still pass because
   scope check would catch it: regression-test integrity failure.

   Fix: register the e2e-oauth-test client with admin in its allowed
   scopes (was 'read write', now 'read write admin'), and have F7
   tests mint admin-scoped tokens explicitly. Adding admin to the
   client's allowed ceiling does not auto-grant it to subset-mint
   calls — other tests minting 'read' / 'read write' still get the
   subset they ask for.

The persistence test's assertion 'rows.find(r => r.operation ===
"tools/call")' was also updated to match the actual logging convention
(operation = inner tool name on call paths, JSON-RPC method on
list/scope/unknown paths).

E2E result: 29/29 pass on a fresh pgvector container (fixed 4, kept
the 25 that were passing). Unit suite: 4191 pass, 0 fail, unchanged.
Typecheck: clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: regenerate llms-full.txt after CLAUDE.md update

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 13:17:27 -07:00

153 lines
6.5 KiB
TypeScript

/**
* Tests for probeHealth(), probeLiveness(), and HEALTH_TIMEOUT_MS in
* src/commands/serve-http.ts.
*
* v0.28.10 split: /health now calls probeLiveness (sql`SELECT 1`); the heavier
* probeHealth (engine.getStats()) moved behind requireAdmin at
* /admin/api/full-stats. Both share ProbeHealthResult so the route handlers
* stay 2-line dispatches.
*
* Calls each probe directly with a mock — no Express test client, no module
* mocking. Each probe gets happy / timeout / db-error coverage.
*
* Express-layer wiring (timeout actually propagates through the route, body
* shape after JSON serialization) is covered by /health + /admin/api/full-stats
* cases in test/e2e/serve-http-oauth.test.ts.
*/
import { describe, test, expect } from 'bun:test';
import { HEALTH_TIMEOUT_MS, probeHealth, probeLiveness } from '../src/commands/serve-http.ts';
import type { BrainEngine } from '../src/core/engine.ts';
import type { SqlQuery } from '../src/core/oauth-provider.ts';
/**
* Minimal mock engine: only `getStats()` is exercised by probeHealth.
* Cast to BrainEngine is safe — probeHealth doesn't touch other methods.
*/
function makeMockEngine(getStats: () => Promise<unknown>): BrainEngine {
return { getStats } as unknown as BrainEngine;
}
/**
* Minimal mock sql tag: probeLiveness only awaits the result of `sql\`SELECT 1\``
* — the tag function's return value is what's raced, success/throw is what
* matters. We ignore the template strings and simulate a connection by calling
* the supplied factory.
*/
function makeMockSql(fn: () => Promise<unknown>): SqlQuery {
const tag: any = (_strings: TemplateStringsArray, ..._values: unknown[]) => fn();
return tag as SqlQuery;
}
describe('HEALTH_TIMEOUT_MS', () => {
test('exported as 3000 (Fly.io headroom over the 5s default)', () => {
expect(HEALTH_TIMEOUT_MS).toBe(3000);
});
});
describe('probeHealth', () => {
test('happy path: returns 200 + status:ok + spread stats', async () => {
const engine = makeMockEngine(async () => ({ pages: 42, links: 10 }));
const result = await probeHealth(engine, 'pglite', '0.27.1', 100);
expect(result.ok).toBe(true);
expect(result.status).toBe(200);
if (result.ok) {
expect(result.body.status).toBe('ok');
expect(result.body.version).toBe('0.27.1');
expect(result.body.engine).toBe('pglite');
expect(result.body.pages).toBe(42);
expect(result.body.links).toBe(10);
}
});
test('timeout path: getStats() hangs forever → 503 with health_timeout description within 1s', async () => {
const engine = makeMockEngine(() => new Promise(() => { /* never resolves */ }));
const start = Date.now();
const result = await probeHealth(engine, 'pglite', '0.27.1', 100);
const elapsed = Date.now() - start;
expect(elapsed).toBeLessThan(1000);
expect(result.ok).toBe(false);
expect(result.status).toBe(503);
if (!result.ok) {
expect(result.body.error).toBe('service_unavailable');
expect(result.body.error_description).toBe(
'Health check timed out (database pool may be saturated)',
);
}
});
test('db-error path: getStats() rejects → 503 with database_failed description', async () => {
const engine = makeMockEngine(() => Promise.reject(new Error('ECONNREFUSED')));
const result = await probeHealth(engine, 'postgres', '0.27.1', 100);
expect(result.ok).toBe(false);
expect(result.status).toBe(503);
if (!result.ok) {
expect(result.body.error).toBe('service_unavailable');
expect(result.body.error_description).toBe('Database connection failed');
}
});
});
describe('probeLiveness (v0.28.10)', () => {
test('happy path: returns 200 + status:ok with NO engine-stats fields', async () => {
const sql = makeMockSql(async () => [{ '?column?': 1 }]);
const result = await probeLiveness(sql, 'postgres', '0.28.10', 100);
expect(result.ok).toBe(true);
expect(result.status).toBe(200);
if (result.ok) {
expect(result.body.status).toBe('ok');
expect(result.body.version).toBe('0.28.10');
expect(result.body.engine).toBe('postgres');
// Regression: the lightweight body must NOT spread getStats() fields.
// The original PR's pre-refactor /health leaked page_count etc.;
// tightening this assertion is the iron-rule regression test.
expect(Object.keys(result.body).sort()).toEqual(['engine', 'status', 'version']);
expect((result.body as Record<string, unknown>).page_count).toBeUndefined();
expect((result.body as Record<string, unknown>).chunk_count).toBeUndefined();
}
});
test('timeout path: sql hangs → 503 with health_timeout description within 1s', async () => {
const sql = makeMockSql(() => new Promise(() => { /* never resolves */ }));
const start = Date.now();
const result = await probeLiveness(sql, 'postgres', '0.28.10', 100);
const elapsed = Date.now() - start;
expect(elapsed).toBeLessThan(1000);
expect(result.ok).toBe(false);
expect(result.status).toBe(503);
if (!result.ok) {
expect(result.body.error).toBe('service_unavailable');
expect(result.body.error_description).toBe(
'Health check timed out (database pool may be saturated)',
);
}
});
test('db-error path: sql throws → 503 with database_failed description', async () => {
const sql = makeMockSql(() => Promise.reject(new Error('ECONNREFUSED')));
const result = await probeLiveness(sql, 'postgres', '0.28.10', 100);
expect(result.ok).toBe(false);
expect(result.status).toBe(503);
if (!result.ok) {
expect(result.body.error).toBe('service_unavailable');
expect(result.body.error_description).toBe('Database connection failed');
}
});
test('timer-cleanup: 100 fast successful probes do not leak pending timers', async () => {
const sql = makeMockSql(async () => [{ '?column?': 1 }]);
// Snapshot active handles before; same after. If the finally-block
// clearTimeout regressed, every probe would leak a 100ms-pending timer.
const beforeHandles = (process as any)._getActiveHandles?.()?.length ?? 0;
await Promise.all(
Array.from({ length: 100 }, () => probeLiveness(sql, 'postgres', '0.28.10', 100)),
);
// Allow microtask + process tick drain to let any leaked timers settle.
await new Promise(r => setImmediate(r));
const afterHandles = (process as any)._getActiveHandles?.()?.length ?? 0;
// Loose bound: bun's internal handles can drift by a small amount across
// many fetches; we only care that we don't ramp by ~100 leaked timers.
expect(afterHandles - beforeHandles).toBeLessThan(20);
});
});