mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* v0.36.5.0 feat: secure DATABASE_URL access for shell jobs (inherit: ["database_url"]) Replaces PR #1137's plaintext-config / plaintext-env workarounds with code. Shell-job params gain `inherit: ["database_url"]`, validated pre-enqueue in both the CLI (`gbrain jobs submit`) and `submit_job` MCP op handler. Worker resolves the value from its own loadConfig() at child-spawn time; the persisted `minion_jobs.data` row stores only the name. Plain `env: { GBRAIN_DATABASE_URL: ... }` / `env: { DATABASE_URL: ... }` / `env: { GBRAIN_DIRECT_DATABASE_URL: ... }` are rejected pre-enqueue with a paste-ready hint pointing at `inherit:`. Codex pre-landing review caught two bypasses + one missing shadow name: - H1: cmd/argv inline-secret regex scan (cmd:"GBRAIN_DATABASE_URL=... gbrain sync" was a clean bypass — fixed) - H3: GBRAIN_DIRECT_DATABASE_URL added to shadowKeys - H2: honest docs about output-side leakage (stdout_tail/stderr_tail can still carry the value if the script prints it; that's the script author's responsibility, not gbrain's) Also: gbrain doctor learns home_dir_in_worktree (warns when ~/.gbrain lives inside a git worktree); ~/.gbrain/.gitignore retroactive via saveConfig + post-upgrade. New canonical guide: docs/guides/agent-to-gbrain.md (two-domain framing for downstream agent authors: MCP ops via OAuth vs localOnly admin ops via shell-job inherit:). Closes #1137. Tests: +53 new (21 validator + 12 inherit-record + 6 ensureGitignore + 5 doctor + 2 PGLite E2E + 7 codex-driven H1/H3 cases). Credit: @wintermute filed PR #1137 which made the env-stripping gap visible enough to fix in code. Thank you. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * v0.36.5.0 redesign: free-form inherit:, drop closed enum User feedback: "agent spawning minions should have agency to do what it wants with secrets and pass only the ones that it needs. don't be a security nazi please." Replaces the closed INHERITABLE enum (database_url only) with three small helpers in shell-inherit.ts: - INHERIT_NAME_RE: snake_case shape guard. Rejects __proto__, leading underscore, uppercase, path-traversal. Prototype-pollution defense. - deriveEnvKey(name): config-key → child-env-key. Uppercase by default with one override: database_url → GBRAIN_DATABASE_URL. - resolveInheritValue(cfg, name): value lookup with Object.hasOwn. inherit: now accepts any snake_case config-key the worker has. Agent picks what it needs per-job (database_url, anthropic_api_key, voyage_api_key, or any custom field). Validator does NOT police WHICH keys — single-uid trust model treats agent as peer of worker. Drops the v0.36.5.0-RC rules that were paternalistic for the actual threat model: - closed-enum check - env-shadow rejection - cmd/argv inline-secret scan Keeps the parts that defend real problems: - pre-enqueue validation (closes the persistence-before-throw window) - snake_case regex (prototype-pollution + audit-log readability) - fail-fast on missing config value (UX guardrail, not security) Tests: shell-validate (existing rules + new free-form + prototype-pollution defense + T1 regression guard) and shell-inherit (regex matrix, deriveEnvKey per-name, resolveInheritValue with hasOwn defense). E2E case now exercises inherit:["anthropic_api_key"] to prove genuinely free-form. Docs and CHANGELOG rewritten to reflect the open design + the design-arc story (closed → cut → free-form). Migration file too. 7653 unit tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * v0.36.5.0 add: redact_secrets opt-in for stdout/stderr scrubbing Honest defense for the documented output-side leakage. When a script prints an inherited secret, the value lands plaintext in result.stdout_tail / result.stderr_tail / error_text. v0.36.5.0 adds: - `redact_secrets: true` ShellJobParams field - `--redact-secrets` CLI convenience flag on `gbrain jobs submit shell` - shell-redact.ts: pure `redactSecretsInText(text, secrets)` helper (string-mode replaceAll; regex metachars in values stay literal) - Handler post-processes both tails before throw/return, so the persisted row carries `<REDACTED:name>` tokens instead of values Only inherit-resolved values are scrubbed. env: values are not (those are the agent's "fine in the row" channel by design). Heuristic — defeats accidental `echo "$GBRAIN_DATABASE_URL"`, not adversarial encode-then-print. Default false for back-compat. Tests: - test/minions-shell-redact.test.ts (9 cases): pure-function behavior, regex-metachar safety, multi-secret independent redaction, substring overlap, empty-input/map edge cases - test/minions-shell-validate.test.ts: +4 cases for redact_secrets shape - test/e2e/minions-shell-pglite.test.ts: +2 cases proving redact_secrets: true scrubs persisted row AND redact_secrets:false preserves plaintext (back-compat regression guard) Docs + CHANGELOG + migration file + CLAUDE.md updated. 7667 unit tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
248 lines
10 KiB
TypeScript
248 lines
10 KiB
TypeScript
/**
|
|
* Tests for `src/core/minions/handlers/shell-validate.ts` — the pre-enqueue
|
|
* validator.
|
|
*
|
|
* v0.36.5.0 design: `inherit:` is free-form (any snake_case config-key name).
|
|
* No closed-enum check, no env-shadow rejection, no inline-cmd scan. The
|
|
* single-uid trust model treats the agent as a peer of the worker — it
|
|
* decides which secrets to pass.
|
|
*
|
|
* What the validator DOES check:
|
|
* - Shape (cmd XOR argv, cwd absolute, env is string→string)
|
|
* - `inherit` is array of snake_case strings (prototype-pollution defense)
|
|
* - Every `inherit` name resolves on `loadConfig()` (UX fail-fast)
|
|
*
|
|
* The T1 regression guard at the bottom pins the load-bearing invariant:
|
|
* validation throws BEFORE any persistence call.
|
|
*/
|
|
|
|
import { describe, test, expect } from 'bun:test';
|
|
import { validateShellJobParams } from '../src/core/minions/handlers/shell-validate.ts';
|
|
import { UnrecoverableError } from '../src/core/minions/types.ts';
|
|
import type { GBrainConfig } from '../src/core/config.ts';
|
|
|
|
const dbUrl = 'postgresql://test:test@localhost:5432/test';
|
|
const fakeCfg: GBrainConfig = {
|
|
engine: 'postgres',
|
|
database_url: dbUrl,
|
|
anthropic_api_key: 'sk-ant-test',
|
|
openai_api_key: 'sk-test',
|
|
};
|
|
|
|
describe('validateShellJobParams — existing param shape checks', () => {
|
|
test('cmd XOR argv: both → reject', () => {
|
|
expect(() => validateShellJobParams({ cmd: 'echo', argv: ['echo'], cwd: '/tmp' }, { config: fakeCfg }))
|
|
.toThrow(UnrecoverableError);
|
|
});
|
|
test('cmd XOR argv: neither → reject', () => {
|
|
expect(() => validateShellJobParams({ cwd: '/tmp' }, { config: fakeCfg }))
|
|
.toThrow(UnrecoverableError);
|
|
});
|
|
test('cwd must be absolute', () => {
|
|
expect(() => validateShellJobParams({ cmd: 'echo', cwd: 'relative/path' }, { config: fakeCfg }))
|
|
.toThrow(/absolute path/);
|
|
});
|
|
test('cwd required', () => {
|
|
expect(() => validateShellJobParams({ cmd: 'echo', cwd: '' }, { config: fakeCfg }))
|
|
.toThrow(/cwd/);
|
|
});
|
|
test('env must be object of string values', () => {
|
|
expect(() => validateShellJobParams({ cmd: 'echo', cwd: '/tmp', env: 'oops' as unknown as Record<string, string> }, { config: fakeCfg }))
|
|
.toThrow(/env/);
|
|
expect(() => validateShellJobParams({ cmd: 'echo', cwd: '/tmp', env: { K: 1 as unknown as string } }, { config: fakeCfg }))
|
|
.toThrow(/string/);
|
|
});
|
|
test('happy path: cmd + cwd accepted', () => {
|
|
const p = validateShellJobParams({ cmd: 'echo hi', cwd: '/tmp' }, { config: fakeCfg });
|
|
expect(p.cmd).toBe('echo hi');
|
|
expect(p.argv).toBeUndefined();
|
|
expect(p.cwd).toBe('/tmp');
|
|
});
|
|
});
|
|
|
|
describe('inherit — free-form config-key names (v0.36.5.0)', () => {
|
|
test('inherit:["database_url"] accepted', () => {
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['database_url'] },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.inherit).toEqual(['database_url']);
|
|
});
|
|
test('inherit:["anthropic_api_key"] accepted (was scope-creep in closed enum)', () => {
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['anthropic_api_key'] },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.inherit).toEqual(['anthropic_api_key']);
|
|
});
|
|
test('inherit multiple keys at once', () => {
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['database_url', 'anthropic_api_key', 'openai_api_key'] },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.inherit).toEqual(['database_url', 'anthropic_api_key', 'openai_api_key']);
|
|
});
|
|
test('inherit must be an array', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: 'database_url' as unknown as string[] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/inherit must be an array/);
|
|
});
|
|
test('inherit non-string element rejected', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: [1] as unknown as string[] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/non-empty strings/);
|
|
});
|
|
test('inherit empty string rejected', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: [''] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/non-empty strings/);
|
|
});
|
|
});
|
|
|
|
describe('inherit — snake_case shape guard (prototype-pollution defense)', () => {
|
|
test('"__proto__" rejected (not snake_case shape)', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['__proto__'] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/must match \[a-z\]/);
|
|
});
|
|
test('"constructor" rejected (uppercase letters)', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['constructor'] as unknown as string[] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/worker has no constructor configured/);
|
|
});
|
|
test('path-traversal-looking name rejected', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['../etc/passwd'] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/must match \[a-z\]/);
|
|
});
|
|
test('"FOO_BAR" (uppercase) rejected — config keys are snake_case', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['FOO_BAR'] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/must match \[a-z\]/);
|
|
});
|
|
test('digits-after-letter allowed (matches regex)', () => {
|
|
// Won't actually resolve since fakeCfg doesn't have field2 — fail-fast hits
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['field2'] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/worker has no field2 configured/);
|
|
});
|
|
});
|
|
|
|
describe('inherit — fail-fast on missing config value', () => {
|
|
test('inherit:["database_url"] + config without database_url → reject with set-hint', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['database_url'] },
|
|
{ config: { engine: 'postgres' } as GBrainConfig },
|
|
)).toThrow(/gbrain config set database_url/);
|
|
});
|
|
test('inherit:["database_url"] + null config → reject', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['database_url'] },
|
|
{ config: null },
|
|
)).toThrow(/worker has no database_url/);
|
|
});
|
|
test('inherit:["voyage_api_key"] when not set → reject (fakeCfg has only db_url + anthropic + openai)', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['voyage_api_key'] },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/worker has no voyage_api_key configured/);
|
|
});
|
|
test('inherit:["database_url"] + empty-string database_url → reject', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['database_url'] },
|
|
{ config: { engine: 'postgres', database_url: '' } },
|
|
)).toThrow(/worker has no database_url/);
|
|
});
|
|
});
|
|
|
|
describe('what the validator deliberately does NOT do (agency for the agent)', () => {
|
|
test('caller can use env: for ANY key, including ones with secret-looking names', () => {
|
|
// No shadow rejection. Agent decides if they want to put a URL in env:
|
|
// directly (and accept that it lands in the row plaintext). v0.36.5.0
|
|
// honors the agent's call.
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', env: { GBRAIN_DATABASE_URL: dbUrl, DATABASE_URL: dbUrl } },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.env).toEqual({ GBRAIN_DATABASE_URL: dbUrl, DATABASE_URL: dbUrl });
|
|
});
|
|
test('caller can put inline secret-key=value in cmd', () => {
|
|
// No inline-cmd scan. The agent knows what it's writing.
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'GBRAIN_DATABASE_URL=postgresql://... gbrain sync', cwd: '/tmp' },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.cmd).toContain('GBRAIN_DATABASE_URL');
|
|
});
|
|
test('inherit + env: with overlapping intent both work (last write wins per overlay order)', () => {
|
|
// Agent might want inherit for value-from-config AND env: for an
|
|
// additional non-secret. Both are honored.
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['database_url'], env: { MY_FLAG: '1' } },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.inherit).toEqual(['database_url']);
|
|
expect(p.env).toEqual({ MY_FLAG: '1' });
|
|
});
|
|
});
|
|
|
|
describe('redact_secrets shape check', () => {
|
|
test('redact_secrets: true accepted', () => {
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', inherit: ['database_url'], redact_secrets: true },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.redact_secrets).toBe(true);
|
|
});
|
|
test('redact_secrets: false accepted', () => {
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', redact_secrets: false },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.redact_secrets).toBe(false);
|
|
});
|
|
test('redact_secrets: undefined is fine (default)', () => {
|
|
const p = validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp' },
|
|
{ config: fakeCfg },
|
|
);
|
|
expect(p.redact_secrets).toBeUndefined();
|
|
});
|
|
test('redact_secrets: non-boolean rejected', () => {
|
|
expect(() => validateShellJobParams(
|
|
{ cmd: 'echo', cwd: '/tmp', redact_secrets: 'yes' as unknown as boolean },
|
|
{ config: fakeCfg },
|
|
)).toThrow(/redact_secrets must be a boolean/);
|
|
});
|
|
});
|
|
|
|
describe('T1 regression guard: validation runs BEFORE persistence', () => {
|
|
// This test pins the load-bearing invariant codex caught: validation must
|
|
// throw before any persistence call. If a future refactor moves the
|
|
// validation call back into the shell.ts handler, this test fails.
|
|
test('bad payload throws synchronously, no queue.add could have been called', () => {
|
|
let queueAddCalled = false;
|
|
const fakeQueueAdd = () => { queueAddCalled = true; };
|
|
|
|
// Bad shape: inherit name doesn't pass snake_case regex.
|
|
const data = { cmd: 'echo', cwd: '/tmp', inherit: ['NotSnake'] };
|
|
let validatorThrew = false;
|
|
try {
|
|
validateShellJobParams(data, { config: fakeCfg });
|
|
fakeQueueAdd();
|
|
} catch {
|
|
validatorThrew = true;
|
|
}
|
|
expect(validatorThrew).toBe(true);
|
|
expect(queueAddCalled).toBe(false);
|
|
});
|
|
});
|