From 7a80ce25c2b2c26e1e59975db132ce39d05640b5 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 1 May 2026 08:21:23 -0700 Subject: [PATCH] =?UTF-8?q?feat(v0.25.0):=20CONTRIBUTOR=5FMODE=20flag=20?= =?UTF-8?q?=E2=80=94=20capture=20off=20by=20default=20for=20users?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eval capture was on for everyone in the v0.25.0 draft. Privacy footgun: end users had retrieval traffic accumulate in their brain DB without asking, even with PII scrubbing. Flips to off by default + explicit opt-in for contributors who actually use the replay loop. Resolution order in isEvalCaptureEnabled(): 1. config.eval.capture === true → on 2. config.eval.capture === false → off 3. process.env.GBRAIN_CONTRIBUTOR_MODE === '1' → on 4. otherwise → off The env var is the contributor-facing toggle (one line in .zshrc, no JSON edit). Explicit config wins both directions for users who want to override per-brain. PII scrubbing gate stays independent — default true regardless of CONTRIBUTOR_MODE — so any brain that does capture still scrubs. Tests rewritten: env var hygiene per-test (origMode preserved + restored in finally). 9/9 pass; total v0.25.0 suite is 198/198. Docs: - README.md gains a Contributing-section pointer to the env var. - CONTRIBUTING.md gains a "CONTRIBUTOR_MODE — turn on the dev loop" section with verification commands and resolution-order table. - docs/eval-bench.md leads with the prerequisite (must set the env var for the rest of the doc to be useful). - docs/eval-capture.md "Config" section split into Path A (env var) + Path B (config) with explicit resolution-order rules. - CHANGELOG v0.25.0 entry corrected ("on by default" was wrong) plus a new top itemized bullet calling out the gate change. - CLAUDE.md eval-capture entry annotated with the new gate logic. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 9 ++-- CLAUDE.md | 2 +- CONTRIBUTING.md | 50 +++++++++++++++++-- README.md | 2 + docs/eval-bench.md | 41 ++++++++++++++-- docs/eval-capture.md | 28 ++++++++--- llms-full.txt | 4 +- src/core/eval-capture.ts | 29 ++++++++--- test/eval-capture.test.ts | 100 +++++++++++++++++++++++++++++++++----- 9 files changed, 227 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 45abf4325..596b0875b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ All notable changes to GBrain will be documented in this file. v0.20 (gbrain-evals extraction) and v0.21 (Cathedral II) gave gbrain its install surface back and turned code into a first-class graph. The remaining gap was data: `amara-life`, the fictional 418-item corpus over in gbrain-evals, is great for reproducibility but not for catching regressions against the queries your agents *actually* serve. v0.22 ships the substrate: every `query` and `search` call through MCP, CLI, or the subagent tool-bridge gets recorded into a new `eval_candidates` table. `gbrain eval export` streams the rows as NDJSON. Point gbrain-evals at the stream and you have BrainBench-Real on every release. -Capture is on by default. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird. +Capture is **off by default**. Production users get a quiet brain — no surprise data accumulation, no privacy footgun. Contributors flip it on with one shell rc line: `export GBRAIN_CONTRIBUTOR_MODE=1`. From that shell forward, every query/search lands in `eval_candidates`. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird. Cathedral II (v0.21.0) callers are unaffected. `hybridSearch` still returns `Promise` ... meta arrives via an optional `onMeta` callback in `HybridSearchOpts`, used only by the op-layer capture wrapper to record what hybridSearch *actually* did (vector ran or fell back, expansion fired or didn't, post-auto-detect detail). @@ -30,7 +30,9 @@ Measured on this branch's diff against v0.21.0: ### What this means for you -**Brain operators (the 99%):** run `gbrain upgrade`. Capture turns on. Run agents as usual. After a week, `gbrain eval export --since 7d > real.ndjson` has your workload snapshot. Point gbrain-evals at it for regression gating on your next release. To disable: edit `~/.gbrain/config.json` and set `"eval": {"capture": false}`. +**Brain operators (the 99%):** run `gbrain upgrade`. Nothing changes — capture stays off, your brain stays quiet. The substrate is in place if you ever want to turn it on (e.g. to debug a retrieval issue), but you don't have to. + +**Contributors / maintainers:** add `export GBRAIN_CONTRIBUTOR_MODE=1` to your shell rc. Every `query`/`search` from then on writes to `eval_candidates`. After a week of dogfooding, snapshot with `gbrain eval export --since 7d > real.ndjson` and `gbrain eval replay --against real.ndjson` against your branch to gate retrieval changes. To opt out per-brain regardless of the env var: write `{"eval":{"capture":false}}` to `~/.gbrain/config.json`. **Anyone calling `hybridSearch` directly:** no change required. The return type is still `Promise`. If you want the new meta side-channel, pass `onMeta: (m) => { ... }` in opts — otherwise leave it undefined and pay no cost. @@ -63,6 +65,7 @@ Measured on this branch's diff against v0.21.0: ### Itemized changes +- **`GBRAIN_CONTRIBUTOR_MODE=1` env var gates capture.** Default flipped from on-for-everyone to off-unless-opted-in. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Production users get a quiet brain by default; contributors flip the env var in `.zshrc` and unlock the full export → replay loop. README + CONTRIBUTING document the flag prominently. - v31 migration: `eval_candidates` + `eval_capture_failures` on both Postgres + PGLite, RLS gated on BYPASSRLS, CHECK constraints + indexes - Op-layer capture wrapper in `src/core/operations.ts` (covers MCP + CLI + subagent tool-bridge from one site) - PII scrubber in `src/core/eval-capture-scrub.ts` (6 regex families, adversarial-input safe) @@ -74,7 +77,7 @@ Measured on this branch's diff against v0.21.0: - `hybridSearch` adds `onMeta?: (m) => void` to opts (Cathedral II callers unaffected) - `BrainEngine` gains 5 methods (breaking-interface for custom engines, drives v0.25.0 minor bump) - `test/public-exports.test.ts` + `scripts/check-exports-count.sh` lock the 17-subpath public surface -- Config gains `eval: {capture?, scrub_pii?}` (file-plane only) +- Config gains `eval: {capture?, scrub_pii?}` (file-plane only); env var `GBRAIN_CONTRIBUTOR_MODE=1` is the contributor-facing toggle that doesn't require editing JSON - `listEvalCandidates` orders `created_at DESC, id DESC` (deterministic export windows) - `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals - `gbrain doctor` `eval_capture` check now distinguishes pre-v31 missing-table (status: ok / skipped) from RLS-denied SELECT or transient DB error (status: warn) — the diagnostic that surfaces a misconfigured RLS role no longer goes silent diff --git a/CLAUDE.md b/CLAUDE.md index 6511f4575..b3c90f9e8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -54,7 +54,7 @@ strict behavior when unset. - `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count. - `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". -- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. +- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. - `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. - `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 137039f40..750b215a1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -124,12 +124,55 @@ See `docs/ENGINES.md` for the full guide. In short: The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`. +## CONTRIBUTOR_MODE — turn on the dev loop + +gbrain captures retrieval traffic so you can replay real queries against +your code changes before merging. **This is off by default** (production +users get a quiet brain, no surprise data accumulation). Contributors turn +it on with one shell rc line: + +```bash +# In ~/.zshrc or ~/.bashrc: +export GBRAIN_CONTRIBUTOR_MODE=1 +``` + +That's it. Every `query` / `search` you (or agents pointed at your dev +brain) run from that shell now writes a row to `eval_candidates`, and the +[replay tool](#running-real-world-eval-benchmarks-touching-retrieval-code) +has data to work against. + +What CONTRIBUTOR_MODE actually does: + +- Turns on `query`/`search` capture into the local `eval_candidates` table. + Without it the gate is closed and capture is a no-op. +- That's all. PII scrubbing, retention, and replay are independent. + +Resolution order (most explicit wins): + +1. `eval.capture: true` in `~/.gbrain/config.json` → on +2. `eval.capture: false` in `~/.gbrain/config.json` → off +3. `GBRAIN_CONTRIBUTOR_MODE=1` → on +4. otherwise → off + +Quick check that capture is actually running: + +```bash +gbrain query "anything" >/dev/null +psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' +# (or `gbrain doctor` — surfaces silent capture failures cross-process) +``` + +To disable capture even with the env var set, write +`{"eval": {"capture": false}}` to `~/.gbrain/config.json` — explicit config +beats the env var both directions. + ## Running real-world eval benchmarks (touching retrieval code) If your PR touches retrieval — search ranking, RRF fusion, embeddings, intent classification, query expansion, source boost, or the `query` / `search` op handlers — run `gbrain eval replay` against a snapshot of -real traffic before merging. +real traffic before merging. Requires `CONTRIBUTOR_MODE` (above) so you +have captured rows to replay against. Quick loop: @@ -154,8 +197,9 @@ Trigger paths (rerun if your diff touches any of these): searchVector SQL) See [`docs/eval-bench.md`](./docs/eval-bench.md) for the full guide -including CI integration, hand-crafted NDJSON corpora, and the cost -considerations. The NDJSON wire format is documented in +including CI integration, hand-crafted NDJSON corpora (so a fresh checkout +without captured data can still replay), and cost considerations. The +NDJSON wire format is documented in [`docs/eval-capture.md`](./docs/eval-capture.md). ## Welcome PRs diff --git a/README.md b/README.md index d78591517..b13263e48 100644 --- a/README.md +++ b/README.md @@ -746,6 +746,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration. +If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in. + PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`. ## License diff --git a/docs/eval-bench.md b/docs/eval-bench.md index e2ed20c8a..5399e90de 100644 --- a/docs/eval-bench.md +++ b/docs/eval-bench.md @@ -8,10 +8,36 @@ For the **NDJSON wire format** consumed by gbrain-evals, see [`eval-capture.md`](./eval-capture.md). This doc is the human dev loop that lives on top of that format. +## Prerequisite: turn on contributor mode + +Capture is **off by default** for production users (privacy-positive — no +surprise data accumulation). Contributors flip it on with one line: + +```bash +# In ~/.zshrc or ~/.bashrc: +export GBRAIN_CONTRIBUTOR_MODE=1 +``` + +Verify: + +```bash +gbrain query "anything" >/dev/null +psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0 +``` + +To override (force on/off regardless of env var), edit `~/.gbrain/config.json`: + +```json +{"eval": {"capture": true}} // force on +{"eval": {"capture": false}} // force off +``` + +Explicit config beats the env var both directions. + ## The 4-command loop ```bash -# ① Capture: already happening on every MCP / CLI / subagent query, capture is on by default. +# ① Capture: writes to eval_candidates whenever CONTRIBUTOR_MODE is set. # Inspect what's been collected: gbrain doctor # surfaces capture failures psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' @@ -172,15 +198,20 @@ pipeline (`gbrain eval --qrels` with the sibling ## Off-switch -If you don't want capture at all (privacy concern, dev environment), edit -`~/.gbrain/config.json`: +Two ways to disable capture: + +```bash +unset GBRAIN_CONTRIBUTOR_MODE # easy: just unset the env var +``` + +Or force off regardless of the env var via `~/.gbrain/config.json`: ```json {"eval": {"capture": false}} ``` -Capture stops; existing `eval_candidates` rows stay until you `gbrain eval -prune --older-than 0d` (or just drop the table). +Existing `eval_candidates` rows stay until you `gbrain eval prune +--older-than 0d` (or just drop the table). ## Failure modes diff --git a/docs/eval-capture.md b/docs/eval-capture.md index 162371727..2e671d9d2 100644 --- a/docs/eval-capture.md +++ b/docs/eval-capture.md @@ -120,9 +120,18 @@ Reason enum (stable): `db_down` | `rls_reject` | `check_violation` | point — `gbrain doctor` runs in its own process and reads the table directly, so in-process counters wouldn't work. -## Config (file-plane only) +## Config + CONTRIBUTOR_MODE -`~/.gbrain/config.json`: +Capture is **off by default** as of v0.25.0 (was on for everyone in +earlier drafts). Two paths to turn it on: + +**Path A — env var (contributor opt-in, the common case):** + +```bash +export GBRAIN_CONTRIBUTOR_MODE=1 # in ~/.zshrc or ~/.bashrc +``` + +**Path B — explicit config (`~/.gbrain/config.json`, file-plane only):** ```json { @@ -135,10 +144,17 @@ directly, so in-process counters wouldn't work. } ``` -Both default to `true`. Set `eval.capture: false` to disable capture -entirely; set `eval.scrub_pii: false` to preserve raw query text -(only if you control the brain's distribution). +Resolution order (most explicit wins): + +1. `eval.capture: true` in config → on +2. `eval.capture: false` in config → off (overrides CONTRIBUTOR_MODE=1) +3. `GBRAIN_CONTRIBUTOR_MODE === '1'` → on +4. otherwise → off + +`scrub_pii` defaults to `true` independent of capture. Set +`eval.scrub_pii: false` to preserve raw query text (only if you control +the brain's distribution). `gbrain config set eval.capture false` does **not** work — that command writes the DB-plane config, and the MCP server reads the -file-plane. Edit the JSON directly. +file-plane. Edit the JSON directly or use the env var. diff --git a/llms-full.txt b/llms-full.txt index bf86def4f..84a97298d 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -140,7 +140,7 @@ strict behavior when unset. - `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count. - `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". -- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. +- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. - `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. - `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. @@ -2115,6 +2115,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration. +If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in. + PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`. ## License diff --git a/src/core/eval-capture.ts b/src/core/eval-capture.ts index e1f56e6e6..ff8807672 100644 --- a/src/core/eval-capture.ts +++ b/src/core/eval-capture.ts @@ -173,19 +173,36 @@ export async function captureEvalCandidate( /** * Check whether capture is enabled for this process. * - * File-plane only: reads `config.eval.capture` from the loaded GBrainConfig. - * `gbrain config set` writes the DB plane and does NOT affect this flag. - * Capture is on by default; returns false only when config explicitly sets - * `eval.capture: false`. + * Resolution order: + * 1. `config.eval.capture === true` → on (explicit user opt-in wins) + * 2. `config.eval.capture === false` → off (explicit user opt-out wins) + * 3. `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'` → on (contributor opt-in) + * 4. otherwise → off (default-off, privacy-positive for end users) + * + * The default flipped in v0.25.0 from "on for everyone" to "off unless you + * opt in." Capturing every query a real user runs without their consent is + * a footgun even with PII scrubbing; tying capture to CONTRIBUTOR_MODE makes + * the developer-skill nature of the feature explicit. Production users get + * a quiet brain; contributors get the BrainBench-Real replay loop with one + * shell rc line. See docs/eval-bench.md and CONTRIBUTING.md. * * Takes the already-loaded config so callers control the loadConfig() * lifecycle (MCP server loads once at boot, CLI commands load per-invocation). */ export function isEvalCaptureEnabled(config: GBrainConfig | null | undefined): boolean { - return config?.eval?.capture !== false; + if (config?.eval?.capture === true) return true; + if (config?.eval?.capture === false) return false; + return process.env.GBRAIN_CONTRIBUTOR_MODE === '1'; } -/** PII scrubbing enabled? Defaults to true; explicit `false` opts out. */ +/** + * PII scrubbing enabled? Defaults to true; explicit `false` opts out. + * + * Independent of `isEvalCaptureEnabled` — scrubbing only matters when capture + * is actually running, but the gate stays separate so CONTRIBUTOR_MODE + * doesn't accidentally turn off scrubbing on a brain that happens to also + * have explicit `capture: true`. + */ export function isEvalScrubEnabled(config: GBrainConfig | null | undefined): boolean { return config?.eval?.scrub_pii !== false; } diff --git a/test/eval-capture.test.ts b/test/eval-capture.test.ts index 686d9b9c5..ad60bcea9 100644 --- a/test/eval-capture.test.ts +++ b/test/eval-capture.test.ts @@ -214,24 +214,98 @@ describe('captureEvalCandidate — best-effort failure handling', () => { }); }); -describe('isEvalCaptureEnabled / isEvalScrubEnabled', () => { - test('defaults to enabled when config is null or missing', () => { - expect(isEvalCaptureEnabled(null)).toBe(true); - expect(isEvalCaptureEnabled(undefined)).toBe(true); - expect(isEvalScrubEnabled(null)).toBe(true); +describe('isEvalCaptureEnabled / isEvalScrubEnabled (CONTRIBUTOR_MODE-gated)', () => { + // v0.25.0 flipped the default: was on for everyone, now off unless either + // the env var or an explicit config flag is set. Tests scope env mutation + // so they don't leak across describe blocks. + const origMode = process.env.GBRAIN_CONTRIBUTOR_MODE; + const restore = () => { + if (origMode === undefined) delete process.env.GBRAIN_CONTRIBUTOR_MODE; + else process.env.GBRAIN_CONTRIBUTOR_MODE = origMode; + }; + + test('defaults to OFF when config is missing AND CONTRIBUTOR_MODE unset', () => { + delete process.env.GBRAIN_CONTRIBUTOR_MODE; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + expect(isEvalCaptureEnabled(undefined)).toBe(false); + } finally { restore(); } }); - test('returns false only when explicitly set to false', () => { + test('CONTRIBUTOR_MODE=1 turns capture on without any config', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + expect(isEvalCaptureEnabled(null)).toBe(true); + expect(isEvalCaptureEnabled(undefined)).toBe(true); + } finally { restore(); } + }); + + test('CONTRIBUTOR_MODE=anything-else does NOT turn capture on (strict equality)', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = 'true'; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + } finally { restore(); } + process.env.GBRAIN_CONTRIBUTOR_MODE = 'yes'; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + } finally { restore(); } + process.env.GBRAIN_CONTRIBUTOR_MODE = ''; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + } finally { restore(); } + }); + + test('explicit config eval.capture=true wins over absent CONTRIBUTOR_MODE', () => { + delete process.env.GBRAIN_CONTRIBUTOR_MODE; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const enabled: any = { engine: 'pglite', eval: { capture: true } }; + expect(isEvalCaptureEnabled(enabled)).toBe(true); + } finally { restore(); } + }); + + test('explicit config eval.capture=false wins over CONTRIBUTOR_MODE=1', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const disabled: any = { engine: 'pglite', eval: { capture: false } }; + expect(isEvalCaptureEnabled(disabled)).toBe(false); + } finally { restore(); } + }); + + test('config eval present but capture key undefined falls through to env check', () => { + delete process.env.GBRAIN_CONTRIBUTOR_MODE; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const partial: any = { engine: 'pglite', eval: { scrub_pii: false } }; + expect(isEvalCaptureEnabled(partial)).toBe(false); + } finally { restore(); } + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const partial: any = { engine: 'pglite', eval: { scrub_pii: false } }; + expect(isEvalCaptureEnabled(partial)).toBe(true); + } finally { restore(); } + }); + + test('isEvalScrubEnabled: defaults true, only false when explicitly disabled', () => { + expect(isEvalScrubEnabled(null)).toBe(true); + expect(isEvalScrubEnabled(undefined)).toBe(true); // eslint-disable-next-line @typescript-eslint/no-explicit-any - const disabled: any = { engine: 'pglite', eval: { capture: false, scrub_pii: false } }; - expect(isEvalCaptureEnabled(disabled)).toBe(false); + const partial: any = { engine: 'pglite', eval: {} }; + expect(isEvalScrubEnabled(partial)).toBe(true); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const disabled: any = { engine: 'pglite', eval: { scrub_pii: false } }; expect(isEvalScrubEnabled(disabled)).toBe(false); }); - test('defaults to enabled when the eval key exists but the flag is undefined', () => { - // eslint-disable-next-line @typescript-eslint/no-explicit-any - const partial: any = { engine: 'pglite', eval: {} }; - expect(isEvalCaptureEnabled(partial)).toBe(true); - expect(isEvalScrubEnabled(partial)).toBe(true); + test('isEvalScrubEnabled is independent of CONTRIBUTOR_MODE', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + expect(isEvalScrubEnabled(null)).toBe(true); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const opt: any = { engine: 'pglite', eval: { scrub_pii: false } }; + expect(isEvalScrubEnabled(opt)).toBe(false); + } finally { restore(); } }); });