mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
feat(v0.25.0): CONTRIBUTOR_MODE flag — capture off by default for users
Eval capture was on for everyone in the v0.25.0 draft. Privacy footgun:
end users had retrieval traffic accumulate in their brain DB without
asking, even with PII scrubbing. Flips to off by default + explicit
opt-in for contributors who actually use the replay loop.
Resolution order in isEvalCaptureEnabled():
1. config.eval.capture === true → on
2. config.eval.capture === false → off
3. process.env.GBRAIN_CONTRIBUTOR_MODE === '1' → on
4. otherwise → off
The env var is the contributor-facing toggle (one line in .zshrc, no
JSON edit). Explicit config wins both directions for users who want to
override per-brain.
PII scrubbing gate stays independent — default true regardless of
CONTRIBUTOR_MODE — so any brain that does capture still scrubs.
Tests rewritten: env var hygiene per-test (origMode preserved + restored
in finally). 9/9 pass; total v0.25.0 suite is 198/198.
Docs:
- README.md gains a Contributing-section pointer to the env var.
- CONTRIBUTING.md gains a "CONTRIBUTOR_MODE — turn on the dev loop"
section with verification commands and resolution-order table.
- docs/eval-bench.md leads with the prerequisite (must set the env var
for the rest of the doc to be useful).
- docs/eval-capture.md "Config" section split into Path A (env var) +
Path B (config) with explicit resolution-order rules.
- CHANGELOG v0.25.0 entry corrected ("on by default" was wrong) plus a
new top itemized bullet calling out the gate change.
- CLAUDE.md eval-capture entry annotated with the new gate logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
a5c375dd15
commit
7a80ce25c2
+6
-3
@@ -9,7 +9,7 @@ All notable changes to GBrain will be documented in this file.
|
||||
|
||||
v0.20 (gbrain-evals extraction) and v0.21 (Cathedral II) gave gbrain its install surface back and turned code into a first-class graph. The remaining gap was data: `amara-life`, the fictional 418-item corpus over in gbrain-evals, is great for reproducibility but not for catching regressions against the queries your agents *actually* serve. v0.22 ships the substrate: every `query` and `search` call through MCP, CLI, or the subagent tool-bridge gets recorded into a new `eval_candidates` table. `gbrain eval export` streams the rows as NDJSON. Point gbrain-evals at the stream and you have BrainBench-Real on every release.
|
||||
|
||||
Capture is on by default. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird.
|
||||
Capture is **off by default**. Production users get a quiet brain — no surprise data accumulation, no privacy footgun. Contributors flip it on with one shell rc line: `export GBRAIN_CONTRIBUTOR_MODE=1`. From that shell forward, every query/search lands in `eval_candidates`. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird.
|
||||
|
||||
Cathedral II (v0.21.0) callers are unaffected. `hybridSearch` still returns `Promise<SearchResult[]>` ... meta arrives via an optional `onMeta` callback in `HybridSearchOpts`, used only by the op-layer capture wrapper to record what hybridSearch *actually* did (vector ran or fell back, expansion fired or didn't, post-auto-detect detail).
|
||||
|
||||
@@ -30,7 +30,9 @@ Measured on this branch's diff against v0.21.0:
|
||||
|
||||
### What this means for you
|
||||
|
||||
**Brain operators (the 99%):** run `gbrain upgrade`. Capture turns on. Run agents as usual. After a week, `gbrain eval export --since 7d > real.ndjson` has your workload snapshot. Point gbrain-evals at it for regression gating on your next release. To disable: edit `~/.gbrain/config.json` and set `"eval": {"capture": false}`.
|
||||
**Brain operators (the 99%):** run `gbrain upgrade`. Nothing changes — capture stays off, your brain stays quiet. The substrate is in place if you ever want to turn it on (e.g. to debug a retrieval issue), but you don't have to.
|
||||
|
||||
**Contributors / maintainers:** add `export GBRAIN_CONTRIBUTOR_MODE=1` to your shell rc. Every `query`/`search` from then on writes to `eval_candidates`. After a week of dogfooding, snapshot with `gbrain eval export --since 7d > real.ndjson` and `gbrain eval replay --against real.ndjson` against your branch to gate retrieval changes. To opt out per-brain regardless of the env var: write `{"eval":{"capture":false}}` to `~/.gbrain/config.json`.
|
||||
|
||||
**Anyone calling `hybridSearch` directly:** no change required. The return type is still `Promise<SearchResult[]>`. If you want the new meta side-channel, pass `onMeta: (m) => { ... }` in opts — otherwise leave it undefined and pay no cost.
|
||||
|
||||
@@ -63,6 +65,7 @@ Measured on this branch's diff against v0.21.0:
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **`GBRAIN_CONTRIBUTOR_MODE=1` env var gates capture.** Default flipped from on-for-everyone to off-unless-opted-in. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Production users get a quiet brain by default; contributors flip the env var in `.zshrc` and unlock the full export → replay loop. README + CONTRIBUTING document the flag prominently.
|
||||
- v31 migration: `eval_candidates` + `eval_capture_failures` on both Postgres + PGLite, RLS gated on BYPASSRLS, CHECK constraints + indexes
|
||||
- Op-layer capture wrapper in `src/core/operations.ts` (covers MCP + CLI + subagent tool-bridge from one site)
|
||||
- PII scrubber in `src/core/eval-capture-scrub.ts` (6 regex families, adversarial-input safe)
|
||||
@@ -74,7 +77,7 @@ Measured on this branch's diff against v0.21.0:
|
||||
- `hybridSearch` adds `onMeta?: (m) => void` to opts (Cathedral II callers unaffected)
|
||||
- `BrainEngine` gains 5 methods (breaking-interface for custom engines, drives v0.25.0 minor bump)
|
||||
- `test/public-exports.test.ts` + `scripts/check-exports-count.sh` lock the 17-subpath public surface
|
||||
- Config gains `eval: {capture?, scrub_pii?}` (file-plane only)
|
||||
- Config gains `eval: {capture?, scrub_pii?}` (file-plane only); env var `GBRAIN_CONTRIBUTOR_MODE=1` is the contributor-facing toggle that doesn't require editing JSON
|
||||
- `listEvalCandidates` orders `created_at DESC, id DESC` (deterministic export windows)
|
||||
- `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals
|
||||
- `gbrain doctor` `eval_capture` check now distinguishes pre-v31 missing-table (status: ok / skipped) from RLS-denied SELECT or transient DB error (status: warn) — the diagnostic that surfaces a misconfigured RLS role no longer goes silent
|
||||
|
||||
@@ -54,7 +54,7 @@ strict behavior when unset.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process.
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||
|
||||
+47
-3
@@ -124,12 +124,55 @@ See `docs/ENGINES.md` for the full guide. In short:
|
||||
|
||||
The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`.
|
||||
|
||||
## CONTRIBUTOR_MODE — turn on the dev loop
|
||||
|
||||
gbrain captures retrieval traffic so you can replay real queries against
|
||||
your code changes before merging. **This is off by default** (production
|
||||
users get a quiet brain, no surprise data accumulation). Contributors turn
|
||||
it on with one shell rc line:
|
||||
|
||||
```bash
|
||||
# In ~/.zshrc or ~/.bashrc:
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||
```
|
||||
|
||||
That's it. Every `query` / `search` you (or agents pointed at your dev
|
||||
brain) run from that shell now writes a row to `eval_candidates`, and the
|
||||
[replay tool](#running-real-world-eval-benchmarks-touching-retrieval-code)
|
||||
has data to work against.
|
||||
|
||||
What CONTRIBUTOR_MODE actually does:
|
||||
|
||||
- Turns on `query`/`search` capture into the local `eval_candidates` table.
|
||||
Without it the gate is closed and capture is a no-op.
|
||||
- That's all. PII scrubbing, retention, and replay are independent.
|
||||
|
||||
Resolution order (most explicit wins):
|
||||
|
||||
1. `eval.capture: true` in `~/.gbrain/config.json` → on
|
||||
2. `eval.capture: false` in `~/.gbrain/config.json` → off
|
||||
3. `GBRAIN_CONTRIBUTOR_MODE=1` → on
|
||||
4. otherwise → off
|
||||
|
||||
Quick check that capture is actually running:
|
||||
|
||||
```bash
|
||||
gbrain query "anything" >/dev/null
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
|
||||
# (or `gbrain doctor` — surfaces silent capture failures cross-process)
|
||||
```
|
||||
|
||||
To disable capture even with the env var set, write
|
||||
`{"eval": {"capture": false}}` to `~/.gbrain/config.json` — explicit config
|
||||
beats the env var both directions.
|
||||
|
||||
## Running real-world eval benchmarks (touching retrieval code)
|
||||
|
||||
If your PR touches retrieval — search ranking, RRF fusion, embeddings,
|
||||
intent classification, query expansion, source boost, or the `query` /
|
||||
`search` op handlers — run `gbrain eval replay` against a snapshot of
|
||||
real traffic before merging.
|
||||
real traffic before merging. Requires `CONTRIBUTOR_MODE` (above) so you
|
||||
have captured rows to replay against.
|
||||
|
||||
Quick loop:
|
||||
|
||||
@@ -154,8 +197,9 @@ Trigger paths (rerun if your diff touches any of these):
|
||||
searchVector SQL)
|
||||
|
||||
See [`docs/eval-bench.md`](./docs/eval-bench.md) for the full guide
|
||||
including CI integration, hand-crafted NDJSON corpora, and the cost
|
||||
considerations. The NDJSON wire format is documented in
|
||||
including CI integration, hand-crafted NDJSON corpora (so a fresh checkout
|
||||
without captured data can still replay), and cost considerations. The
|
||||
NDJSON wire format is documented in
|
||||
[`docs/eval-capture.md`](./docs/eval-capture.md).
|
||||
|
||||
## Welcome PRs
|
||||
|
||||
@@ -746,6 +746,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
||||
|
||||
If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in.
|
||||
|
||||
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
||||
|
||||
## License
|
||||
|
||||
+36
-5
@@ -8,10 +8,36 @@ For the **NDJSON wire format** consumed by gbrain-evals, see
|
||||
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
|
||||
that lives on top of that format.
|
||||
|
||||
## Prerequisite: turn on contributor mode
|
||||
|
||||
Capture is **off by default** for production users (privacy-positive — no
|
||||
surprise data accumulation). Contributors flip it on with one line:
|
||||
|
||||
```bash
|
||||
# In ~/.zshrc or ~/.bashrc:
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
gbrain query "anything" >/dev/null
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
|
||||
```
|
||||
|
||||
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
|
||||
|
||||
```json
|
||||
{"eval": {"capture": true}} // force on
|
||||
{"eval": {"capture": false}} // force off
|
||||
```
|
||||
|
||||
Explicit config beats the env var both directions.
|
||||
|
||||
## The 4-command loop
|
||||
|
||||
```bash
|
||||
# ① Capture: already happening on every MCP / CLI / subagent query, capture is on by default.
|
||||
# ① Capture: writes to eval_candidates whenever CONTRIBUTOR_MODE is set.
|
||||
# Inspect what's been collected:
|
||||
gbrain doctor # surfaces capture failures
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
|
||||
@@ -172,15 +198,20 @@ pipeline (`gbrain eval --qrels` with the sibling
|
||||
|
||||
## Off-switch
|
||||
|
||||
If you don't want capture at all (privacy concern, dev environment), edit
|
||||
`~/.gbrain/config.json`:
|
||||
Two ways to disable capture:
|
||||
|
||||
```bash
|
||||
unset GBRAIN_CONTRIBUTOR_MODE # easy: just unset the env var
|
||||
```
|
||||
|
||||
Or force off regardless of the env var via `~/.gbrain/config.json`:
|
||||
|
||||
```json
|
||||
{"eval": {"capture": false}}
|
||||
```
|
||||
|
||||
Capture stops; existing `eval_candidates` rows stay until you `gbrain eval
|
||||
prune --older-than 0d` (or just drop the table).
|
||||
Existing `eval_candidates` rows stay until you `gbrain eval prune
|
||||
--older-than 0d` (or just drop the table).
|
||||
|
||||
## Failure modes
|
||||
|
||||
|
||||
+22
-6
@@ -120,9 +120,18 @@ Reason enum (stable): `db_down` | `rls_reject` | `check_violation` |
|
||||
point — `gbrain doctor` runs in its own process and reads the table
|
||||
directly, so in-process counters wouldn't work.
|
||||
|
||||
## Config (file-plane only)
|
||||
## Config + CONTRIBUTOR_MODE
|
||||
|
||||
`~/.gbrain/config.json`:
|
||||
Capture is **off by default** as of v0.25.0 (was on for everyone in
|
||||
earlier drafts). Two paths to turn it on:
|
||||
|
||||
**Path A — env var (contributor opt-in, the common case):**
|
||||
|
||||
```bash
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1 # in ~/.zshrc or ~/.bashrc
|
||||
```
|
||||
|
||||
**Path B — explicit config (`~/.gbrain/config.json`, file-plane only):**
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -135,10 +144,17 @@ directly, so in-process counters wouldn't work.
|
||||
}
|
||||
```
|
||||
|
||||
Both default to `true`. Set `eval.capture: false` to disable capture
|
||||
entirely; set `eval.scrub_pii: false` to preserve raw query text
|
||||
(only if you control the brain's distribution).
|
||||
Resolution order (most explicit wins):
|
||||
|
||||
1. `eval.capture: true` in config → on
|
||||
2. `eval.capture: false` in config → off (overrides CONTRIBUTOR_MODE=1)
|
||||
3. `GBRAIN_CONTRIBUTOR_MODE === '1'` → on
|
||||
4. otherwise → off
|
||||
|
||||
`scrub_pii` defaults to `true` independent of capture. Set
|
||||
`eval.scrub_pii: false` to preserve raw query text (only if you control
|
||||
the brain's distribution).
|
||||
|
||||
`gbrain config set eval.capture false` does **not** work — that
|
||||
command writes the DB-plane config, and the MCP server reads the
|
||||
file-plane. Edit the JSON directly.
|
||||
file-plane. Edit the JSON directly or use the env var.
|
||||
|
||||
+3
-1
@@ -140,7 +140,7 @@ strict behavior when unset.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process.
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||
@@ -2115,6 +2115,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
||||
|
||||
If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in.
|
||||
|
||||
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
||||
|
||||
## License
|
||||
|
||||
@@ -173,19 +173,36 @@ export async function captureEvalCandidate(
|
||||
/**
|
||||
* Check whether capture is enabled for this process.
|
||||
*
|
||||
* File-plane only: reads `config.eval.capture` from the loaded GBrainConfig.
|
||||
* `gbrain config set` writes the DB plane and does NOT affect this flag.
|
||||
* Capture is on by default; returns false only when config explicitly sets
|
||||
* `eval.capture: false`.
|
||||
* Resolution order:
|
||||
* 1. `config.eval.capture === true` → on (explicit user opt-in wins)
|
||||
* 2. `config.eval.capture === false` → off (explicit user opt-out wins)
|
||||
* 3. `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'` → on (contributor opt-in)
|
||||
* 4. otherwise → off (default-off, privacy-positive for end users)
|
||||
*
|
||||
* The default flipped in v0.25.0 from "on for everyone" to "off unless you
|
||||
* opt in." Capturing every query a real user runs without their consent is
|
||||
* a footgun even with PII scrubbing; tying capture to CONTRIBUTOR_MODE makes
|
||||
* the developer-skill nature of the feature explicit. Production users get
|
||||
* a quiet brain; contributors get the BrainBench-Real replay loop with one
|
||||
* shell rc line. See docs/eval-bench.md and CONTRIBUTING.md.
|
||||
*
|
||||
* Takes the already-loaded config so callers control the loadConfig()
|
||||
* lifecycle (MCP server loads once at boot, CLI commands load per-invocation).
|
||||
*/
|
||||
export function isEvalCaptureEnabled(config: GBrainConfig | null | undefined): boolean {
|
||||
return config?.eval?.capture !== false;
|
||||
if (config?.eval?.capture === true) return true;
|
||||
if (config?.eval?.capture === false) return false;
|
||||
return process.env.GBRAIN_CONTRIBUTOR_MODE === '1';
|
||||
}
|
||||
|
||||
/** PII scrubbing enabled? Defaults to true; explicit `false` opts out. */
|
||||
/**
|
||||
* PII scrubbing enabled? Defaults to true; explicit `false` opts out.
|
||||
*
|
||||
* Independent of `isEvalCaptureEnabled` — scrubbing only matters when capture
|
||||
* is actually running, but the gate stays separate so CONTRIBUTOR_MODE
|
||||
* doesn't accidentally turn off scrubbing on a brain that happens to also
|
||||
* have explicit `capture: true`.
|
||||
*/
|
||||
export function isEvalScrubEnabled(config: GBrainConfig | null | undefined): boolean {
|
||||
return config?.eval?.scrub_pii !== false;
|
||||
}
|
||||
|
||||
+87
-13
@@ -214,24 +214,98 @@ describe('captureEvalCandidate — best-effort failure handling', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('isEvalCaptureEnabled / isEvalScrubEnabled', () => {
|
||||
test('defaults to enabled when config is null or missing', () => {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(true);
|
||||
expect(isEvalCaptureEnabled(undefined)).toBe(true);
|
||||
expect(isEvalScrubEnabled(null)).toBe(true);
|
||||
describe('isEvalCaptureEnabled / isEvalScrubEnabled (CONTRIBUTOR_MODE-gated)', () => {
|
||||
// v0.25.0 flipped the default: was on for everyone, now off unless either
|
||||
// the env var or an explicit config flag is set. Tests scope env mutation
|
||||
// so they don't leak across describe blocks.
|
||||
const origMode = process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
const restore = () => {
|
||||
if (origMode === undefined) delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
else process.env.GBRAIN_CONTRIBUTOR_MODE = origMode;
|
||||
};
|
||||
|
||||
test('defaults to OFF when config is missing AND CONTRIBUTOR_MODE unset', () => {
|
||||
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
expect(isEvalCaptureEnabled(undefined)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('returns false only when explicitly set to false', () => {
|
||||
test('CONTRIBUTOR_MODE=1 turns capture on without any config', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(true);
|
||||
expect(isEvalCaptureEnabled(undefined)).toBe(true);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('CONTRIBUTOR_MODE=anything-else does NOT turn capture on (strict equality)', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = 'true';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
} finally { restore(); }
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = 'yes';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
} finally { restore(); }
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('explicit config eval.capture=true wins over absent CONTRIBUTOR_MODE', () => {
|
||||
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const enabled: any = { engine: 'pglite', eval: { capture: true } };
|
||||
expect(isEvalCaptureEnabled(enabled)).toBe(true);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('explicit config eval.capture=false wins over CONTRIBUTOR_MODE=1', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const disabled: any = { engine: 'pglite', eval: { capture: false } };
|
||||
expect(isEvalCaptureEnabled(disabled)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('config eval present but capture key undefined falls through to env check', () => {
|
||||
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const partial: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalCaptureEnabled(partial)).toBe(false);
|
||||
} finally { restore(); }
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const partial: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalCaptureEnabled(partial)).toBe(true);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('isEvalScrubEnabled: defaults true, only false when explicitly disabled', () => {
|
||||
expect(isEvalScrubEnabled(null)).toBe(true);
|
||||
expect(isEvalScrubEnabled(undefined)).toBe(true);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const disabled: any = { engine: 'pglite', eval: { capture: false, scrub_pii: false } };
|
||||
expect(isEvalCaptureEnabled(disabled)).toBe(false);
|
||||
const partial: any = { engine: 'pglite', eval: {} };
|
||||
expect(isEvalScrubEnabled(partial)).toBe(true);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const disabled: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalScrubEnabled(disabled)).toBe(false);
|
||||
});
|
||||
|
||||
test('defaults to enabled when the eval key exists but the flag is undefined', () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const partial: any = { engine: 'pglite', eval: {} };
|
||||
expect(isEvalCaptureEnabled(partial)).toBe(true);
|
||||
expect(isEvalScrubEnabled(partial)).toBe(true);
|
||||
test('isEvalScrubEnabled is independent of CONTRIBUTOR_MODE', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
expect(isEvalScrubEnabled(null)).toBe(true);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const opt: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalScrubEnabled(opt)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user