mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
feat(v0.25.0): CONTRIBUTOR_MODE flag — capture off by default for users
Eval capture was on for everyone in the v0.25.0 draft. Privacy footgun:
end users had retrieval traffic accumulate in their brain DB without
asking, even with PII scrubbing. Flips to off by default + explicit
opt-in for contributors who actually use the replay loop.
Resolution order in isEvalCaptureEnabled():
1. config.eval.capture === true → on
2. config.eval.capture === false → off
3. process.env.GBRAIN_CONTRIBUTOR_MODE === '1' → on
4. otherwise → off
The env var is the contributor-facing toggle (one line in .zshrc, no
JSON edit). Explicit config wins both directions for users who want to
override per-brain.
PII scrubbing gate stays independent — default true regardless of
CONTRIBUTOR_MODE — so any brain that does capture still scrubs.
Tests rewritten: env var hygiene per-test (origMode preserved + restored
in finally). 9/9 pass; total v0.25.0 suite is 198/198.
Docs:
- README.md gains a Contributing-section pointer to the env var.
- CONTRIBUTING.md gains a "CONTRIBUTOR_MODE — turn on the dev loop"
section with verification commands and resolution-order table.
- docs/eval-bench.md leads with the prerequisite (must set the env var
for the rest of the doc to be useful).
- docs/eval-capture.md "Config" section split into Path A (env var) +
Path B (config) with explicit resolution-order rules.
- CHANGELOG v0.25.0 entry corrected ("on by default" was wrong) plus a
new top itemized bullet calling out the gate change.
- CLAUDE.md eval-capture entry annotated with the new gate logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
a5c375dd15
commit
7a80ce25c2
+6
-3
@@ -9,7 +9,7 @@ All notable changes to GBrain will be documented in this file.
|
|||||||
|
|
||||||
v0.20 (gbrain-evals extraction) and v0.21 (Cathedral II) gave gbrain its install surface back and turned code into a first-class graph. The remaining gap was data: `amara-life`, the fictional 418-item corpus over in gbrain-evals, is great for reproducibility but not for catching regressions against the queries your agents *actually* serve. v0.22 ships the substrate: every `query` and `search` call through MCP, CLI, or the subagent tool-bridge gets recorded into a new `eval_candidates` table. `gbrain eval export` streams the rows as NDJSON. Point gbrain-evals at the stream and you have BrainBench-Real on every release.
|
v0.20 (gbrain-evals extraction) and v0.21 (Cathedral II) gave gbrain its install surface back and turned code into a first-class graph. The remaining gap was data: `amara-life`, the fictional 418-item corpus over in gbrain-evals, is great for reproducibility but not for catching regressions against the queries your agents *actually* serve. v0.22 ships the substrate: every `query` and `search` call through MCP, CLI, or the subagent tool-bridge gets recorded into a new `eval_candidates` table. `gbrain eval export` streams the rows as NDJSON. Point gbrain-evals at the stream and you have BrainBench-Real on every release.
|
||||||
|
|
||||||
Capture is on by default. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird.
|
Capture is **off by default**. Production users get a quiet brain — no surprise data accumulation, no privacy footgun. Contributors flip it on with one shell rc line: `export GBRAIN_CONTRIBUTOR_MODE=1`. From that shell forward, every query/search lands in `eval_candidates`. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird.
|
||||||
|
|
||||||
Cathedral II (v0.21.0) callers are unaffected. `hybridSearch` still returns `Promise<SearchResult[]>` ... meta arrives via an optional `onMeta` callback in `HybridSearchOpts`, used only by the op-layer capture wrapper to record what hybridSearch *actually* did (vector ran or fell back, expansion fired or didn't, post-auto-detect detail).
|
Cathedral II (v0.21.0) callers are unaffected. `hybridSearch` still returns `Promise<SearchResult[]>` ... meta arrives via an optional `onMeta` callback in `HybridSearchOpts`, used only by the op-layer capture wrapper to record what hybridSearch *actually* did (vector ran or fell back, expansion fired or didn't, post-auto-detect detail).
|
||||||
|
|
||||||
@@ -30,7 +30,9 @@ Measured on this branch's diff against v0.21.0:
|
|||||||
|
|
||||||
### What this means for you
|
### What this means for you
|
||||||
|
|
||||||
**Brain operators (the 99%):** run `gbrain upgrade`. Capture turns on. Run agents as usual. After a week, `gbrain eval export --since 7d > real.ndjson` has your workload snapshot. Point gbrain-evals at it for regression gating on your next release. To disable: edit `~/.gbrain/config.json` and set `"eval": {"capture": false}`.
|
**Brain operators (the 99%):** run `gbrain upgrade`. Nothing changes — capture stays off, your brain stays quiet. The substrate is in place if you ever want to turn it on (e.g. to debug a retrieval issue), but you don't have to.
|
||||||
|
|
||||||
|
**Contributors / maintainers:** add `export GBRAIN_CONTRIBUTOR_MODE=1` to your shell rc. Every `query`/`search` from then on writes to `eval_candidates`. After a week of dogfooding, snapshot with `gbrain eval export --since 7d > real.ndjson` and `gbrain eval replay --against real.ndjson` against your branch to gate retrieval changes. To opt out per-brain regardless of the env var: write `{"eval":{"capture":false}}` to `~/.gbrain/config.json`.
|
||||||
|
|
||||||
**Anyone calling `hybridSearch` directly:** no change required. The return type is still `Promise<SearchResult[]>`. If you want the new meta side-channel, pass `onMeta: (m) => { ... }` in opts — otherwise leave it undefined and pay no cost.
|
**Anyone calling `hybridSearch` directly:** no change required. The return type is still `Promise<SearchResult[]>`. If you want the new meta side-channel, pass `onMeta: (m) => { ... }` in opts — otherwise leave it undefined and pay no cost.
|
||||||
|
|
||||||
@@ -63,6 +65,7 @@ Measured on this branch's diff against v0.21.0:
|
|||||||
|
|
||||||
### Itemized changes
|
### Itemized changes
|
||||||
|
|
||||||
|
- **`GBRAIN_CONTRIBUTOR_MODE=1` env var gates capture.** Default flipped from on-for-everyone to off-unless-opted-in. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Production users get a quiet brain by default; contributors flip the env var in `.zshrc` and unlock the full export → replay loop. README + CONTRIBUTING document the flag prominently.
|
||||||
- v31 migration: `eval_candidates` + `eval_capture_failures` on both Postgres + PGLite, RLS gated on BYPASSRLS, CHECK constraints + indexes
|
- v31 migration: `eval_candidates` + `eval_capture_failures` on both Postgres + PGLite, RLS gated on BYPASSRLS, CHECK constraints + indexes
|
||||||
- Op-layer capture wrapper in `src/core/operations.ts` (covers MCP + CLI + subagent tool-bridge from one site)
|
- Op-layer capture wrapper in `src/core/operations.ts` (covers MCP + CLI + subagent tool-bridge from one site)
|
||||||
- PII scrubber in `src/core/eval-capture-scrub.ts` (6 regex families, adversarial-input safe)
|
- PII scrubber in `src/core/eval-capture-scrub.ts` (6 regex families, adversarial-input safe)
|
||||||
@@ -74,7 +77,7 @@ Measured on this branch's diff against v0.21.0:
|
|||||||
- `hybridSearch` adds `onMeta?: (m) => void` to opts (Cathedral II callers unaffected)
|
- `hybridSearch` adds `onMeta?: (m) => void` to opts (Cathedral II callers unaffected)
|
||||||
- `BrainEngine` gains 5 methods (breaking-interface for custom engines, drives v0.25.0 minor bump)
|
- `BrainEngine` gains 5 methods (breaking-interface for custom engines, drives v0.25.0 minor bump)
|
||||||
- `test/public-exports.test.ts` + `scripts/check-exports-count.sh` lock the 17-subpath public surface
|
- `test/public-exports.test.ts` + `scripts/check-exports-count.sh` lock the 17-subpath public surface
|
||||||
- Config gains `eval: {capture?, scrub_pii?}` (file-plane only)
|
- Config gains `eval: {capture?, scrub_pii?}` (file-plane only); env var `GBRAIN_CONTRIBUTOR_MODE=1` is the contributor-facing toggle that doesn't require editing JSON
|
||||||
- `listEvalCandidates` orders `created_at DESC, id DESC` (deterministic export windows)
|
- `listEvalCandidates` orders `created_at DESC, id DESC` (deterministic export windows)
|
||||||
- `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals
|
- `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals
|
||||||
- `gbrain doctor` `eval_capture` check now distinguishes pre-v31 missing-table (status: ok / skipped) from RLS-denied SELECT or transient DB error (status: warn) — the diagnostic that surfaces a misconfigured RLS role no longer goes silent
|
- `gbrain doctor` `eval_capture` check now distinguishes pre-v31 missing-table (status: ok / skipped) from RLS-denied SELECT or transient DB error (status: warn) — the diagnostic that surfaces a misconfigured RLS role no longer goes silent
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ strict behavior when unset.
|
|||||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process.
|
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||||
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
||||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||||
|
|||||||
+47
-3
@@ -124,12 +124,55 @@ See `docs/ENGINES.md` for the full guide. In short:
|
|||||||
|
|
||||||
The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`.
|
The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`.
|
||||||
|
|
||||||
|
## CONTRIBUTOR_MODE — turn on the dev loop
|
||||||
|
|
||||||
|
gbrain captures retrieval traffic so you can replay real queries against
|
||||||
|
your code changes before merging. **This is off by default** (production
|
||||||
|
users get a quiet brain, no surprise data accumulation). Contributors turn
|
||||||
|
it on with one shell rc line:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In ~/.zshrc or ~/.bashrc:
|
||||||
|
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||||
|
```
|
||||||
|
|
||||||
|
That's it. Every `query` / `search` you (or agents pointed at your dev
|
||||||
|
brain) run from that shell now writes a row to `eval_candidates`, and the
|
||||||
|
[replay tool](#running-real-world-eval-benchmarks-touching-retrieval-code)
|
||||||
|
has data to work against.
|
||||||
|
|
||||||
|
What CONTRIBUTOR_MODE actually does:
|
||||||
|
|
||||||
|
- Turns on `query`/`search` capture into the local `eval_candidates` table.
|
||||||
|
Without it the gate is closed and capture is a no-op.
|
||||||
|
- That's all. PII scrubbing, retention, and replay are independent.
|
||||||
|
|
||||||
|
Resolution order (most explicit wins):
|
||||||
|
|
||||||
|
1. `eval.capture: true` in `~/.gbrain/config.json` → on
|
||||||
|
2. `eval.capture: false` in `~/.gbrain/config.json` → off
|
||||||
|
3. `GBRAIN_CONTRIBUTOR_MODE=1` → on
|
||||||
|
4. otherwise → off
|
||||||
|
|
||||||
|
Quick check that capture is actually running:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
gbrain query "anything" >/dev/null
|
||||||
|
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
|
||||||
|
# (or `gbrain doctor` — surfaces silent capture failures cross-process)
|
||||||
|
```
|
||||||
|
|
||||||
|
To disable capture even with the env var set, write
|
||||||
|
`{"eval": {"capture": false}}` to `~/.gbrain/config.json` — explicit config
|
||||||
|
beats the env var both directions.
|
||||||
|
|
||||||
## Running real-world eval benchmarks (touching retrieval code)
|
## Running real-world eval benchmarks (touching retrieval code)
|
||||||
|
|
||||||
If your PR touches retrieval — search ranking, RRF fusion, embeddings,
|
If your PR touches retrieval — search ranking, RRF fusion, embeddings,
|
||||||
intent classification, query expansion, source boost, or the `query` /
|
intent classification, query expansion, source boost, or the `query` /
|
||||||
`search` op handlers — run `gbrain eval replay` against a snapshot of
|
`search` op handlers — run `gbrain eval replay` against a snapshot of
|
||||||
real traffic before merging.
|
real traffic before merging. Requires `CONTRIBUTOR_MODE` (above) so you
|
||||||
|
have captured rows to replay against.
|
||||||
|
|
||||||
Quick loop:
|
Quick loop:
|
||||||
|
|
||||||
@@ -154,8 +197,9 @@ Trigger paths (rerun if your diff touches any of these):
|
|||||||
searchVector SQL)
|
searchVector SQL)
|
||||||
|
|
||||||
See [`docs/eval-bench.md`](./docs/eval-bench.md) for the full guide
|
See [`docs/eval-bench.md`](./docs/eval-bench.md) for the full guide
|
||||||
including CI integration, hand-crafted NDJSON corpora, and the cost
|
including CI integration, hand-crafted NDJSON corpora (so a fresh checkout
|
||||||
considerations. The NDJSON wire format is documented in
|
without captured data can still replay), and cost considerations. The
|
||||||
|
NDJSON wire format is documented in
|
||||||
[`docs/eval-capture.md`](./docs/eval-capture.md).
|
[`docs/eval-capture.md`](./docs/eval-capture.md).
|
||||||
|
|
||||||
## Welcome PRs
|
## Welcome PRs
|
||||||
|
|||||||
@@ -746,6 +746,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
|||||||
|
|
||||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
||||||
|
|
||||||
|
If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in.
|
||||||
|
|
||||||
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|||||||
+36
-5
@@ -8,10 +8,36 @@ For the **NDJSON wire format** consumed by gbrain-evals, see
|
|||||||
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
|
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
|
||||||
that lives on top of that format.
|
that lives on top of that format.
|
||||||
|
|
||||||
|
## Prerequisite: turn on contributor mode
|
||||||
|
|
||||||
|
Capture is **off by default** for production users (privacy-positive — no
|
||||||
|
surprise data accumulation). Contributors flip it on with one line:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In ~/.zshrc or ~/.bashrc:
|
||||||
|
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
gbrain query "anything" >/dev/null
|
||||||
|
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
|
||||||
|
```
|
||||||
|
|
||||||
|
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"eval": {"capture": true}} // force on
|
||||||
|
{"eval": {"capture": false}} // force off
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicit config beats the env var both directions.
|
||||||
|
|
||||||
## The 4-command loop
|
## The 4-command loop
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# ① Capture: already happening on every MCP / CLI / subagent query, capture is on by default.
|
# ① Capture: writes to eval_candidates whenever CONTRIBUTOR_MODE is set.
|
||||||
# Inspect what's been collected:
|
# Inspect what's been collected:
|
||||||
gbrain doctor # surfaces capture failures
|
gbrain doctor # surfaces capture failures
|
||||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
|
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
|
||||||
@@ -172,15 +198,20 @@ pipeline (`gbrain eval --qrels` with the sibling
|
|||||||
|
|
||||||
## Off-switch
|
## Off-switch
|
||||||
|
|
||||||
If you don't want capture at all (privacy concern, dev environment), edit
|
Two ways to disable capture:
|
||||||
`~/.gbrain/config.json`:
|
|
||||||
|
```bash
|
||||||
|
unset GBRAIN_CONTRIBUTOR_MODE # easy: just unset the env var
|
||||||
|
```
|
||||||
|
|
||||||
|
Or force off regardless of the env var via `~/.gbrain/config.json`:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"eval": {"capture": false}}
|
{"eval": {"capture": false}}
|
||||||
```
|
```
|
||||||
|
|
||||||
Capture stops; existing `eval_candidates` rows stay until you `gbrain eval
|
Existing `eval_candidates` rows stay until you `gbrain eval prune
|
||||||
prune --older-than 0d` (or just drop the table).
|
--older-than 0d` (or just drop the table).
|
||||||
|
|
||||||
## Failure modes
|
## Failure modes
|
||||||
|
|
||||||
|
|||||||
+22
-6
@@ -120,9 +120,18 @@ Reason enum (stable): `db_down` | `rls_reject` | `check_violation` |
|
|||||||
point — `gbrain doctor` runs in its own process and reads the table
|
point — `gbrain doctor` runs in its own process and reads the table
|
||||||
directly, so in-process counters wouldn't work.
|
directly, so in-process counters wouldn't work.
|
||||||
|
|
||||||
## Config (file-plane only)
|
## Config + CONTRIBUTOR_MODE
|
||||||
|
|
||||||
`~/.gbrain/config.json`:
|
Capture is **off by default** as of v0.25.0 (was on for everyone in
|
||||||
|
earlier drafts). Two paths to turn it on:
|
||||||
|
|
||||||
|
**Path A — env var (contributor opt-in, the common case):**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export GBRAIN_CONTRIBUTOR_MODE=1 # in ~/.zshrc or ~/.bashrc
|
||||||
|
```
|
||||||
|
|
||||||
|
**Path B — explicit config (`~/.gbrain/config.json`, file-plane only):**
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -135,10 +144,17 @@ directly, so in-process counters wouldn't work.
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Both default to `true`. Set `eval.capture: false` to disable capture
|
Resolution order (most explicit wins):
|
||||||
entirely; set `eval.scrub_pii: false` to preserve raw query text
|
|
||||||
(only if you control the brain's distribution).
|
1. `eval.capture: true` in config → on
|
||||||
|
2. `eval.capture: false` in config → off (overrides CONTRIBUTOR_MODE=1)
|
||||||
|
3. `GBRAIN_CONTRIBUTOR_MODE === '1'` → on
|
||||||
|
4. otherwise → off
|
||||||
|
|
||||||
|
`scrub_pii` defaults to `true` independent of capture. Set
|
||||||
|
`eval.scrub_pii: false` to preserve raw query text (only if you control
|
||||||
|
the brain's distribution).
|
||||||
|
|
||||||
`gbrain config set eval.capture false` does **not** work — that
|
`gbrain config set eval.capture false` does **not** work — that
|
||||||
command writes the DB-plane config, and the MCP server reads the
|
command writes the DB-plane config, and the MCP server reads the
|
||||||
file-plane. Edit the JSON directly.
|
file-plane. Edit the JSON directly or use the env var.
|
||||||
|
|||||||
+3
-1
@@ -140,7 +140,7 @@ strict behavior when unset.
|
|||||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process.
|
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||||
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
||||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||||
@@ -2115,6 +2115,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
|||||||
|
|
||||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
||||||
|
|
||||||
|
If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in.
|
||||||
|
|
||||||
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|||||||
@@ -173,19 +173,36 @@ export async function captureEvalCandidate(
|
|||||||
/**
|
/**
|
||||||
* Check whether capture is enabled for this process.
|
* Check whether capture is enabled for this process.
|
||||||
*
|
*
|
||||||
* File-plane only: reads `config.eval.capture` from the loaded GBrainConfig.
|
* Resolution order:
|
||||||
* `gbrain config set` writes the DB plane and does NOT affect this flag.
|
* 1. `config.eval.capture === true` → on (explicit user opt-in wins)
|
||||||
* Capture is on by default; returns false only when config explicitly sets
|
* 2. `config.eval.capture === false` → off (explicit user opt-out wins)
|
||||||
* `eval.capture: false`.
|
* 3. `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'` → on (contributor opt-in)
|
||||||
|
* 4. otherwise → off (default-off, privacy-positive for end users)
|
||||||
|
*
|
||||||
|
* The default flipped in v0.25.0 from "on for everyone" to "off unless you
|
||||||
|
* opt in." Capturing every query a real user runs without their consent is
|
||||||
|
* a footgun even with PII scrubbing; tying capture to CONTRIBUTOR_MODE makes
|
||||||
|
* the developer-skill nature of the feature explicit. Production users get
|
||||||
|
* a quiet brain; contributors get the BrainBench-Real replay loop with one
|
||||||
|
* shell rc line. See docs/eval-bench.md and CONTRIBUTING.md.
|
||||||
*
|
*
|
||||||
* Takes the already-loaded config so callers control the loadConfig()
|
* Takes the already-loaded config so callers control the loadConfig()
|
||||||
* lifecycle (MCP server loads once at boot, CLI commands load per-invocation).
|
* lifecycle (MCP server loads once at boot, CLI commands load per-invocation).
|
||||||
*/
|
*/
|
||||||
export function isEvalCaptureEnabled(config: GBrainConfig | null | undefined): boolean {
|
export function isEvalCaptureEnabled(config: GBrainConfig | null | undefined): boolean {
|
||||||
return config?.eval?.capture !== false;
|
if (config?.eval?.capture === true) return true;
|
||||||
|
if (config?.eval?.capture === false) return false;
|
||||||
|
return process.env.GBRAIN_CONTRIBUTOR_MODE === '1';
|
||||||
}
|
}
|
||||||
|
|
||||||
/** PII scrubbing enabled? Defaults to true; explicit `false` opts out. */
|
/**
|
||||||
|
* PII scrubbing enabled? Defaults to true; explicit `false` opts out.
|
||||||
|
*
|
||||||
|
* Independent of `isEvalCaptureEnabled` — scrubbing only matters when capture
|
||||||
|
* is actually running, but the gate stays separate so CONTRIBUTOR_MODE
|
||||||
|
* doesn't accidentally turn off scrubbing on a brain that happens to also
|
||||||
|
* have explicit `capture: true`.
|
||||||
|
*/
|
||||||
export function isEvalScrubEnabled(config: GBrainConfig | null | undefined): boolean {
|
export function isEvalScrubEnabled(config: GBrainConfig | null | undefined): boolean {
|
||||||
return config?.eval?.scrub_pii !== false;
|
return config?.eval?.scrub_pii !== false;
|
||||||
}
|
}
|
||||||
|
|||||||
+87
-13
@@ -214,24 +214,98 @@ describe('captureEvalCandidate — best-effort failure handling', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('isEvalCaptureEnabled / isEvalScrubEnabled', () => {
|
describe('isEvalCaptureEnabled / isEvalScrubEnabled (CONTRIBUTOR_MODE-gated)', () => {
|
||||||
test('defaults to enabled when config is null or missing', () => {
|
// v0.25.0 flipped the default: was on for everyone, now off unless either
|
||||||
expect(isEvalCaptureEnabled(null)).toBe(true);
|
// the env var or an explicit config flag is set. Tests scope env mutation
|
||||||
expect(isEvalCaptureEnabled(undefined)).toBe(true);
|
// so they don't leak across describe blocks.
|
||||||
expect(isEvalScrubEnabled(null)).toBe(true);
|
const origMode = process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||||
|
const restore = () => {
|
||||||
|
if (origMode === undefined) delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||||
|
else process.env.GBRAIN_CONTRIBUTOR_MODE = origMode;
|
||||||
|
};
|
||||||
|
|
||||||
|
test('defaults to OFF when config is missing AND CONTRIBUTOR_MODE unset', () => {
|
||||||
|
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||||
|
try {
|
||||||
|
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||||
|
expect(isEvalCaptureEnabled(undefined)).toBe(false);
|
||||||
|
} finally { restore(); }
|
||||||
});
|
});
|
||||||
|
|
||||||
test('returns false only when explicitly set to false', () => {
|
test('CONTRIBUTOR_MODE=1 turns capture on without any config', () => {
|
||||||
|
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||||
|
try {
|
||||||
|
expect(isEvalCaptureEnabled(null)).toBe(true);
|
||||||
|
expect(isEvalCaptureEnabled(undefined)).toBe(true);
|
||||||
|
} finally { restore(); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('CONTRIBUTOR_MODE=anything-else does NOT turn capture on (strict equality)', () => {
|
||||||
|
process.env.GBRAIN_CONTRIBUTOR_MODE = 'true';
|
||||||
|
try {
|
||||||
|
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||||
|
} finally { restore(); }
|
||||||
|
process.env.GBRAIN_CONTRIBUTOR_MODE = 'yes';
|
||||||
|
try {
|
||||||
|
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||||
|
} finally { restore(); }
|
||||||
|
process.env.GBRAIN_CONTRIBUTOR_MODE = '';
|
||||||
|
try {
|
||||||
|
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||||
|
} finally { restore(); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('explicit config eval.capture=true wins over absent CONTRIBUTOR_MODE', () => {
|
||||||
|
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||||
|
try {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
const enabled: any = { engine: 'pglite', eval: { capture: true } };
|
||||||
|
expect(isEvalCaptureEnabled(enabled)).toBe(true);
|
||||||
|
} finally { restore(); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('explicit config eval.capture=false wins over CONTRIBUTOR_MODE=1', () => {
|
||||||
|
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||||
|
try {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
const disabled: any = { engine: 'pglite', eval: { capture: false } };
|
||||||
|
expect(isEvalCaptureEnabled(disabled)).toBe(false);
|
||||||
|
} finally { restore(); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('config eval present but capture key undefined falls through to env check', () => {
|
||||||
|
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||||
|
try {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
const partial: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||||
|
expect(isEvalCaptureEnabled(partial)).toBe(false);
|
||||||
|
} finally { restore(); }
|
||||||
|
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||||
|
try {
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
const partial: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||||
|
expect(isEvalCaptureEnabled(partial)).toBe(true);
|
||||||
|
} finally { restore(); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('isEvalScrubEnabled: defaults true, only false when explicitly disabled', () => {
|
||||||
|
expect(isEvalScrubEnabled(null)).toBe(true);
|
||||||
|
expect(isEvalScrubEnabled(undefined)).toBe(true);
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
const disabled: any = { engine: 'pglite', eval: { capture: false, scrub_pii: false } };
|
const partial: any = { engine: 'pglite', eval: {} };
|
||||||
expect(isEvalCaptureEnabled(disabled)).toBe(false);
|
expect(isEvalScrubEnabled(partial)).toBe(true);
|
||||||
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
const disabled: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||||
expect(isEvalScrubEnabled(disabled)).toBe(false);
|
expect(isEvalScrubEnabled(disabled)).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('defaults to enabled when the eval key exists but the flag is undefined', () => {
|
test('isEvalScrubEnabled is independent of CONTRIBUTOR_MODE', () => {
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||||
const partial: any = { engine: 'pglite', eval: {} };
|
try {
|
||||||
expect(isEvalCaptureEnabled(partial)).toBe(true);
|
expect(isEvalScrubEnabled(null)).toBe(true);
|
||||||
expect(isEvalScrubEnabled(partial)).toBe(true);
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
|
const opt: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||||
|
expect(isEvalScrubEnabled(opt)).toBe(false);
|
||||||
|
} finally { restore(); }
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user