mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 14:59:47 +00:00
* feat(eval-longmemeval): --by-type flag + question field + resume-replace
Per-question JSONL row gains `question`, `question_type`, and (when
ground truth is available) `recall_hit` — additive fields that existing
consumers (LongMemEval's `evaluate_qa.py`) ignore. New `--by-type` flag
emits a `{kind:"by_type_summary", recall_by_type, aggregate}` line at
the end of the output, resume-safe: rebuilt from existing rows so the
final aggregate covers cumulative resumed questions, prior summary at
the tail replaced rather than appended. New `--by-type-floor F` exits
non-zero per breached question_type. Empty-bucket guard emits null rate
not NaN. Exports `buildByTypeSummary` + `emitByTypeSummary` +
`seedRecallByTypeFromFile` for unit testing.
* feat(eval-cross-modal): --batch flag + semaphore + DI seam
Adds `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT]
[--yes]` to the existing eval cross-modal command. Mutually exclusive
with --task. Reads LongMemEval-shape JSONL output, filters by_type_summary
rows automatically, fans out via a new `runWithLimit<T>` semaphore
primitive (default --concurrent 3 x 3 model slots = 9 simultaneous calls;
below tier-1 rate limits on all 3 providers). Pre-flight cost estimate
refuses past --max-usd (default $5) unless --yes. Per-question receipts
written to a per-batch tempdir + deleted at end of run so
~/.gbrain/eval-receipts/ stays clean; summary receipt inlines verdicts.
Exit precedence (new batch-level policy, not inherited from aggregate.ts):
ERROR > FAIL > INCONCLUSIVE > PASS — any per-question runtime error exits 2.
New `runEvalCrossModal(args, opts?: {runEval?})` DI seam mirrors the
existing eval-longmemeval pattern. Tests pass a stub runEval so unit tests
don't need API keys; gateway availability check is also skipped when
opts.runEval is provided. Pinned by 17 cases.
* test: hermetic qrels retrieval gate against synthetic basis-vector corpus
Adds test/eval-replay-gate.test.ts as a unit-shard test (NOT under
test/e2e/ — the unit-shard CI matrix runs every PR via bun test;
test/e2e/ is fixed-file). Seeds a PGLite engine with synthetic
placeholder-name pages whose embeddings are basis vectors (same pattern
as test/e2e/search-quality.test.ts:23-28) so retrieval is hermetic — no
API keys, no DATABASE_URL, fully deterministic.
The qrels fixture at test/fixtures/eval-baselines/qrels-search.json has
12 hand-curated queries; each maps to a ranked list of relevant slugs +
`first_relevant_slug` (expected top-1). For each query, the gate asserts
`top1_match_rate >= 0.80` AND `recall_at_10 >= 0.85`. Env-overridable
floors via GBRAIN_REPLAY_GATE_TOP1_FLOOR / GBRAIN_REPLAY_GATE_RECALL_FLOOR
through withEnv(). Gate-fire prints per-query HIT/miss + recall to stderr.
When ranking changes intentionally move expected slugs, edit
qrels-search.json directly with a 'Why:' line in the commit body —
documented in docs/eval-bench.md.
scripts/check-test-real-names.sh allowlist gains 6 entries for the
privacy-grep regression guard inside the test, which must literally
spell the names it forbids to assert they're NOT in the fixture (same
meta-rule exception as skillpack-harvest privacy tests).
* feat(autopilot): opt-in nightly cross-modal quality probe + doctor check
Composes `gbrain eval longmemeval --by-type` + `gbrain eval cross-modal
--batch` into a 24h-cadenced quality check. Default DISABLED — opt-in via
`gbrain config set autopilot.nightly_quality_probe.enabled true` so new
users don't discover background API spend.
src/core/cycle/nightly-quality-probe.ts ships the phase implementation
with a full NightlyProbeDeps DI surface (isEnabled, hasEmbeddingProvider,
resolveMaxUsd, resolveRepoRoot, runLongMemEval, runCrossModalBatch, now)
so tests stub every external effect — no PGLite, no real LLM calls.
Pure `shouldRunNightly(now, recentEvents, windowMs?)` rate-limit fn.
src/core/audit-quality-probe.ts is the ISO-week-rotated JSONL writer
(mirrors audit-slug-fallback.ts; honors GBRAIN_AUDIT_DIR). One event per
run: outcome (pass/fail/inconclusive/error/budget_exceeded/rate_limited/
no_embedding_key), exit code, pass/fail/error counts, est_cost_usd,
fixture_sha8.
src/commands/doctor.ts gains a `nightly_quality_probe_health` check:
SKIPPED with paste-ready enable command when disabled; OK with timestamp
when all PASS in last 7 days; WARN with per-outcome counts when any
FAIL/ERROR/BUDGET_EXCEEDED. Extracted as pure
`computeNightlyQualityProbeHealthCheck(probeEnabled, events)` for
unit testing.
test/fixtures/longmemeval-nightly.jsonl is a 10-question placeholder
dataset (synthetic names only) distinct from the existing 5-question
mini fixture so the probe has consistent regression signal.
Real expected cost: ~$0.35/night = ~$10.50/month. Worst-case at
default $5 cap: $150/month.
Pinned by 21 cases in test/nightly-quality-probe.test.ts covering the
rate-limit pure function, every outcome branch, and all 7 branches of
the doctor check.
Autopilot scheduler wiring deferred to v0.41+ — the phase is callable
in isolation today (via the DI surface); cycle-loop dispatcher
integration filed in TODOS.md as a follow-up.
* docs: document Track D eval surfaces + file v0.41+ follow-up TODOs
docs/eval-bench.md gains a 'v0.40.1.0 Track D — Eval infrastructure'
section covering: --by-type usage + resume-replace semantics, the
hermetic qrels gate workflow + 'Why:' commit-body refresh convention,
--batch end-to-end with cost-bound + concurrency knobs, and the opt-in
nightly probe enable workflow + cost ceiling.
TODOS.md files two follow-ups:
- v0.41+: contributor-mode CI capture for BrainBench-Real replay gate
(the deferred original Task 2 design — replay against real captured
queries is more valuable than synthetic qrels long-term, but needs CI
secret + nightly capture pipeline + commit automation; deferred to a
dedicated wave)
- v0.41+: wire the nightly quality probe into autopilot scheduling
(phase callable in isolation today; cycle-loop dispatcher integration
is a ~3-hour follow-up)
CLAUDE.md Key Files annotations extended for the four lanes:
eval-longmemeval gains the --by-type description, eval-cross-modal
gains the --batch + DI seam description, new entries for the qrels
gate test + the nightly probe + audit-quality-probe writer.
* chore: bump version and changelog (v0.40.1.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(eval): close 4 codex-flagged eval-integrity bugs
Codex adversarial review on the Track D wave found 4 real ways the new
eval-gate code could silently bypass its gates. Each fix below either
counts what was previously dropped, fails fast on a parser edge case,
or enforces a gate that was previously skipped on an early-return path.
CDX-1: cross-modal --batch silently dropped failed/corrupt LongMemEval
rows. `gbrain eval longmemeval` emits {error:..., hypothesis:''} when
runOneQuestion throws; the batch reader's missing-field skip threw those
rows away, shrinking the denominator. A green eval on a subset is now
impossible:
- eval-longmemeval.ts: error rows now carry `question` + `question_type`
so the batch consumer can identify them as upstream failures, not
skip them as malformed.
- eval-cross-modal.ts: readBatchRows now returns {rows, upstream_errors,
malformed_count}. Upstream errors fold into per_question with verdict
'upstream_error'. BatchSummary gains `upstream_error_count` and
`malformed_count`. ERROR exit precedence widens to include both, so
any upstream failure exits 2.
CDX-2: --limit 0 was a direct CI bypass — zero-row check fired before
slicing, then the empty result fell through to verdict='pass'. Fixed
with a hard `limit >= 1` check.
CDX-3: --resume-from + --by-type-floor was a real gate skip. When a
prior run had every question answered, the early "nothing to do" return
fired BEFORE summary emission and floor enforcement. Now the no-op
resume path still seeds recallByType from the existing file, emits the
by_type_summary at the tail, and runs the floor gate.
CDX-5: doctor nightly_quality_probe_health only flagged fail / error /
budget_exceeded as warn. no_embedding_key / rate_limited / inconclusive
were silently reported as PASS — hiding misconfigurations and queue
backpressure. The bad-event filter is now `outcome !== 'pass'`, and the
counts string surfaces every bucket so the operator sees exactly what
went wrong.
scripts/check-privacy.sh: adds test/eval-replay-gate.test.ts to the
allowlist (the qrels test's privacy-grep regression guard literally
names what it forbids, same meta-rule exception as the existing
test/recency-decay.test.ts + skillpack-harvest allowlist entries).
Pinned by 8 new regression cases across eval-longmemeval (CDX-3),
eval-cross-modal-batch (CDX-1 + CDX-2), and nightly-quality-probe
(CDX-5). 76 Track D tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
213 lines
7.6 KiB
Bash
Executable File
213 lines
7.6 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
# check-privacy.sh — CLAUDE.md:550 enforcement.
|
|
#
|
|
# CLAUDE.md forbids the private OpenClaw fork name in public artifacts:
|
|
# CHANGELOG.md, README.md, docs/, skills/, PR titles + bodies, commit
|
|
# messages, and comments in checked-in code. This script greps for the
|
|
# banned name in either the staged index (for pre-commit hooks) or the
|
|
# working tree (for CI) and fails loudly if found.
|
|
#
|
|
# The allow-list below whitelists files where the name is legitimate
|
|
# — specifically, this script itself (where we reference the name to
|
|
# describe the rule) and upgrade guides that historically referenced
|
|
# the pre-rename fork name.
|
|
#
|
|
# Usage:
|
|
# scripts/check-privacy.sh # scan working tree
|
|
# scripts/check-privacy.sh --staged # scan git staged index
|
|
# scripts/check-privacy.sh --help
|
|
#
|
|
# Exit codes:
|
|
# 0 clean
|
|
# 1 banned name found (or --help printed)
|
|
# 2 git / grep not available
|
|
|
|
set -euo pipefail
|
|
|
|
BANNED_NAME='wintermute'
|
|
# v0.25.1 (codex T7): additional patterns from wintermute-specific filesystem
|
|
# layouts that would leak private fork context if they slipped through a port.
|
|
# `wintermute_only` already matches via the case-insensitive `wintermute` regex
|
|
# above; this list is for orthogonal patterns.
|
|
BANNED_PATHS=(
|
|
'/data/brain/'
|
|
'/data/.openclaw/'
|
|
)
|
|
|
|
usage() {
|
|
cat <<EOF
|
|
scripts/check-privacy.sh — scan for the banned OpenClaw fork name.
|
|
|
|
USAGE:
|
|
scripts/check-privacy.sh Scan all tracked files in the working tree.
|
|
scripts/check-privacy.sh --staged Scan only files staged for commit.
|
|
scripts/check-privacy.sh --help Show this message.
|
|
|
|
The script greps for '${BANNED_NAME}' (case-insensitive) in:
|
|
- CHANGELOG.md, README.md
|
|
- docs/**
|
|
- skills/**
|
|
- src/**
|
|
- test/**
|
|
- scripts/**
|
|
|
|
Allow-list (references to the name are permitted):
|
|
- scripts/check-privacy.sh itself
|
|
- docs/UPGRADING_DOWNSTREAM_AGENTS.md (historical context for pre-rename upgrades)
|
|
- .git/** (branch names, commit history — not checked in artifacts)
|
|
|
|
Exit codes: 0 clean, 1 banned name found, 2 setup error.
|
|
EOF
|
|
}
|
|
|
|
MODE=working
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--staged) MODE=staged ;;
|
|
--help|-h) usage; exit 1 ;;
|
|
*)
|
|
echo "Unknown argument: $arg" >&2
|
|
usage >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
done
|
|
|
|
if ! command -v git >/dev/null 2>&1; then
|
|
echo "check-privacy: git not found" >&2
|
|
exit 2
|
|
fi
|
|
|
|
# Build the file list by scanning-mode.
|
|
if [ "$MODE" = staged ]; then
|
|
FILES=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null || true)
|
|
else
|
|
FILES=$(git ls-files 2>/dev/null || true)
|
|
fi
|
|
|
|
if [ -z "$FILES" ]; then
|
|
exit 0
|
|
fi
|
|
|
|
# Allow-list: files in which the banned name is legitimate.
|
|
# Meta-rule docs (define the rule itself), auto-generated LLM indexes,
|
|
# historical upgrade guides, and the test that enforces the rule
|
|
# against recipes/ all reference the banned name by necessity.
|
|
ALLOW_LIST=(
|
|
'scripts/check-privacy.sh'
|
|
'CLAUDE.md'
|
|
'llms-full.txt'
|
|
'docs/UPGRADING_DOWNSTREAM_AGENTS.md'
|
|
'test/integrations.test.ts'
|
|
# v0.25.1 (codex T7) BANNED_PATHS allow-list:
|
|
# Historical docs, frozen migration files, test fixtures, and env-var
|
|
# fallbacks where /data/brain/ or /data/.openclaw/ appears legitimately.
|
|
# New skills/, src/, and tests must NOT slip onto this list — extend the
|
|
# banned check above instead.
|
|
'docs/GBRAIN_RECOMMENDED_SCHEMA.md'
|
|
'docs/GBRAIN_V0.md'
|
|
'docs/guides/minions-shell-jobs.md'
|
|
'scripts/smoke-test.sh'
|
|
'skills/migrations/v0.9.0.md'
|
|
'skills/migrations/v0.14.0.md'
|
|
'test/storage-status.test.ts'
|
|
# CHANGELOG.md documents the rule (the v0.25.1 entry references the
|
|
# banned literals in describing what's banned). Same exception status
|
|
# as CLAUDE.md and this script itself: meta-documentation needs to
|
|
# name the patterns it forbids.
|
|
'CHANGELOG.md'
|
|
# skills/migrations/v0.25.1.md is the agent-readable upgrade
|
|
# walkthrough; it explains the privacy-guard extension to the
|
|
# operating agent and references the banned literals while doing so.
|
|
'skills/migrations/v0.25.1.md'
|
|
# v0.29.1: the recency-decay default-map test asserts that
|
|
# DEFAULT_RECENCY_DECAY's keys do NOT include fork-specific path
|
|
# prefixes. The test must name the banned tokens to assert their
|
|
# absence — same exception status as scripts/check-privacy.sh,
|
|
# CHANGELOG.md, and CLAUDE.md (meta-rule enforcement requires
|
|
# mentioning what the rule forbids).
|
|
'test/recency-decay.test.ts'
|
|
# v0.32.5: the sibling check-test-real-names.sh enforces the same
|
|
# privacy rule for test fixtures and lists the banned names literally
|
|
# (Wintermute, Hermes, etc) inside its BANNED_NAMES + ALLOWLIST arrays.
|
|
# Same meta-rule-enforcement exception as scripts/check-privacy.sh itself.
|
|
'scripts/check-test-real-names.sh'
|
|
# v0.34 / Lane CI: scripts/check-proposal-pii.sh and its test list the
|
|
# banned literal as part of the structural denylist they enforce against
|
|
# docs/proposals/*.md. Same meta-rule-enforcement exception as the two
|
|
# entries above — describing what the rule forbids requires naming it.
|
|
'scripts/check-proposal-pii.sh'
|
|
'test/scripts/check-proposal-pii.test.ts'
|
|
# v0.32.3.0: the functional-area-resolver skill's behavior-contract
|
|
# section describes the privacy guarantees the skill preserves and
|
|
# references the banned literals while doing so (line 306). Same
|
|
# meta-rule-enforcement exception as scripts/check-privacy.sh and
|
|
# CHANGELOG.md — describing what the rule forbids requires naming it.
|
|
'skills/functional-area-resolver/SKILL.md'
|
|
# v0.36.0.0: the gbrain skillpack harvest privacy linter's whole job
|
|
# is to catch the banned literal leaking into gbrain. The regex
|
|
# pattern in harvest-lint.ts is `\bWintermute\b` by necessity; the
|
|
# tests verify that pattern fires by feeding it the banned string;
|
|
# the harvest skill markdown describes the substitution policy
|
|
# ("Wintermute → your OpenClaw") as part of the genericization
|
|
# checklist. Same meta-rule-enforcement exception as the privacy
|
|
# checks themselves.
|
|
'src/core/skillpack/harvest-lint.ts'
|
|
'test/skillpack-harvest-lint.test.ts'
|
|
'test/skillpack-harvest.test.ts'
|
|
'test/e2e/skillpack-flow.test.ts'
|
|
'skills/skillpack-harvest/SKILL.md'
|
|
# v0.40.1.0 Track D / T5: the qrels gate test contains a privacy-grep
|
|
# regression guard whose block list names the banned literal to assert
|
|
# it's NOT in the qrels fixture. Same meta-rule-enforcement exception
|
|
# as the other test entries above.
|
|
'test/eval-replay-gate.test.ts'
|
|
)
|
|
|
|
is_allowed() {
|
|
local f="$1"
|
|
for a in "${ALLOW_LIST[@]}"; do
|
|
if [ "$f" = "$a" ]; then
|
|
return 0
|
|
fi
|
|
done
|
|
return 1
|
|
}
|
|
|
|
FOUND=0
|
|
while IFS= read -r file; do
|
|
[ -z "$file" ] && continue
|
|
[ ! -f "$file" ] && continue
|
|
if is_allowed "$file"; then
|
|
continue
|
|
fi
|
|
# Case-insensitive grep; only specific extensions + known docs.
|
|
case "$file" in
|
|
*.md|*.ts|*.mjs|*.js|*.py|*.sh|*.json|*.yaml|*.yml|*.txt|README*|CHANGELOG*|CLAUDE*|AGENTS*)
|
|
if grep -in "$BANNED_NAME" "$file" >/dev/null 2>&1; then
|
|
echo "[check-privacy] BANNED NAME in $file:" >&2
|
|
grep -in "$BANNED_NAME" "$file" | sed 's|^| |' >&2
|
|
FOUND=1
|
|
fi
|
|
# Banned wintermute-specific filesystem paths (codex T7).
|
|
for path in "${BANNED_PATHS[@]}"; do
|
|
if grep -nF "$path" "$file" >/dev/null 2>&1; then
|
|
echo "[check-privacy] BANNED PATH '$path' in $file:" >&2
|
|
grep -nF "$path" "$file" | sed 's|^| |' >&2
|
|
FOUND=1
|
|
fi
|
|
done
|
|
;;
|
|
esac
|
|
done <<< "$FILES"
|
|
|
|
if [ "$FOUND" -eq 1 ]; then
|
|
echo "" >&2
|
|
echo "The private OpenClaw fork name is banned in public artifacts." >&2
|
|
echo "CLAUDE.md:550. Replace with 'your OpenClaw', 'OpenClaw reference deployment', or 'openclaw-reference'." >&2
|
|
exit 1
|
|
fi
|
|
|
|
exit 0
|