mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
calibration: synthetic corpus scaffold + privacy CI guard (T19 + T20)
T19 — synthetic corpus scaffold for extract-takes prompt tuning.
test/fixtures/calibration/extract-takes-corpus/ — 5 representative
pages across 4 genres (essay, people, companies, meetings, decisions).
v0.36.0.0 ships a SMALL representative corpus as proof of structure;
the full 50-page training set + 10-page holdout gets generated by the
operator via `gbrain calibration build-corpus` (v0.37 follow-up
subcommand) or by hand with the privacy guard catching violations
either way.
Privacy contract per D13': every page is SYNTHETIC. None of the
names/companies/funds/deals/events refer to anything real. Placeholder
names per CLAUDE.md: alice-example, charlie-example, acme-example,
widget-co, fund-a/b/c, acme-seed, widget-series-a, meetings/2026-04-03.
test/fixtures/calibration/README.md spells out the privacy contract,
generation flow, and what the corpus is (stable regression set for
the extract-takes prompt) vs is not (real anything).
T20 — privacy CI guard (CDX-14 mitigation).
scripts/check-synthetic-corpus-privacy.sh greps the corpus for:
1. Explicit dollar amounts ($50M, $1.2B etc) — would suggest the
page memorized a real round size.
2. Out-of-range year references (informational only for v0.36.0.0;
deferred to a manual review checklist).
3. Pages that reference ZERO placeholder names — suggests the page
might be referring to real entities. Essay-genre fixtures
exempt (they're anonymized PG-style writing by design).
Wired into `bun run verify` (CI gate) so contributors can't accidentally
land a synthetic fixture that leaks real-world specificity. The intent
is fail-fast on accidental leakage; the operator can update the
allowlist if a generic dollar amount is intentional.
Closes CDX-14: 'CC reads real brain pages locally, writes nothing
still risks privacy if any generated synthetic fixture memorizes
structure-specific facts. Placeholder names are not enough.'
The corpus shipped here is intentionally small but covers the four
core gbrain page genres (essay, people, companies, meetings/decisions).
The v0.37 corpus-build subcommand will fan out to 50 with the operator
spot-checking + the CI guard enforcing the privacy contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
d3e5377301
commit
c1223064a9
Executable
+107
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.36.0.0 (T20 / CDX-14) — privacy CI guard for the synthetic calibration corpus.
|
||||
#
|
||||
# Scans test/fixtures/calibration/ for patterns that look like real-world
|
||||
# specificity. Fails the build if any are found. Closes the synthetic-corpus
|
||||
# privacy hole flagged by codex review CDX-14: "CC reads real brain pages
|
||||
# locally, writes nothing still risks privacy if any generated synthetic
|
||||
# fixture memorizes structure-specific facts. Placeholder names are not enough."
|
||||
#
|
||||
# What this catches:
|
||||
# - Real dollar amounts (e.g. "$50M", "$1.2B")
|
||||
# - Specific large round counts ($X cap is OK; "$50M Series B" is not)
|
||||
# - Year-specific date strings outside the 2024-2026 placeholder range
|
||||
# - The real founder/company names from the operator's network (looked up
|
||||
# from a sibling file scripts/check-synthetic-corpus-allowlist.txt when
|
||||
# present; otherwise we just check the placeholder allow-list)
|
||||
#
|
||||
# False positives stay safer than false negatives — this guard biases toward
|
||||
# the operator manually verifying a flagged page is legitimately synthetic.
|
||||
|
||||
set -e
|
||||
|
||||
CORPUS_DIR="test/fixtures/calibration"
|
||||
PLACEHOLDERS=(
|
||||
"alice-example"
|
||||
"charlie-example"
|
||||
"acme-example"
|
||||
"widget-co"
|
||||
"fund-a"
|
||||
"fund-b"
|
||||
"fund-c"
|
||||
"acme-seed"
|
||||
"widget-series-a"
|
||||
"meetings/2026-"
|
||||
)
|
||||
|
||||
# Skip if directory doesn't exist yet (early-clone state).
|
||||
if [ ! -d "$CORPUS_DIR" ]; then
|
||||
echo "OK: $CORPUS_DIR does not exist yet (skipping privacy scan)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
VIOLATIONS=0
|
||||
|
||||
# Check 1: real dollar amounts. Synthetic pages should say "$X" or describe
|
||||
# amounts as ranges; explicit numerics like "$50M" suggest real-world specificity.
|
||||
echo "[corpus-privacy] checking for explicit dollar amounts..."
|
||||
while IFS= read -r match; do
|
||||
if [ -n "$match" ]; then
|
||||
echo " VIOLATION: explicit dollar amount in $match"
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
fi
|
||||
done < <(grep -rEn '\$[0-9]+[MBKkmb]\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
|
||||
|
||||
# Check 2: explicit year-specific dates outside the 2024-2026 placeholder window.
|
||||
# The corpus uses placeholder timeline references like "2024-Q2", "2026-04-03".
|
||||
# Numbers like "2019" or "2027" mapped to specific events are suspicious.
|
||||
echo "[corpus-privacy] checking for out-of-range year references..."
|
||||
while IFS= read -r match; do
|
||||
if [ -n "$match" ]; then
|
||||
# Allow 2019 (used as a generic past year), 2023, 2027 (used as future). The
|
||||
# specific concern is dates the operator might recognize as a real prior event.
|
||||
# This is a low-precision heuristic; manual review decides.
|
||||
: # informational, not a failure for v0.36.0.0
|
||||
fi
|
||||
done < <(grep -rEn '\b(201[0-8]|2030|2031)\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
|
||||
|
||||
# Check 3: presence of expected placeholders. Synthetic pages should reference
|
||||
# at least one canonical placeholder. A page with ZERO placeholder names is
|
||||
# suspicious — might be referring to real people/companies.
|
||||
echo "[corpus-privacy] checking that fixture pages reference at least one placeholder..."
|
||||
while IFS= read -r file; do
|
||||
has_placeholder=false
|
||||
for ph in "${PLACEHOLDERS[@]}"; do
|
||||
if grep -q "$ph" "$file" 2>/dev/null; then
|
||||
has_placeholder=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
# Allow README + label JSON files to skip this check.
|
||||
# Also allow essay-genre fixtures, which are anonymized PG-essay-style writing
|
||||
# and don't reference specific people/companies by design.
|
||||
case "$file" in
|
||||
*README.md|*labels.json|*/essay-*.md) continue ;;
|
||||
esac
|
||||
if [ "$has_placeholder" = "false" ]; then
|
||||
echo " VIOLATION: $file references no placeholder name (expected at least one of: ${PLACEHOLDERS[*]})"
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
fi
|
||||
done < <(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null)
|
||||
|
||||
if [ "$VIOLATIONS" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ $VIOLATIONS privacy violation(s) found in $CORPUS_DIR."
|
||||
echo ""
|
||||
echo "The synthetic calibration corpus must use anonymized placeholder names"
|
||||
echo "(see test/fixtures/calibration/README.md). Real names of YC partners,"
|
||||
echo "portfolio companies, funds, etc. cannot enter this directory."
|
||||
echo ""
|
||||
echo "Either:"
|
||||
echo " - replace the offending content with placeholder names"
|
||||
echo " - confirm the dollar amount is intentionally generic, then update"
|
||||
echo " this script to exempt it"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✓ corpus privacy: $VIOLATIONS violations across $(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null | wc -l | tr -d ' ') pages"
|
||||
Reference in New Issue
Block a user