calibration: synthetic corpus scaffold + privacy CI guard (T19 + T20)

T19 — synthetic corpus scaffold for extract-takes prompt tuning.
  test/fixtures/calibration/extract-takes-corpus/ — 5 representative
  pages across 4 genres (essay, people, companies, meetings, decisions).
  v0.36.0.0 ships a SMALL representative corpus as proof of structure;
  the full 50-page training set + 10-page holdout gets generated by the
  operator via `gbrain calibration build-corpus` (v0.37 follow-up
  subcommand) or by hand with the privacy guard catching violations
  either way.

  Privacy contract per D13': every page is SYNTHETIC. None of the
  names/companies/funds/deals/events refer to anything real. Placeholder
  names per CLAUDE.md: alice-example, charlie-example, acme-example,
  widget-co, fund-a/b/c, acme-seed, widget-series-a, meetings/2026-04-03.

  test/fixtures/calibration/README.md spells out the privacy contract,
  generation flow, and what the corpus is (stable regression set for
  the extract-takes prompt) vs is not (real anything).

T20 — privacy CI guard (CDX-14 mitigation).
  scripts/check-synthetic-corpus-privacy.sh greps the corpus for:
    1. Explicit dollar amounts ($50M, $1.2B etc) — would suggest the
       page memorized a real round size.
    2. Out-of-range year references (informational only for v0.36.0.0;
       deferred to a manual review checklist).
    3. Pages that reference ZERO placeholder names — suggests the page
       might be referring to real entities. Essay-genre fixtures
       exempt (they're anonymized PG-style writing by design).

  Wired into `bun run verify` (CI gate) so contributors can't accidentally
  land a synthetic fixture that leaks real-world specificity. The intent
  is fail-fast on accidental leakage; the operator can update the
  allowlist if a generic dollar amount is intentional.

  Closes CDX-14: 'CC reads real brain pages locally, writes nothing
  still risks privacy if any generated synthetic fixture memorizes
  structure-specific facts. Placeholder names are not enough.'

The corpus shipped here is intentionally small but covers the four
core gbrain page genres (essay, people, companies, meetings/decisions).
The v0.37 corpus-build subcommand will fan out to 50 with the operator
spot-checking + the CI guard enforcing the privacy contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-17 17:07:24 -07:00
co-authored by Claude Opus 4.7
parent d3e5377301
commit c1223064a9
8 changed files with 345 additions and 1 deletions
+107
View File
@@ -0,0 +1,107 @@
#!/usr/bin/env bash
# v0.36.0.0 (T20 / CDX-14) — privacy CI guard for the synthetic calibration corpus.
#
# Scans test/fixtures/calibration/ for patterns that look like real-world
# specificity. Fails the build if any are found. Closes the synthetic-corpus
# privacy hole flagged by codex review CDX-14: "CC reads real brain pages
# locally, writes nothing still risks privacy if any generated synthetic
# fixture memorizes structure-specific facts. Placeholder names are not enough."
#
# What this catches:
# - Real dollar amounts (e.g. "$50M", "$1.2B")
# - Specific large round counts ($X cap is OK; "$50M Series B" is not)
# - Year-specific date strings outside the 2024-2026 placeholder range
# - The real founder/company names from the operator's network (looked up
# from a sibling file scripts/check-synthetic-corpus-allowlist.txt when
# present; otherwise we just check the placeholder allow-list)
#
# False positives stay safer than false negatives — this guard biases toward
# the operator manually verifying a flagged page is legitimately synthetic.
set -e
CORPUS_DIR="test/fixtures/calibration"
PLACEHOLDERS=(
"alice-example"
"charlie-example"
"acme-example"
"widget-co"
"fund-a"
"fund-b"
"fund-c"
"acme-seed"
"widget-series-a"
"meetings/2026-"
)
# Skip if directory doesn't exist yet (early-clone state).
if [ ! -d "$CORPUS_DIR" ]; then
echo "OK: $CORPUS_DIR does not exist yet (skipping privacy scan)"
exit 0
fi
VIOLATIONS=0
# Check 1: real dollar amounts. Synthetic pages should say "$X" or describe
# amounts as ranges; explicit numerics like "$50M" suggest real-world specificity.
echo "[corpus-privacy] checking for explicit dollar amounts..."
while IFS= read -r match; do
if [ -n "$match" ]; then
echo " VIOLATION: explicit dollar amount in $match"
VIOLATIONS=$((VIOLATIONS + 1))
fi
done < <(grep -rEn '\$[0-9]+[MBKkmb]\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
# Check 2: explicit year-specific dates outside the 2024-2026 placeholder window.
# The corpus uses placeholder timeline references like "2024-Q2", "2026-04-03".
# Numbers like "2019" or "2027" mapped to specific events are suspicious.
echo "[corpus-privacy] checking for out-of-range year references..."
while IFS= read -r match; do
if [ -n "$match" ]; then
# Allow 2019 (used as a generic past year), 2023, 2027 (used as future). The
# specific concern is dates the operator might recognize as a real prior event.
# This is a low-precision heuristic; manual review decides.
: # informational, not a failure for v0.36.0.0
fi
done < <(grep -rEn '\b(201[0-8]|2030|2031)\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
# Check 3: presence of expected placeholders. Synthetic pages should reference
# at least one canonical placeholder. A page with ZERO placeholder names is
# suspicious — might be referring to real people/companies.
echo "[corpus-privacy] checking that fixture pages reference at least one placeholder..."
while IFS= read -r file; do
has_placeholder=false
for ph in "${PLACEHOLDERS[@]}"; do
if grep -q "$ph" "$file" 2>/dev/null; then
has_placeholder=true
break
fi
done
# Allow README + label JSON files to skip this check.
# Also allow essay-genre fixtures, which are anonymized PG-essay-style writing
# and don't reference specific people/companies by design.
case "$file" in
*README.md|*labels.json|*/essay-*.md) continue ;;
esac
if [ "$has_placeholder" = "false" ]; then
echo " VIOLATION: $file references no placeholder name (expected at least one of: ${PLACEHOLDERS[*]})"
VIOLATIONS=$((VIOLATIONS + 1))
fi
done < <(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null)
if [ "$VIOLATIONS" -gt 0 ]; then
echo ""
echo "$VIOLATIONS privacy violation(s) found in $CORPUS_DIR."
echo ""
echo "The synthetic calibration corpus must use anonymized placeholder names"
echo "(see test/fixtures/calibration/README.md). Real names of YC partners,"
echo "portfolio companies, funds, etc. cannot enter this directory."
echo ""
echo "Either:"
echo " - replace the offending content with placeholder names"
echo " - confirm the dollar amount is intentionally generic, then update"
echo " this script to exempt it"
exit 1
fi
echo "✓ corpus privacy: $VIOLATIONS violations across $(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null | wc -l | tr -d ' ') pages"