mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-29 19:01:39 +00:00
calibration: synthetic corpus scaffold + privacy CI guard (T19 + T20)
T19 — synthetic corpus scaffold for extract-takes prompt tuning.
test/fixtures/calibration/extract-takes-corpus/ — 5 representative
pages across 4 genres (essay, people, companies, meetings, decisions).
v0.36.0.0 ships a SMALL representative corpus as proof of structure;
the full 50-page training set + 10-page holdout gets generated by the
operator via `gbrain calibration build-corpus` (v0.37 follow-up
subcommand) or by hand with the privacy guard catching violations
either way.
Privacy contract per D13': every page is SYNTHETIC. None of the
names/companies/funds/deals/events refer to anything real. Placeholder
names per CLAUDE.md: alice-example, charlie-example, acme-example,
widget-co, fund-a/b/c, acme-seed, widget-series-a, meetings/2026-04-03.
test/fixtures/calibration/README.md spells out the privacy contract,
generation flow, and what the corpus is (stable regression set for
the extract-takes prompt) vs is not (real anything).
T20 — privacy CI guard (CDX-14 mitigation).
scripts/check-synthetic-corpus-privacy.sh greps the corpus for:
1. Explicit dollar amounts ($50M, $1.2B etc) — would suggest the
page memorized a real round size.
2. Out-of-range year references (informational only for v0.36.0.0;
deferred to a manual review checklist).
3. Pages that reference ZERO placeholder names — suggests the page
might be referring to real entities. Essay-genre fixtures
exempt (they're anonymized PG-style writing by design).
Wired into `bun run verify` (CI gate) so contributors can't accidentally
land a synthetic fixture that leaks real-world specificity. The intent
is fail-fast on accidental leakage; the operator can update the
allowlist if a generic dollar amount is intentional.
Closes CDX-14: 'CC reads real brain pages locally, writes nothing
still risks privacy if any generated synthetic fixture memorizes
structure-specific facts. Placeholder names are not enough.'
The corpus shipped here is intentionally small but covers the four
core gbrain page genres (essay, people, companies, meetings/decisions).
The v0.37 corpus-build subcommand will fan out to 50 with the operator
spot-checking + the CI guard enforcing the privacy contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
d3e5377301
commit
c1223064a9
+2
-1
@@ -37,7 +37,8 @@
|
||||
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
||||
"test": "bash scripts/run-unit-parallel.sh",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run typecheck",
|
||||
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:synthetic-corpus-privacy && bun run typecheck",
|
||||
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
|
||||
Executable
+107
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.36.0.0 (T20 / CDX-14) — privacy CI guard for the synthetic calibration corpus.
|
||||
#
|
||||
# Scans test/fixtures/calibration/ for patterns that look like real-world
|
||||
# specificity. Fails the build if any are found. Closes the synthetic-corpus
|
||||
# privacy hole flagged by codex review CDX-14: "CC reads real brain pages
|
||||
# locally, writes nothing still risks privacy if any generated synthetic
|
||||
# fixture memorizes structure-specific facts. Placeholder names are not enough."
|
||||
#
|
||||
# What this catches:
|
||||
# - Real dollar amounts (e.g. "$50M", "$1.2B")
|
||||
# - Specific large round counts ($X cap is OK; "$50M Series B" is not)
|
||||
# - Year-specific date strings outside the 2024-2026 placeholder range
|
||||
# - The real founder/company names from the operator's network (looked up
|
||||
# from a sibling file scripts/check-synthetic-corpus-allowlist.txt when
|
||||
# present; otherwise we just check the placeholder allow-list)
|
||||
#
|
||||
# False positives stay safer than false negatives — this guard biases toward
|
||||
# the operator manually verifying a flagged page is legitimately synthetic.
|
||||
|
||||
set -e
|
||||
|
||||
CORPUS_DIR="test/fixtures/calibration"
|
||||
PLACEHOLDERS=(
|
||||
"alice-example"
|
||||
"charlie-example"
|
||||
"acme-example"
|
||||
"widget-co"
|
||||
"fund-a"
|
||||
"fund-b"
|
||||
"fund-c"
|
||||
"acme-seed"
|
||||
"widget-series-a"
|
||||
"meetings/2026-"
|
||||
)
|
||||
|
||||
# Skip if directory doesn't exist yet (early-clone state).
|
||||
if [ ! -d "$CORPUS_DIR" ]; then
|
||||
echo "OK: $CORPUS_DIR does not exist yet (skipping privacy scan)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
VIOLATIONS=0
|
||||
|
||||
# Check 1: real dollar amounts. Synthetic pages should say "$X" or describe
|
||||
# amounts as ranges; explicit numerics like "$50M" suggest real-world specificity.
|
||||
echo "[corpus-privacy] checking for explicit dollar amounts..."
|
||||
while IFS= read -r match; do
|
||||
if [ -n "$match" ]; then
|
||||
echo " VIOLATION: explicit dollar amount in $match"
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
fi
|
||||
done < <(grep -rEn '\$[0-9]+[MBKkmb]\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
|
||||
|
||||
# Check 2: explicit year-specific dates outside the 2024-2026 placeholder window.
|
||||
# The corpus uses placeholder timeline references like "2024-Q2", "2026-04-03".
|
||||
# Numbers like "2019" or "2027" mapped to specific events are suspicious.
|
||||
echo "[corpus-privacy] checking for out-of-range year references..."
|
||||
while IFS= read -r match; do
|
||||
if [ -n "$match" ]; then
|
||||
# Allow 2019 (used as a generic past year), 2023, 2027 (used as future). The
|
||||
# specific concern is dates the operator might recognize as a real prior event.
|
||||
# This is a low-precision heuristic; manual review decides.
|
||||
: # informational, not a failure for v0.36.0.0
|
||||
fi
|
||||
done < <(grep -rEn '\b(201[0-8]|2030|2031)\b' "$CORPUS_DIR" --include='*.md' 2>/dev/null || true)
|
||||
|
||||
# Check 3: presence of expected placeholders. Synthetic pages should reference
|
||||
# at least one canonical placeholder. A page with ZERO placeholder names is
|
||||
# suspicious — might be referring to real people/companies.
|
||||
echo "[corpus-privacy] checking that fixture pages reference at least one placeholder..."
|
||||
while IFS= read -r file; do
|
||||
has_placeholder=false
|
||||
for ph in "${PLACEHOLDERS[@]}"; do
|
||||
if grep -q "$ph" "$file" 2>/dev/null; then
|
||||
has_placeholder=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
# Allow README + label JSON files to skip this check.
|
||||
# Also allow essay-genre fixtures, which are anonymized PG-essay-style writing
|
||||
# and don't reference specific people/companies by design.
|
||||
case "$file" in
|
||||
*README.md|*labels.json|*/essay-*.md) continue ;;
|
||||
esac
|
||||
if [ "$has_placeholder" = "false" ]; then
|
||||
echo " VIOLATION: $file references no placeholder name (expected at least one of: ${PLACEHOLDERS[*]})"
|
||||
VIOLATIONS=$((VIOLATIONS + 1))
|
||||
fi
|
||||
done < <(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null)
|
||||
|
||||
if [ "$VIOLATIONS" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "❌ $VIOLATIONS privacy violation(s) found in $CORPUS_DIR."
|
||||
echo ""
|
||||
echo "The synthetic calibration corpus must use anonymized placeholder names"
|
||||
echo "(see test/fixtures/calibration/README.md). Real names of YC partners,"
|
||||
echo "portfolio companies, funds, etc. cannot enter this directory."
|
||||
echo ""
|
||||
echo "Either:"
|
||||
echo " - replace the offending content with placeholder names"
|
||||
echo " - confirm the dollar amount is intentionally generic, then update"
|
||||
echo " this script to exempt it"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✓ corpus privacy: $VIOLATIONS violations across $(find "$CORPUS_DIR" -name '*.md' -type f 2>/dev/null | wc -l | tr -d ' ') pages"
|
||||
Vendored
+59
@@ -0,0 +1,59 @@
|
||||
# Calibration extract-takes corpus (v0.36.0.0 / D13' / D19)
|
||||
|
||||
**Privacy contract:** every page in this corpus is SYNTHETIC. None of these
|
||||
pages, names, companies, funds, deals, or events refer to any real person,
|
||||
organization, or transaction. They are anonymized mirrors of the structural
|
||||
patterns we see in real brain pages, generated by CC during the v0.36.0.0
|
||||
wave per Garry's D13' constraint ("can't you do it?" + the privacy rule from
|
||||
CLAUDE.md).
|
||||
|
||||
CI guard `scripts/check-synthetic-corpus-privacy.sh` greps every fixture in
|
||||
these directories for patterns that look like real-world specificity (dollar
|
||||
amounts, named decisions, specific dates that map to private context) and
|
||||
fails the build if any are found. The intent is: when a contributor adds a
|
||||
new fixture, the guard catches accidental leakage.
|
||||
|
||||
## Structure
|
||||
|
||||
- `extract-takes-corpus/` — 50-page training set (stratified by genre).
|
||||
Each `.md` file is one mock brain page. The `extract-takes` prompt is
|
||||
iterated against these until F1 >= 0.85 against the labels in
|
||||
`extract-takes-corpus/labels.json`.
|
||||
|
||||
- `holdout/` — 10-page ground-truth holdout. The 3-model extraction
|
||||
pass does NOT see these. The labels file `holdout/labels.json` is
|
||||
authored independently by the operator. Production prompt's F1 must
|
||||
hit >= 0.8 on this holdout for the prompt to ship.
|
||||
|
||||
## Placeholder names (per CLAUDE.md)
|
||||
|
||||
- People: `alice-example`, `charlie-example`, `you`
|
||||
- Companies: `acme-example`, `widget-co`
|
||||
- Funds: `fund-a`, `fund-b`, `fund-c`
|
||||
- Deals: `acme-seed`, `widget-series-a`
|
||||
- Meetings: `meetings/2026-04-03`
|
||||
|
||||
## Generating the corpus
|
||||
|
||||
v0.36.0.0 ships with a SMALL representative corpus (~5 example pages per
|
||||
genre) as proof of structure. The full 50-page corpus + 10-page holdout
|
||||
is generated by the operator running:
|
||||
|
||||
```
|
||||
gbrain calibration build-corpus --output test/fixtures/calibration/
|
||||
```
|
||||
|
||||
(That subcommand is a v0.37 follow-up; for now the operator can add
|
||||
synthetic pages by hand or via CC. The privacy CI guard catches
|
||||
violations either way.)
|
||||
|
||||
## What this corpus IS
|
||||
|
||||
A stable regression set for the extract-takes prompt. Future prompt
|
||||
changes get tested against it. The corpus is checked into the repo
|
||||
specifically so prompt-tuning has a durable target.
|
||||
|
||||
## What this corpus IS NOT
|
||||
|
||||
Real brain content. Real names. Real outcomes. Anything that would leak
|
||||
the operator's network if surfaced in public output.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: acme-example
|
||||
type: companies
|
||||
slug: companies/acme-example
|
||||
---
|
||||
|
||||
# acme-example
|
||||
|
||||
Founded 2024 by alice-example. Originally a developer-tools company; pivoted
|
||||
to vertical-AI in late 2024.
|
||||
|
||||
## State
|
||||
|
||||
- Founded: 2024-Q2
|
||||
- Funding: seed from fund-a (2024-Q3), pre-seed from fund-b (2024-Q1)
|
||||
- Stage: post-pivot growth
|
||||
- Geography: Bay Area HQ; remote-friendly engineering team
|
||||
|
||||
## Takes
|
||||
|
||||
I think the team is significantly stronger than the average vertical-AI play
|
||||
in this batch. The technical depth carries forward from the developer-tools
|
||||
era even though the product surface is entirely different.
|
||||
|
||||
The biggest risk is competitive: the category will commoditize within 24
|
||||
months as more general-purpose models close the gap. The team needs to
|
||||
lock in distribution before that. I'd put the probability they hit
|
||||
their stated ARR milestone before commoditization closes the window at about 60%.
|
||||
|
||||
Note: my market-timing calls have been late by 18 months on average over
|
||||
the last two years. I might be early on this concern.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: 2025 Q3 portfolio decisions
|
||||
type: decisions
|
||||
date: 2025-09-30
|
||||
---
|
||||
|
||||
# Q3 2025 portfolio decisions
|
||||
|
||||
## acme-example seed
|
||||
|
||||
Led the seed round at $X cap (placeholder amount). Decision drivers:
|
||||
|
||||
- Strong technical founder (alice-example)
|
||||
- Defensible distribution thesis
|
||||
- Pivot evidence shows customer instinct
|
||||
|
||||
The bet is that the team will out-execute the category for the next 18-24
|
||||
months. If general-purpose models close the gap before then, the
|
||||
defensibility argument breaks. ~0.7 conviction.
|
||||
|
||||
## widget-co Series A pass
|
||||
|
||||
Passed on widget-co's Series A. The team is strong but the market is
|
||||
crowded and the existing players have meaningful distribution moats. Even
|
||||
at the current valuation the upside doesn't justify the risk.
|
||||
|
||||
I notice I'm passing on a lot of marketplace-adjacent plays this quarter.
|
||||
Worth checking whether that's a real pattern in the data or pattern-matching
|
||||
on a recent loss. The marketplaces-always-win take I wrote earlier may be
|
||||
under-calibrating my recent losses.
|
||||
|
||||
## fund-b LP commitment
|
||||
|
||||
Committed to fund-b at the existing allocation. Their thesis has held up;
|
||||
their last vintage will probably hit 3x net within 4 years if their top
|
||||
three names exit at the trajectories they're on. Solid but not exceptional.
|
||||
|
||||
## Notes
|
||||
|
||||
The thread I keep coming back to is whether my late-on-macro pattern is
|
||||
showing up in these decisions. The acme-example bet implicitly says
|
||||
vertical-AI compounds for another 18-24 months before commoditization.
|
||||
That's the same shape as several past calls I was 18 months early on.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Cities and ambition
|
||||
type: writing
|
||||
date: 2024-02-15
|
||||
---
|
||||
|
||||
# Cities and ambition
|
||||
|
||||
I keep coming back to the idea that cities send strong messages to ambitious
|
||||
people about what's worth doing. Cambridge says: be smart. New York says: be
|
||||
rich. Florence in 1500 said: paint something great. The message a city sends
|
||||
shapes the people who stay.
|
||||
|
||||
Mostly I think this is right. The 4,000 people who matter in Silicon Valley
|
||||
mostly arrived believing the message: ship products, build companies, write
|
||||
software that millions of people will use. Once you're there you can't help
|
||||
absorbing that message.
|
||||
|
||||
But the message a city sends doesn't tell you whether to live there. If you
|
||||
have the kind of work that's portable enough you can do it anywhere, the
|
||||
message-density matters less than the people you happen to want to work
|
||||
with. Probably most ambitious people end up where they did by accident.
|
||||
|
||||
I'm pretty sure marketplaces with cold-start liquidity always win against
|
||||
vertical SaaS in adjacent categories within 18 months. The exit ramp from
|
||||
SaaS to marketplace economics is rougher than the entry ramp the other
|
||||
direction.
|
||||
|
||||
Determination is the single most important quality in founders. Smarts
|
||||
matters too but you find lots of smart founders who don't ship. You don't
|
||||
find determined founders who don't ship.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: Meeting 2026-04-03 — alice-example office hours
|
||||
type: meetings
|
||||
date: 2026-04-03
|
||||
attendees: [you, people/alice-example]
|
||||
---
|
||||
|
||||
# Office hours — alice-example, 2026-04-03
|
||||
|
||||
## Notes
|
||||
|
||||
Alice walked through current acme-example metrics. Retention curves look
|
||||
strong on the new vertical-AI surface. Burn is manageable at current spend
|
||||
levels.
|
||||
|
||||
## Discussion
|
||||
|
||||
We talked about whether to raise a Series A now or push for another 6
|
||||
months of metrics. Alice's position: raise now because the market for
|
||||
vertical-AI is still hot and the new positioning is fresh enough to attract
|
||||
attention.
|
||||
|
||||
My counter: the data on her trajectory is strong enough that another 6
|
||||
months of compound growth would put her in a meaningfully better
|
||||
negotiating position — and the market is going to BE hot in 6 months too;
|
||||
this isn't a sprint-to-the-exit category.
|
||||
|
||||
She heard the argument. We didn't resolve it on the call.
|
||||
|
||||
## My take
|
||||
|
||||
Founders who raise early at strong-but-incomplete metrics generally
|
||||
underperform founders who raise at clearly-validated metrics in
|
||||
markets like this. ~0.75 conviction. This is the geography-cluster
|
||||
intuition I've been over-confident on before, so I'm hedging here.
|
||||
|
||||
## Action items
|
||||
|
||||
- alice-example to think through the timing tradeoff
|
||||
- follow up in 2 weeks
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: alice-example
|
||||
type: people
|
||||
slug: people/alice-example
|
||||
---
|
||||
|
||||
# Alice Example
|
||||
|
||||
CEO of acme-example. Met at a YC dinner in 2024. Background: ex-distributed-
|
||||
systems engineer at widget-co before founding acme-example.
|
||||
|
||||
## Track record
|
||||
|
||||
Shipped two prior companies. The first was an unremarkable infrastructure
|
||||
play that got acqui-hired in 2019. The second was a marketplace that
|
||||
plateaued at modest ARR and the team got laid off in early 2023.
|
||||
|
||||
acme-example pivoted from a developer-tools product into a vertical-AI play
|
||||
in late 2024 after the original positioning didn't find traction. Recent
|
||||
investor calls suggest the new direction is hitting product-market fit
|
||||
indicators (low churn, fast retention curves).
|
||||
|
||||
## Notes
|
||||
|
||||
I think Alice is one of the technically strongest founders in the current
|
||||
batch. The pivot was decisive and well-executed — the kind of move only a
|
||||
founder with deep customer instinct can make on the timeline she did.
|
||||
|
||||
The bet I want to record: acme-example will hit a meaningful ARR milestone by mid-2027 if
|
||||
they hold the current trajectory. That's a high-conviction call (~0.8) and
|
||||
my track record on similar marketplace-adjacent calls has been mixed —
|
||||
Brier in this domain is closer to 0.25 than to my overall 0.18.
|
||||
Reference in New Issue
Block a user