* docs: Knowledge Runtime design doc (draft) — 4-layer architecture + reduced-scope delta
Captures the Knowledge Runtime design thinking from the CEO review session:
Resolver SDK, Enrichment Orchestrator, Scheduler, Deterministic Output Builder.
The original 7-phase plan was drafted before v0.12.0 (knowledge graph layer)
and v0.11.x (Minions agent runtime) shipped. Cross-referenced against what's
already merged on master, roughly 60% of the 4-layer vision is already in
production under different names:
- Minions = scheduler + plugin contract (L1 + L3)
- Knowledge graph auto-link = deterministic output at L4 + orchestrator at L2
- BrainBench v1 benchmarks already validate the graph layer
The doc is kept as a draft design reference; the actual build-out will scope
down to the real delta (typed Resolver interface, BrainWriter API + validators,
BudgetLedger, CompletenessScorer, quiet-hours + stagger). See the CEO review
notes for the reduced plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolvers): Resolver SDK pass 1 — interface + registry (PR 1/5)
Adds the typed plugin interface that unifies external-lookup calls (X API,
Perplexity, HEAD check, brain-local slug resolution) behind a single shape:
registry.resolve('x_handle_to_tweet', { handle, keywords }, ctx)
→ { value, confidence, source, fetchedAt, raw? }
Zero behavior change — the registry is empty by default. Builtins
(url_reachable, x_handle_to_tweet) land in the next pass. ScheduledResolver
wrapping via Minions lands in PR 5.
New files:
- src/core/resolvers/interface.ts — Resolver<I,O>, ResolverResult<O>,
ResolverContext (engine, storage, config, logger, requestId, remote,
deadline, signal), ResolverError (not_found, already_registered,
unavailable, timeout, rate_limited, auth, schema, aborted, upstream)
- src/core/resolvers/registry.ts — ResolverRegistry (register/get/has/
list/resolve/clear/size) + getDefaultRegistry() for process-wide use
- src/core/resolvers/index.ts — barrel export
Design rules enforced by types:
- Every result carries confidence (0.0-1.0) + source attribution
- LLM-backed resolvers return confidence<1.0 by convention
- ctx.remote propagates the trust boundary (mirrors OperationContext.remote)
- AbortSignal threads through for cooperative cancellation
Smoke: imports + runs, list()/get()/resolve() behave as typed.
Dependency-free beyond types and storage/engine type imports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(fail-improve): optional AbortSignal — Resolver SDK pass 2 (PR 1/5)
Extends FailImproveLoop.execute with an optional `opts.signal` that threads
through the deterministic-first / LLM-fallback flow. Needed by the Resolver
SDK so long-running lookups can be cooperatively cancelled when a caller
aborts (deadline hit, Minion job timeout, user ctrl-c).
Additive and backwards-compatible:
- execute() signature widens callbacks to (input, signal?) => ...; existing
two-arg callbacks are structurally compatible and ignore the extra arg.
- opts is optional; callers that omit it get pre-extension behavior.
- Aborts throw a DOM-style AbortError (name='AbortError'), matching what
fetch() throws, so downstream `err.name === 'AbortError'` branches work
unchanged.
- Aborted runs are NOT logged to the failure JSONL — not informative and
would pollute pattern analysis.
Abort check fires in three places:
- Before the deterministic call (pre-flight)
- Between deterministic miss and LLM call (mid-flight)
- Inside llmFallbackFn if the implementation respects signal itself
Smoke tests: 5 scenarios (existing sig, llm fallback, pre-abort, mid-flight
abort, signal threaded to fallback) — all pass. Existing test/fail-improve.test.ts
(13 tests, 27 expects) unchanged and passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolvers): url_reachable + x_handle_to_tweet — SDK pass 3 (PR 1/5)
Two reference resolver implementations that validate the interface against
real-world requirements: a deterministic free-cost check and a rate-limited
paid-backend lookup.
src/core/resolvers/builtin/url-reachable.ts
HEAD-check a URL, follow redirects (max 5), detect dead links. Reused
isInternalUrl() from the wave-3 SSRF hardening; re-validates every redirect
hop against the same filter. Falls back from HEAD to GET on 405/501.
Composes caller's AbortSignal with a per-request timeout via
AbortSignal.any (with manual-propagation fallback). Confidence=1 when the
backend answers; confidence=0 only on transport failure (DNS/connect/timeout).
src/core/resolvers/builtin/x-api/handle-to-tweet.ts
Find a tweet by handle + free-text keyword hint. Used by the upcoming
`gbrain integrity --auto` loop to repair the 1,424 bare-tweet citations
in Garry's brain. Confidence buckets align with the three-bucket contract:
- >=0.8 auto-repair (single strong match, or dominant in small candidate set)
- 0.5-0.8 review queue (ambiguous but promising)
- <0.5 skip (many candidates or weak match)
Scoring: normalized keyword-token overlap against tweet text, with margin
boost for dominant matches. Strict handle regex (X's username rules).
Retries on 429 up to 2x with Retry-After honor. Terminal 401/403 surfaces
as auth ResolverError so the caller stops hammering. Bearer token read
from ctx.config.x_api_bearer_token or X_API_BEARER_TOKEN env — never logged.
Smoke: registry accepts both, SSRF blocks localhost + file://, available()
returns false when token missing, schema validator rejects bad handles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolvers): tests + gbrain resolvers CLI — SDK pass 4 (PR 1/5 complete)
Closes out PR 1. 43 new tests in test/resolvers.test.ts covering registry
contract, both reference builtins, all three confidence buckets, and every
ResolverError subcode.
test/resolvers.test.ts
- ResolverRegistry: register, duplicate-id rejection, get/has, list with
cost+backend filters, resolve, unavailable propagation, clear, default
singleton lifecycle.
- url_reachable: available(), SSRF guard on localhost + RFC1918 + 169.254
metadata + file:// scheme, empty-url schema error, 200/404 status
propagation, HEAD→GET fallback on 405, redirect chain, per-hop SSRF
re-validation, network failure → reachable=false, AbortSignal mid-flight.
- x_handle_to_tweet: token gate via env AND via ctx.config, invalid/long
handle schema errors, zero-candidate + single-strong + single-weak +
many-ambiguous confidence buckets (gates >=0.5 url emission), 401/403
auth error, 500 upstream error, 429 retry-then-rate_limited, X operator
stripping (prompt injection defense).
src/commands/resolvers.ts
- `gbrain resolvers list [--cost | --backend | --json]` pretty table
or JSON.
- `gbrain resolvers describe <id>` schema + availability detail.
- registerBuiltinResolvers() is idempotent; ready to be called from
future entry points (gbrain integrity, MCP server).
src/cli.ts wires `resolvers` into CLI_ONLY + dispatches to runResolvers.
Full suite: 1343 pass / 0 fail / 141 skip (E2E without DATABASE_URL).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(output): BrainWriter + Scaffolder + SlugRegistry — PR 2 pass 1/4
Lands the transactional writer library that the rest of the Knowledge
Runtime sits on top of. No callers routed through it yet — publish.ts /
backlinks.ts / put_page migrations are pass 4 and PR 2.5.
src/core/output/scaffold.ts
Deterministic URL / citation / link builders. Callers pass typed inputs
(handle + tweetId, account + messageId, slug + display text) and get
canonical markdown bytes out. LLM-generated URLs never touch disk.
- tweetCitation({handle, tweetId, dateISO?})
- emailCitation({account, messageId, subject, dateISO?})
- sourceCitation(resolverResult, {url?, label?})
- entityLink({slug, displayText, relativePrefix?})
- timelineLine({dateISO, summary, citation?})
ScaffoldError with codes for invalid_handle / invalid_tweet_id /
invalid_slug / invalid_message_id / invalid_date / empty.
src/core/output/slug-registry.ts
Solves the "Marc Benioff vs Marc-Benioff both slug to marc-benioff" bug.
create() probes engine.getPage and either returns the desired slug or
disambiguates (alice-smith → alice-smith-2). isFree() + suggestDisambiguators()
for interactive UX. Errors: collision, disambiguator_exhausted, invalid_slug.
src/core/output/writer.ts
BrainWriter.transaction(fn, ctx) wraps engine.transaction. The `fn`
callback receives a WriteTx with createEntity / appendTimeline /
setCompiledTruth / setFrontmatterField / putRawData / addLink (the last
creates both forward + reverse back-link atomically). On commit, per-page
validators run against all touchedSlugs. Strict mode throws on
error-severity findings, rolling back the outer tx. Lint mode (default for
PR 2 rollout) returns the report but commits regardless. Pages with
`validate: false` frontmatter skip validators entirely (grandfather hook
for PR 2 migration).
Integration smoke against PGLite: createEntity → disambiguator (2nd call
with same desired slug), addLink writes both forward + back-link,
strict-mode validator failure rolls back the transaction bit-identically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(output): 4 pre-commit validators + tests — PR 2 pass 2/4
Lands the validator suite that BrainWriter runs before committing a
transaction. Paragraph-level deterministic checks, markdown-aware, skip
legacy pages via validate:false frontmatter.
src/core/output/validators/citation.ts
Every factual paragraph in compiled_truth carries at least one citation
marker: [Source: ...] or a linked URL. Splits paragraphs on blank lines,
strips fenced code / inline code / HTML comments before checking.
Ignores headings, key-value lines ("**Status:** Active"), table rows,
pure wikilink bullets (## See Also), and short labels without a factual
verb. Deterministic — no LLM, no semantic judgment.
src/core/output/validators/link.ts
Every [text](path) wikilink resolves to a page that exists (unless it's
an external http(s) URL, which this validator doesn't check; that's
url_reachable's job in PR 3). Strips relative prefix and .md extension.
Batches engine.getPage lookups per unique target. mailto/anchor/other
schemes flagged as warning. Links inside fenced code blocks are skipped.
src/core/output/validators/back-link.ts
Iron Law: if page X → page Y, then Y → X. Reads engine.getLinks(ctx.slug),
and for each target checks engine.getLinks(target) for a reverse edge.
Missing reverses flagged as warning (runAutoLink is the authoritative
enforcer on put_page; this is defense-in-depth for pages edited outside
the main write path).
src/core/output/validators/triple-hr.ts
Catches hygiene issues on the compiled_truth / timeline split: bare `---`
in compiled_truth would re-split on round-trip through parseMarkdown;
headings in the timeline section signal authoring mistakes. Both warn
(not error) — legacy pages legitimately use thematic breaks.
src/core/output/validators/index.ts
registerBuiltinValidators(writer) wires all four.
test/writer.test.ts
57 tests: Scaffolder (all 5 helpers + error paths), SlugRegistry (create,
disambiguator, collision throw, invalid-slug, isFree, suggestDisambiguators),
BrainWriter (happy path, disambiguate, addLink + reverse, strict rollback,
lint proceeds with report, off skips validators, validate:false grandfather,
setCompiledTruth, setFrontmatterField merge, registered validators list),
citation validator (all 11 shape cases), link validator (normalizeToSlug
including ../../, external URL skip, mailto warning, code-fence skip),
back-link validator (no outbound, missing reverse → warning, bidirectional
clean), triple-hr validator (clean, bare --- warning, fenced --- skipped,
heading in timeline warning, ## Timeline header allowed).
Full suite: 1400 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(migrations): v0.13.0 grandfather validate:false — PR 2 pass 3/4
Adds the TS migration that makes BrainWriter's strict-mode rollout safe:
every existing page gets `validate: false` in frontmatter so the new
citation / link / back-link / triple-HR validators skip legacy content.
gbrain integrity --auto (PR 3) clears the flag per-page once real citations
are repaired.
src/commands/migrations/v0_13_0_add_validate_false.ts
Four-phase orchestrator following the v0_12_0 pattern:
A. connect — loadConfig + createEngine. Does NOT write config (prior
learning: gbrain init --migrate-only semantics; never
flip Postgres users to PGLite via bare init).
B. snapshot — engine.getAllSlugs() upfront (prior learning:
listpages-pagination-mutation; OFFSET iteration is
self-invalidating when each write bumps updated_at).
C. grandfather — per slug, skip if frontmatter.validate already set,
else append-log pre-mutation snapshot to
~/.gbrain/migrations/v0_13_0-rollback.jsonl and
putPage with validate:false merged in. Batched 100
at a time so interruption losses are bounded.
D. verify — SQL count of pages with validate=false ≥ expectedTouched.
Idempotent: second run is a no-op. Reversible: rollback log is
append-only JSONL; future `gbrain apply-migrations --rollback v0.13.0`
replays it. Safe on empty brains (returns complete with 0 touched).
src/commands/migrations/index.ts
Registers v0_13_0 after v0_12_0 in semver order.
test/migrations-v0_13_0.test.ts
Registry integration (v0.13.0 present, semver-after-v0.12.0, pitch
metadata well-formed), orchestrator handles no-config gracefully,
dryRun skips the connect phase.
test/apply-migrations.test.ts
Updated two assertions that hard-coded the v0.12.0 skippedFuture list
to also include v0.13.0 (now skippedFuture when installed < 0.13.0).
Full suite: 1405 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(integrity): gbrain integrity — bare-tweet repair + dead-link scan (PR 3)
Ships the user-visible milestone for the Knowledge Runtime delta: a
command that finds brain-integrity issues and repairs them through the
BrainWriter + Resolver SDK infrastructure from PRs 1 and 2.
Targets the two quantified pain points from brain/CITATIONS.md:
- 1,424 of 3,115 people pages have bare tweet references without URLs
- An unknown fraction of existing URL citations have rotted
Subcommands:
gbrain integrity check Read-only report, optional --json
gbrain integrity auto Three-bucket repair loop
gbrain integrity review Print review-queue path + count
gbrain integrity reset-progress Clear the progress file
Three-bucket contract (matches x_handle_to_tweet resolver's confidence
scoring):
>=0.8 → auto-repair via BrainWriter transaction. Appends a timeline
entry on the page with a Scaffolder-built tweet citation (URL
from the API response, never from LLM text).
0.5-0.8 → append to ~/.gbrain/integrity-review.md with all candidates
sorted by match score, for batch human review.
<0.5 → log reason to ~/.gbrain/integrity.log.jsonl and skip.
Resumable: every processed slug hits ~/.gbrain/integrity-progress.jsonl
so an interrupted run resumes from the last slug. --fresh clears it.
Bare-tweet detection patterns (regex, deterministic, skip code fences
and already-cited lines):
- "tweeted about"
- "in/on a (recent|viral) tweet"
- "wrote a tweet/post"
- "posted on X"
- "via X" (but not "via X/handle" — already cited)
- possessive "his/her/their tweet"
External-link detection extracts all [text](https?://...) pairs (code
fences skipped) for optional dead-link probing via url_reachable.
Dead links are surfaced, not auto-repaired — no "correct" replacement
exists without human judgment.
Wiring: runIntegrity dispatches subcommands, registers builtin resolvers
into the default registry, connects to the brain engine, and uses
BrainWriter in strict-off mode (integrity is the repair path, not the
write-gate path).
Unit tests: 21 cover bare-tweet regex (all 9 phrase shapes + code-fence
skip + URL-already-present skip + per-line dedup), external-link
extraction (http+https, line numbers, fenced skip), frontmatter handle
extraction (x_handle, twitter, twitter_handle, x; preference order;
leading @ strip; null paths). End-to-end auto flow verified manually
via the resolver SDK tests + BrainWriter tests it composes.
src/cli.ts wires `integrity` into CLI_ONLY + dispatches to runIntegrity.
Full suite: 1426 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(enrichment): BudgetLedger + CompletenessScorer — PR 4
Two layer-2 primitives that slot under the resolver SDK and BrainWriter:
cost-aware spend caps and evidence-weighted per-page completeness scoring.
Schema migration v11 adds two tables:
budget_ledger (scope, resolver_id, local_date) PK — midnight rollover by
date column means a new calendar day upserts a new row; no rollover
thread, no race.
budget_reservations (reservation_id) — TTL-bounded held reservations
(default 60s) so process death between reserve() and commit() doesn't
strand money.
Rollback plan: DROP TABLE. Budget data is regenerable from resolver call
logs; no durable product value lives in the ledger.
src/core/enrichment/budget.ts
BudgetLedger.reserve({resolverId, estimateUsd, capUsd?, ttlSeconds?})
serializes concurrent reserves on {scope, resolver_id, local_date} via
SELECT ... FOR UPDATE. Returns {kind:'held', reservationId, ...} or
{kind:'exhausted', reason, spent, pending, cap} — never over-spends.
commit(id, actualUsd) moves money from reserved_usd to committed_usd and
marks the reservation status='committed'. rollback(id) zeros out the
reservation without touching committed. Commit-after-commit throws
already_finalized; rollback-after-commit is a no-op (callers don't need
to guard). commit-unknown-id throws reservation_not_found.
cleanupExpired() sweeps held reservations past expires_at and rolls them
back; reserve() opportunistically reclaims the target row's expired
reservations before acquiring its own lock.
IANA timezone config via opts.tz (default America/Los_Angeles); midnight
rollover is naturally expressed as a date column + Intl.DateTimeFormat
with en-CA locale (YYYY-MM-DD). DST is handled by the formatter.
src/core/enrichment/completeness.ts
Seven per-type rubrics (person, company, project, deal, concept, source,
media) + default. Each rubric's dimension weights sum to 1.0, checked at
module load. scorePage(page) returns {score, dimensionScores, rubric}
where score is 0.000–1.000.
Person rubric dimensions: has_role_and_company, has_source_urls,
has_timeline_entries, has_citations, has_backlinks, recency_score,
non_redundancy. The last two are the explicit fix for the two pathologies
called out in the codex review of the earlier design: stale pages that
never decay (30-day re-enrich forever) and Wilco-style repeated blocks
that pass Wintermute's length heuristic.
Pure functions. No engine calls — BrainWriter invokes scorePage after a
transaction and caches the result in frontmatter.completeness.
test/enrichment.test.ts — 23 tests:
BudgetLedger: under-cap held, over-cap exhausted, commit moves money,
rollback clears, commit-rollback no-op, commit-commit throws, commit-
unknown throws, invalid input, empty state null, scope isolation,
parallel reserves respect cap (10 parallel, cap 1.0, est 0.3 each →
≤ 3 held; state.reservedUsd ≤ 1.0), cleanupExpired reclaims TTL=0.
CompletenessScorer: all 8 rubrics sum to 1.0, empty person scores <0.3,
fully-enriched person >0.8, dimension scores exposed, role detection,
company/concept/source/media/default routing, recency decay with age,
non_redundancy penalizes repeated lines.
Full suite: 1449 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): quiet-hours + stagger + claim-time gate — PR 5
Closes the scheduler gap per CEO plan: Minions v7 shipped a durable
runtime but nothing about when jobs should NOT run. This wires
quiet-hours enforcement at claim time (the codex correction — dispatch-
time is wrong because a queued job can become claimable after its window
opens) plus deterministic stagger slots to prevent cron-boundary storms.
Schema migration v12 adds two columns to minion_jobs:
quiet_hours JSONB — {start, end, tz, policy} window config
stagger_key TEXT — partitioning key for deterministic offset
Plus a partial index on stagger_key for later slot-assignment queries.
src/core/minions/quiet-hours.ts
evaluateQuietHours(cfg, now?) → 'allow' | 'skip' | 'defer'. Pure,
deterministic, no engine. Handles straight-line and wrap-around windows
(e.g. 22→7 spans midnight). IANA timezone via Intl.DateTimeFormat;
unknown tz fails open (allow) — safer than hard-blocking every job.
'skip' policy drops the event; 'defer' (default) re-queues for later.
src/core/minions/stagger.ts
staggerMinuteOffset(key) → 0–59, FNV-1a hash. Same key → same slot.
Pure; no module-level state. Used by scheduled resolvers that want to
avoid cron-boundary collisions ("10 jobs all fire at minute 0").
src/core/minions/worker.ts
MinionWorker.tick now consults evaluateQuietHours on every claimed job.
Verdict 'defer' → UPDATE status='delayed', delay_until = now() + 15m
(prevents immediate re-claim loops when the claim query re-runs).
Verdict 'skip' → UPDATE status='cancelled', error_text='skipped_quiet_hours'.
Both paths clear lock_token and require lock_token match in the WHERE
clause so a concurrent stall recovery can't race us.
test/minions-quiet-hours.test.ts — 25 tests:
evaluateQuietHours: null/undefined/invalid config paths (allow fail-open),
straight-line in/out + exclusive-end, wrap-around in (before midnight +
after), skip vs defer policy, timezone-offset propagation (winter PST
vs summer PDT), localHour parity with Date.getUTCHours.
staggerMinuteOffset: deterministic same key → same offset, different
keys spread across buckets (10 keys → ≥5 unique buckets), empty/non-
string edge cases.
Schema v12: quiet_hours and stagger_key columns exist on minion_jobs,
idx_minion_jobs_stagger_key index present.
Full suite: 1474 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(output): post-write validator lint hook — PR 2.5
Minimal integration of BrainWriter validators into the main write path,
feature-flag-gated and non-blocking. The CEO plan explicitly scoped PR 2.5
as a pre-soak landing step: the hook plugs in now, observability lands,
but strict-mode rejection is deferred to a follow-on release gated on the
7-day soak + BrainBench regression ≤1pt.
src/core/output/post-write.ts
runPostWriteLint(engine, slug, opts?) invokes the four BrainWriter
validators (citation, link, back-link, triple-hr) against a freshly
written page and returns a PostWriteLintResult. Skips cleanly when:
- config `writer.lint_on_put_page` is not truthy (default OFF; opts.force overrides)
- the page is not found (shouldn't happen in normal put_page flow)
- the page has frontmatter.validate === false (grandfathered)
Findings are logged to:
- ~/.gbrain/validator-lint.jsonl (capped at 20 findings per line)
- engine.logIngest (ingest_log table) for durable agent-inspectable history
Validator-level exceptions are swallowed so a buggy validator never
breaks put_page.
src/core/operations.ts put_page handler
After importFromContent + runAutoLink, imports runPostWriteLint and
invokes it. Result returns writer_lint: {error_count, warning_count} or
{skipped: reason}. Try/catch wraps the whole hook so an import or
runtime error never blocks the main write.
Enable locally:
gbrain config set writer.lint_on_put_page true
Then every put_page emits a writer_lint summary + appends structured
findings to the ingest log for analysis before the strict-mode flip.
test/post-write-lint.test.ts — 11 tests:
Flag reader (default off, true/1/on, other values false, explicit false)
Hook behavior (flag-off skip, page-not-found skip, validate:false
grandfather skip, force=true overrides flag, dirty page yields citation
error, clean page yields zero findings).
Full suite: 1485 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(migrations-v0_13_0): drop flaky no-config assertion
The 'does not succeed when no brain is configured' test assumed loadConfig
would return null when HOME is empty, but it also reads DATABASE_URL from
the environment. When .env.testing sources DATABASE_URL into the shell
(normal E2E lifecycle), the orchestrator connects successfully and runs
to completion — the test's assertion was unreachable.
The dry-run path is still covered by the remaining test in the same
describe block; registry integration and semver ordering are covered by
the sibling describe.
Full suite with DATABASE_URL live: 1574 pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(minions): wire quiet_hours + stagger_key into MinionJobInput + queue.add
Codex adversarial review caught that PR 5 (claim-time quiet-hours gate) was
cosmetic: the schema v12 column existed, the worker read it via
`readQuietHoursConfig(job)`, but `MinionJobInput` never accepted it,
`queue.add()` never inserted it, and `rowToMinionJob()` never mapped it out.
Result: every scheduled job saw `quiet_hours: null`, so the gate was a
no-op. Stagger_key had the same broken wiring.
- MinionJob (types.ts): add `quiet_hours` and `stagger_key` fields.
- MinionJobInput: add matching optional fields so callers can submit them.
- rowToMinionJob: parse both columns (JSONB handled the same way as `data`).
- MinionQueue.add: include both columns in the INSERT (idempotent + normal
paths), bound as $19/$20. The `$19::jsonb` cast matches the JSONB column
shape; the wire format is the same native-JS object path that fixed the
JSONB double-encode bug in v0.12.1.
After this, `await queue.add('x', {}, { quiet_hours: {start:22,end:7,
tz:"America/Los_Angeles",policy:"defer"} })` actually stores the window
and the worker's claim-time gate defers the job inside it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(minions): route quiet-hours 'skip' through cancelJob to rollup parents
Codex flagged that handleQuietHoursDefer with verdict='skip' directly set
status='cancelled' via raw UPDATE — bypassing MinionQueue.cancelJob, which
means:
- Parent jobs in 'waiting-children' never get rolled up.
- Descendant jobs don't cascade-cancel.
- Child-done inbox notification is skipped.
Result: a parent waiting on a child that fell inside quiet hours with
policy='skip' stays stuck forever.
Fix: release the lock, then delegate to queue.cancelJob(job.id) which
handles the recursive CTE + parent rollup + inbox posting correctly.
Falls back to a direct UPDATE only if cancelJob errors — even then, the
status transition is status-guarded to avoid stomping terminal states.
Defer path unchanged (no parent rollup needed since the job hasn't reached
a terminal state).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(budget): commit() re-checks cap + rejects negative actuals
Codex caught two cap-bypass bugs in BudgetLedger.commit():
1. reserve({estimateUsd: 0.01, capUsd: 1.0}) + commit(id, 100) silently
charged $100 to a $1-cap bucket. Cap is an advertised invariant that
the code was not enforcing.
2. Negative actuals (commit(id, -5)) were accepted, letting callers
artificially reduce committed_usd below the real spend. Refunds need
a dedicated API, not a side-channel on commit.
Fix:
- Reject non-finite AND negative actualUsd at entrypoint.
- Lock the ledger row FOR UPDATE during commit (same serialization as
reserve).
- Compute effective cap headroom = cap - other_committed - other_reserved
(excluding this reservation from the reserved pool since we're about to
finalize it).
- When actualUsd would exceed available, clamp committed_usd to max
available and throw BudgetError with the overage reported. The
reservation is still marked 'committed' (API call already happened;
don't retry-loop), but the cap is honored.
After this, a $1/day cap actually means $1/day.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(integrity): --dry-run no longer writes progress, poisoning resume
Codex caught that 'gbrain integrity auto --dry-run' appended progress
entries (status='repaired', 'reviewed', 'skipped', 'error') despite doing
no actual writes. The follow-on real run with default --resume would then
skip those slugs — the dry-run silently consumed the work queue.
Fix: gate every appendProgress() call in cmdAuto on !dryRun. Dry-run
still logs to the skip log / review queue (so the user sees what WOULD
happen), but the progress file stays untouched.
Behavior:
--dry-run → buckets counted + summary printed + review-queue
+ log populated, but progress file unchanged.
(default) → progress file tracks every processed slug, so
Ctrl-C + re-run resumes from the right place.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.13.0.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(resolvers): DNS-rebinding defense + X rate-limit header parity
Two non-blocking codex findings on PR #210 rolled into one bisectable
commit because their tests share an import line.
url_reachable: hostname-string SSRF guard is vulnerable to DNS rebinding
(attacker-controlled DNS returns a public IP at validate time and
169.254.169.254 at fetch time). Add checkDnsRebinding() that resolves
the hostname via dns.lookup({all:true}) and rejects any result whose
A/AAAA record lands in a private range (v4 via isPrivateIpv4, v6
loopback/link-local/unique-local/IPv4-mapped). Applied on the initial
URL and on every redirect target. Null on DNS failure so genuine
network problems surface via fetch.
x_handle_to_tweet: rate-limit backoff only honored Retry-After and
ignored X's proprietary x-rate-limit-reset header. computeBackoffMs()
parses both (Retry-After = seconds or HTTP-date; x-rate-limit-reset =
epoch seconds), takes MAX, and clamps to [2s, 60s]. Exported for
testability; callers use it uniformly on every 429.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(writer): advisory lock on desiredSlug prevents cross-process TOCTOU
BrainWriter's createEntity checks engine.getPage(slug) and falls back
to putPage(), which upserts. Two putPage('people/alice') calls from
separate processes (a Claude Code session + a Minions worker, say) can
both read "free" from SlugRegistry and both call putPage, silently
overwriting each other with no disambiguation.
Take a transaction-scoped advisory lock keyed on hashtext(desiredSlug)
before the registry check. Concurrent writers for the same slug now
serialize at the DB level: the second observes the first's commit and
disambiguates to alice-2. PGLite is single-process so this is a
harmless no-op there. Wrapped in try/catch so engines/test doubles
that don't support advisory locks fall through to the existing
within-process check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(validators): empty [Source:] no longer satisfies citation check
Regex /\[Source:[^\]]*\]/ matched decorative markers like [Source:]
and [Source: ] that carry zero provenance. Tighten to require at
least one non-whitespace character before the closing bracket. The
inline URL form ](https://...) already requires a scheme+host so it
stays as-is.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(auto-link): advisory lock serializes concurrent reconciliation
runAutoLink wraps getLinks + addLink/removeLink in a transaction, but
row-level locks alone don't prevent the union-of-writes race: two
concurrent put_page calls on the same slug can both read the same
existingKeys BEFORE either mutates a row, then proceed to add links
the other side's rewrite no longer mentions.
Take a transaction-scoped advisory lock on hashtext("auto_link:" ||
slug) at the start of the reconciliation. Concurrent writers on the
same slug now fully serialize; writers on different slugs still run
in parallel. No-op on engines without advisory locks (PGLite).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: expand coverage on abort-signal threading + integrity CLI dispatch
fail-improve: four new AbortSignal cases — pre-start abort, between
deterministic and LLM, signal forwarded into both callbacks, and
LLM-thrown AbortError propagates without logging a failure entry.
integrity: three new CLI dispatch cases — --help, no-subcommand (help),
and unknown subcommand (stderr + exit 1). Non-engine paths so they
exercise routing without spinning up a DB.
Coverage-only; no source changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): fold integrity sample scan into default health check
Expose scanIntegrity(engine, opts) as a pure library function — same
logic cmdCheck uses — and call it from doctor in non-fast mode with
a 500-page sampling limit. Surfaces bare-tweet phrase count and
external-link count as an 'integrity' check, warn-status when bare
tweets are present with a one-liner pointing at 'gbrain integrity
check' for the full report and 'integrity auto' for repair.
Read-only: no network, no writes, no resolver calls. Pages with
validate:false frontmatter are skipped (grandfathered). --fast mode
skips it entirely so the existing health-snapshot contract holds.
Users no longer need to remember three separate commands (doctor,
lint, integrity check) to audit brain health — doctor surfaces the
integrity signal by default, full scan stays available for deep dives.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(put_page): auto-extract timeline entries alongside auto-link
put_page already chunks, embeds, reconciles tags, and extracts
auto-links on every write. Timeline extraction has lived in a
separate command (gbrain extract timeline) that users had to remember
to run. Fold it into the write path: after the page commits, parse
timeline entries from compiled_truth + timeline body and insert via
addTimelineEntriesBatch. ON CONFLICT DO NOTHING keeps it idempotent
across re-writes.
Mirrors auto-link shape: best-effort post-hook, skipped for remote
(MCP) callers, gated by auto_timeline config (default TRUE). Response
includes auto_timeline: { created } alongside auto_links.
Side effect: a one-shot `gbrain put` now produces a complete page —
chunks, embeddings, links, AND timeline — instead of three commands
the user has to chain manually.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(migrate): verify target health after engine migration
After a PGLite↔Postgres migration, the user was left to run 'gbrain
doctor' themselves to confirm the target is good. Not great, because
the failure modes (partial copy, missing embeddings, schema drift)
all surface at next CLI use when the migration itself looks like it
succeeded.
Add verifyTarget() — inline doctor-lite that checks page count
matches the source, embedding coverage is above 90%, and schema
version is at latest. Prints a 3-line status table at the end of
migrate and points at 'gbrain doctor' for the full check. Non-fatal:
warns on discrepancies instead of failing the command so the user
sees the full picture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(bench): add v0.13 knowledge runtime benchmark deltas
Two new benchmark scripts + one consolidated markdown comparing this
branch against master (c0b6219, v0.12.1):
benchmark-put-page-latency.ts — 200 put_page ops, measures the
per-write cost of Step B's auto-timeline extraction. Branch adds
~0.5ms mean latency and produces 300 timeline entries for free;
master produces zero and requires a separate 'gbrain extract timeline'
pass.
benchmark-knowledge-runtime.ts — three measurements in one script:
time-to-queryable (branch 40/40 vs master 0/40 on post-ingest
timeline queries), integrity repair rate (70/20/10 three-bucket
split via mocked resolver), doctor completeness (surfaces 100% of
real issues after Step A, respects grandfathered pages).
docs/benchmarks/2026-04-19-knowledge-runtime-v0.13.md — consolidated
report. Covers the four moved benchmarks plus side-by-side runs of
graph-quality and search-quality showing they're identical across
master and branch. Proof of no regression on the retrieval hot path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
35 KiB
GBrain Knowledge Runtime — Design Doc
Status: DRAFT for CEO review. Date: 2026-04-18. Supersedes: The earlier "Feynman Ideas Assessment + Phase A/B" plan.
0. Context
During a CEO review of a narrow two-feature plan (bare-tweet citation repair + completeness score, borrowed from Feynman), the scope was reframed. The narrow plan duplicated work Wintermute already does and missed the real leverage point: the bespoke abstractions hiding inside Wintermute — resolvers, enrichment orchestration, scheduling, deterministic output — should live in GBrain as first-class primitives.
North star: "When Wintermute's Claw upgrades to this version of GBrain, it should immediately recognize brilliance and completeness and say 'It's time to switch to these abstractions.'"
That is the test this document is designed against. Everything else is downstream.
1. The Four Layers
The design is four layered abstractions. Each is independently useful; together they are the Knowledge Runtime.
┌───────────────────────────────────────────────────────────────────┐
│ KNOWLEDGE RUNTIME (new) │
├───────────────────────────────────────────────────────────────────┤
│ Layer 4: Deterministic Output Builder │
│ BrainWriter · Scaffolds · Back-link enforcer · Slug registry │
│ Rule: LLM picks WHAT to write. Code guarantees WHERE and HOW. │
├───────────────────────────────────────────────────────────────────┤
│ Layer 3: Scheduler │
│ ScheduledResolver · TZ-aware quiet hours (enforced) · │
│ Auto-stagger · Durable state · Retry/circuit-break │
├───────────────────────────────────────────────────────────────────┤
│ Layer 2: Enrichment Orchestrator │
│ Trigger convergence · Tier routing · Budget · Cascade · │
│ Evidence-weighted completeness · Fail-safe transactions │
├───────────────────────────────────────────────────────────────────┤
│ Layer 1: Resolver SDK │
│ Resolver<I,O> interface · Registry · Factory · Plugin recipes │
│ Ported reference impls: X-API, Perplexity, Mistral, brain │
└───────────────────────────────────────────────────────────────────┘
│ │
▼ ▼
REUSES (polished primitives already in GBrain) REPLACES (ad-hoc code)
FailImproveLoop · backoff · storage factory · enrichment-service ·
check-resolvable · operations validators · embedding · transcription ·
engine interface · publish · backlinks 2 recipe formats
2. Why This Order (L1 → L4)
Every higher layer depends on the lower one. L1 must land first or the rest leaks abstractions.
- L1 (Resolvers) is the substrate. Without a uniform lookup interface, every orchestrator + writer has bespoke callers.
- L2 (Orchestrator) uses L1 to fetch; without L1 it's still ad-hoc.
- L3 (Scheduler) runs L2 periodically; without L2 it's scheduling nothing structured.
- L4 (Output Builder) is what every layer ultimately writes through; without it we have 14 call sites doing
fs.writeFilewith hand-rolled citation discipline.
An earlier implementation could ship L1 + L4 first (the two "purest" layers) and have the most immediate integrity impact, then add L2 + L3. But the end-state must include all four.
3. Layer 1 — Resolver SDK
3.1 What's broken today
Wintermute has 69 distinct external-lookup patterns across X API (14 shapes), Perplexity, Mistral OCR, Gmail, Calendar, Slack, GitHub, YouTube, Diarize.io, YC tools, OSINT collectors, and brain-local lookups. Each one is a bespoke script under scripts/ with its own error handling, retry logic, and output shape. GBrain has 3 ad-hoc wrappers (embedding.ts, transcription.ts, enrichment-service.ts) that don't share an interface.
Common consequences:
- No uniform retry/backoff strategy (some scripts retry, most don't)
- No cost tracking (Perplexity bills eaten silently when calls return no-substance results)
- No confidence/provenance propagation (callers can't tell if an answer is verified or inferred)
- Users can't add a resolver without forking GBrain
3.2 Interface
// src/core/resolvers/interface.ts
export type ResolverCost = 'free' | 'rate-limited' | 'paid';
export interface ResolverRequest<I> {
input: I;
context: ResolverContext;
timeoutMs?: number;
}
export interface ResolverResult<O> {
value: O;
confidence: number; // 0.0–1.0; 1.0 = deterministic from ground-truth API
source: string; // e.g. "x-api-v2", "perplexity-sonar", "brain-local"
fetchedAt: Date;
costEstimate?: number; // dollars; 0 if free
raw?: unknown; // for sidecar preservation via put_raw_data
}
export interface Resolver<I, O> {
readonly id: string; // stable, slug-like: "x_handle_to_tweet"
readonly cost: ResolverCost;
readonly backend: string; // "x-api-v2", "perplexity", "brain-local"
readonly inputSchema: JSONSchema;
readonly outputSchema: JSONSchema;
available(ctx: ResolverContext): Promise<boolean>;
resolve(req: ResolverRequest<I>): Promise<ResolverResult<O>>;
}
3.3 Context
export interface ResolverContext {
engine: BrainEngine;
storage: StorageBackend;
config: GBrainConfig;
logger: Logger;
metrics: MetricsRecorder;
budget: BudgetLedger; // hard spend caps, queried pre-resolve
requestId: string;
remote: boolean; // trust boundary — untrusted callers get stricter validation
deadline?: Date;
}
3.4 Registry + Factory (mirrors src/core/storage.ts)
// src/core/resolvers/registry.ts
export class ResolverRegistry {
register<I, O>(r: Resolver<I, O>): void;
get(id: string): Resolver<unknown, unknown>;
list(filter?: { cost?: ResolverCost; backend?: string }): Resolver[];
async resolve<I, O>(id: string, input: I, ctx: ResolverContext): Promise<ResolverResult<O>>;
}
// src/core/resolvers/factory.ts (dynamic import like engine-factory)
export async function createResolver(
type: 'x-api' | 'perplexity' | 'mistral-ocr' | 'brain-local' | 'plugin',
config: ResolverConfig,
): Promise<Resolver>;
3.5 Plugin format (unifies recipes/ + data-research formats)
A plugin is YAML + JS module, discovered via filesystem scan of ~/.gbrain/resolvers/ and recipes/.
# Example: resolvers/x-api/handle-to-tweet.yaml
id: x_handle_to_tweet
version: 1
category: lookup
cost: rate-limited
backend: x-api-v2
module: ./handle-to-tweet.ts
input_schema:
type: object
properties:
handle: { type: string, pattern: "^[A-Za-z0-9_]{1,15}$" }
keywords: { type: string }
required: [handle]
output_schema:
type: object
properties:
url: { type: string, format: uri }
tweet_id: { type: string }
text: { type: string }
created_at: { type: string, format: date-time }
requires:
env: [X_API_BEARER_TOKEN]
health_check:
kind: http
url: https://api.twitter.com/2/tweets/1
expect: { status: [200, 401] } # 401 = auth failure but endpoint reachable
tests:
- input: { handle: "garrytan" }
expect: { url: { pattern: "^https://x\\.com/garrytan/status/\\d+$" } }
Trust flagging follows the existing src/commands/integrations.ts pattern: only package-bundled resolvers are embedded=true and may run arbitrary commands; user-provided resolvers are restricted to http and validated schemas.
3.6 Wraps every resolver with FailImproveLoop
Existing src/core/fail-improve.ts is the deterministic-first/LLM-fallback pattern. Every resolver automatically gets wrapped: if the deterministic path (e.g. X API) returns a valid result, use it; if it fails, optionally fall back to an LLM-based resolver; log both paths for future pattern analysis and auto-test generation.
3.7 Reference implementations to ship
The Wintermute survey inventoried 69 resolver shapes. Shipping all of them is wrong (over-scoped); shipping zero is under-scoped. The dogfood set:
| # | Resolver | Purpose | Used by |
|---|---|---|---|
| 1 | x_handle_to_tweet |
Bare-tweet citation repair (original Phase A) | gbrain integrity |
| 2 | url_reachable |
Dead-link detection | gbrain integrity |
| 3 | brain_slug_lookup |
Name/email → slug (wraps existing resolveSlugs) |
Output Builder |
| 4 | openai_embedding |
Refactor of src/core/embedding.ts into Resolver |
Import pipeline |
| 5 | perplexity_query |
Query → synthesis + citations | Enrichment Orchestrator |
| 6 | text_to_entities |
LLM entity extraction (structured JSON) | Enrichment Orchestrator |
The remaining 63 Wintermute patterns port incrementally, driven by user need. Each port is a new YAML + module under recipes/ or ~/.gbrain/resolvers/ with no framework changes.
4. Layer 2 — Enrichment Orchestrator
4.1 What's broken today
Wintermute's enrichment is polished at the data layer, hacky at the control layer:
- Completeness = "length > 500 chars + no
needs-enrichmenttag" (lib/enrich.mjs:351-355). Naïve. A rich page of repetitive Perplexity summaries (seebrain/people/0interestrates.md— 38 repeating blocks) passes this check. - 30-day auto-re-enrichment runs forever. No "done" state. A person met once in 2023 still gets re-researched monthly.
- Cascade is convention-only. Person→company stubs are created automatically; company→investors, company→employees traversals are documented but never implemented.
- No hard budget cap. Cost is estimated per batch, never enforced across batches or per day.
- Failure is silent. A bad Perplexity response logs and continues; partial writes can leave a page with a timeline entry but no raw-data sidecar.
4.2 The orchestrator
// src/core/enrichment/orchestrator.ts
export interface EnrichmentRequest {
entitySlug: string;
trigger: 'mention' | 'stub-creation' | 'cron-sweep' | 'manual' | 'cascade';
tier?: 1 | 2 | 3; // optional override; auto-computed if absent
cascadeDepth?: number; // 0 = no cascade; default 1
}
export interface EnrichmentResult {
entitySlug: string;
completenessBefore: number;
completenessAfter: number;
resolversUsed: string[]; // e.g. ["perplexity_query", "x_handle_to_tweet"]
costSpent: number;
writtenTo: string[]; // page paths touched, for transaction audit
cascadedTo: string[]; // related entities enriched
status: 'enriched' | 'skipped' | 'failed' | 'budget-exhausted';
reason?: string;
}
export class EnrichmentOrchestrator {
constructor(
private registry: ResolverRegistry,
private writer: BrainWriter,
private budget: BudgetLedger,
private scorer: CompletenessScorer,
private graph: EntityGraph,
) {}
async enrich(req: EnrichmentRequest): Promise<EnrichmentResult>;
async enrichBatch(reqs: EnrichmentRequest[]): Promise<EnrichmentResult[]>;
}
4.3 Evidence-weighted completeness (replaces length heuristic)
Completeness is a per-entity-type rubric, stored in frontmatter on write and recomputed on demand.
// src/core/enrichment/completeness.ts
export interface CompletenessRubric<Page> {
entityType: PageType;
dimensions: {
name: string;
weight: number; // sum must = 1.0
check: (page: Page) => number; // 0.0–1.0
}[];
}
// Example rubric for persons:
// - has_role_and_company 0.20
// - has_source_urls 0.20 (≥1 URL with resolver-verified reachability)
// - has_timeline_entries 0.15 (≥1)
// - has_citations 0.15 (every claim has [Source: ...])
// - has_backlinks 0.10 (every linked page links back)
// - recency_score 0.10 (last_verified within 90 days)
// - non_redundancy 0.10 (no repeated blocks; distinct-lines/total-lines > 0.8)
Key property: non_redundancy + recency_score explicitly kill the two brain pathologies observed in the audit (Wilco-style repeating blocks; stale pages without last_verified).
The completeness field goes in frontmatter as 0.0–1.0. It becomes queryable via list_pages(where: completeness < 0.5).
4.4 Tier routing with hard budget
Two-dimensional routing: importance (tier 1/2/3 from person-score) × budget state.
// src/core/enrichment/tiers.ts
export const TIER_CONFIG = {
1: { models: ['opus', 'sonar-deep'], maxCostUsd: 0.10, cascadeDepth: 2 },
2: { models: ['sonar'], maxCostUsd: 0.02, cascadeDepth: 1 },
3: { models: ['sonar'], maxCostUsd: 0.005, cascadeDepth: 0 },
};
// src/core/enrichment/budget.ts
export class BudgetLedger {
// Hard caps. Queryable pre-resolve.
dailyCapUsd: number;
perEntityCapUsd: number;
perResolverCapUsd: Map<string, number>;
async reserve(resolverId: string, estimateUsd: number): Promise<Reservation | 'exhausted'>;
async commit(reservation: Reservation, actualUsd: number): Promise<void>;
async rollback(reservation: Reservation): Promise<void>;
async state(): Promise<{ spent: number; remaining: number; perResolver: Record<string, number> }>;
}
Property: if the daily cap is reached, orchestrator.enrich() returns status: 'budget-exhausted' immediately. No silent overages. Circuit-breaker resets at midnight in the user's configured TZ.
4.5 Cascade (entity graph traversal)
// src/core/enrichment/cascade.ts
export class EntityGraph {
// Deterministic, no LLM. Uses engine.getLinks() + engine.getBacklinks().
async neighbors(slug: string, depth: number): Promise<string[]>;
async cascadeFrom(trigger: string, depth: number): Promise<EnrichmentRequest[]>;
}
If person X is enriched and gains a new company: Acme field, cascade checks: does companies/acme exist? If not, create stub + enqueue at tier 2. Does companies/acme link back to X? If not, write the back-link. Iron Law is machine-enforced, not skill-enforced.
4.6 Fail-safe transactions
Every enrichment is wrapped in a BrainWriter transaction (Layer 4). Partial writes are rolled back. No asymmetric state like timeline-entry-without-raw-sidecar.
await writer.transaction(async (tx) => {
const research = await registry.resolve('perplexity_query', {...}, ctx);
await tx.appendTimeline(slug, {...});
await tx.putRawData(slug, 'perplexity', research.raw);
await tx.setFrontmatterField(slug, 'completeness', score);
// All-or-nothing commit on exit.
});
5. Layer 3 — Scheduler
5.1 What's broken today
Wintermute's cron is externally-driven JSON (cron/jobs.json) with ~30 jobs manually stagger-offset at different minutes. GBrain has zero native scheduling — src/commands/autopilot.ts is a single daemon loop, and docs/guides/cron-schedule.md is architectural guidance, not code.
Failures observed in Wintermute's actual state:
X OAuth2 Token Refresh: 11 consecutive timeouts (critical-path silent failure)flight-tracker daily scan: 5 consecutive timeoutsmorning-briefing: 4 consecutive timeouts- Quiet hours are checked at runtime in skills, so a skill that forgets to check will DM at 3 a.m.
- Staggering is manual convention; no protection against two jobs colliding after a config edit.
5.2 ScheduledResolver interface
// src/core/scheduling/scheduler.ts
export interface Schedule {
kind: 'cron' | 'interval';
expr?: string; // cron string
intervalMs?: number;
tz: string; // IANA: "America/Los_Angeles"
quietHours?: {
startHour: number; // 22 = 10 PM local
endHour: number; // 7 = 7 AM local
policy: 'skip' | 'defer' | 'silent-run';
};
staggerKey?: string; // jobs with same key auto-offset
maxConcurrent?: number; // global concurrency cap
maxDurationMs?: number; // timeout
}
export interface ScheduledResolver extends Resolver<void, ScheduledResult> {
schedule: Schedule;
retryPolicy: { maxRetries: number; backoffMs: number };
circuitBreaker: { failureThreshold: number; cooldownMs: number };
state: DurableState; // watermark, content-hash, idempotency key
}
5.3 Enforcement vs convention (the key delta from Wintermute)
| Concern | Wintermute today | Knowledge Runtime |
|---|---|---|
| Quiet hours | Checked inside each skill (trust-based) | Enforced at scheduler, skill cannot override |
| Staggering | Manual minute-offset in jobs.json |
Scheduler assigns slots via hashed staggerKey |
| Concurrency | MAX_BATCH_PROCESSES=2 in backoff, ignored by cron |
Global semaphore in scheduler |
| Timeout | Per-job string in JSON, not always respected | Enforced via AbortController, timeout raises TimeoutError caught by orchestrator |
| Retry | None at cron level | retryPolicy with exponential backoff |
| Silent failure | "11 consecutive timeouts" unnoticed | Circuit breaker opens at threshold → escalation to user |
| Idempotency | State files per job, no framework | DurableState primitive: watermark/ID/content-hash |
5.4 Native engine + OS cron adapter
The scheduler runs as either:
- Embedded (default for
gbrain autopilot): native event loop inside the daemon process. One process, many ScheduledResolvers. - OS-driven (for Railway/launchd/systemd):
gbrain schedule run <id>invoked by OS cron, scheduler state is durable so cross-invocation dedup still works.
Both modes share the same Schedule config + state.
5.5 Observability
Every scheduled run emits structured events: started, skipped-quiet-hours, deferred-to-active-hours, failed-retrying, circuit-opened, completed. Events go to:
~/.gbrain/scheduler/events.jsonl(local, always)engine.logIngest(audit trail in brain DB)- Optional webhook (Slack/Telegram for the user)
gbrain doctor reads the event log and reports: current circuit-breaker state, any resolver with > 3 consecutive failures, any resolver that hasn't fired within 3× its interval (freshness SLA like Wintermute's freshness-check.mjs but built-in).
6. Layer 4 — Deterministic Output Builder
6.1 The anti-hallucination invariant
Iron Law: LLM picks WHAT. Code guarantees WHERE and HOW.
Wintermute's existing lib/enrich.mjs:buildTweetEntry is close to this — tweet URLs are built from tweet.id returned by the X API, never from LLM memory. But:
- A past incident: "Sub-agent test #2 FAILED — hallucinated 'Philip Leung' entity links across all daily files. LLM rewriting of daily files is too error-prone." (Wintermute memory log, 2026-04-13.)
- Back-links depend on
appendTimelinebeing called everywhere; skips are silent. - Slug collisions are unchecked (no conflict detection on
slugify). - Citation format is post-hoc linted weekly, not pre-write enforced.
6.2 BrainWriter
// src/core/output/writer.ts
export class BrainWriter {
constructor(
private engine: BrainEngine,
private slugRegistry: SlugRegistry,
private scaffolder: Scaffolder,
) {}
async transaction<T>(fn: (tx: WriteTx) => Promise<T>): Promise<T>;
}
export interface WriteTx {
// High-level typed operations; never raw string writes.
createEntity(input: EntityInput): Promise<string>; // returns slug, conflict-checked
appendTimeline(slug: string, entry: TimelineInput): Promise<void>;
setCompiledTruth(slug: string, body: CompiledTruthInput): Promise<void>;
setFrontmatterField(slug: string, key: string, value: unknown): Promise<void>;
putRawData(slug: string, source: string, data: object): Promise<void>;
addLink(from: string, to: string, context: string): Promise<void>; // auto-creates reverse back-link
// Validators (called implicitly on commit)
validate(): Promise<ValidationReport>;
}
6.3 Scaffolder — deterministic link + citation construction
Every user-visible URL/link/citation is built by code from resolver outputs, not from LLM text.
// src/core/output/scaffold.ts
export class Scaffolder {
tweetCitation(handle: string, tweetId: string, dateISO: string): string {
// "[Source: [X/garrytan, 2026-04-18](https://x.com/garrytan/status/123456)]"
}
emailCitation(account: string, messageId: string, subject: string): string {
// deterministic Gmail URL per Wintermute pattern
}
sourceCitation(resolverResult: ResolverResult<unknown>): string {
// pulls .source, .fetchedAt, .raw from the result
}
entityLink(slug: string): string {
// slugRegistry checks existence; returns resolvable wikilink
}
}
6.4 SlugRegistry — conflict detection
// src/core/output/slug-registry.ts
export class SlugRegistry {
async create(desiredSlug: string, displayName: string, type: PageType): Promise<CreatedSlug>;
// Throws SlugCollision if another entity already occupies desiredSlug and isn't
// confirmed as the same person (via email / x_handle / disambiguator).
// Auto-resolves near-collisions by appending disambiguator.
async confirmSame(slugA: string, slugB: string, confidence: number): Promise<void>;
async merge(canonical: string, duplicate: string): Promise<void>;
}
6.5 Pre-write validators (fail-closed for integrity)
On WriteTx.validate() before commit:
- Citation validator. Every factual sentence in
compiled_truthmust have an inline[Source: ...]within N lines. Non-compliant paragraphs are flagged. Configurable: strict-mode rejects the transaction, lint-mode warns. - Link validator. Every
[text](path)must point to a page that exists OR to a URL the Scaffolder built (so it's guaranteed-valid). No raw LLM-composed URLs. - Back-link validator. Every outbound link must have a reverse link written in the same transaction.
- Triple-HR validator. Compiled truth / timeline split enforced at the schema level.
Fails closed: the default is strict-mode. Loosening requires explicit writer.transaction({ strictMode: false }, ...) and logs a warning to the ingest log.
6.6 LLM output sanitization
Any LLM output destined for a brain page passes through a JSON-Schema-validated parser first. No free-form markdown goes to disk.
- Entity extraction: JSON array of
{ name, type, context }per existingextractEntitiespattern — strict validation. - Compiled-truth synthesis: LLM emits structured
{ sections: [{heading, paragraphs: [{text, sources: [...]}]}]}, scaffolder renders to markdown. - Timeline entries: LLM emits
{ date, summary, detail, sources }, scaffolder renders.
LLM never sees file paths, never writes files, never emits finished markdown.
7. Integration with existing GBrain
7.1 Reuse (already polished)
| Existing | Used by | Change |
|---|---|---|
src/core/fail-improve.ts (9/10) |
Wraps every Resolver in L1 | None; becomes default wrapper |
src/core/backoff.ts (9/10) |
ResolverContext.backoff | None |
src/core/storage.ts (9/10) |
Template for Resolver factory pattern | None; serves as pattern reference |
src/core/check-resolvable.ts (9/10) |
Extend to validate Resolver plugins | Add checkResolvers() mode |
src/commands/publish.ts (9/10) |
Uses BrainWriter under the hood | Minor: route through L4 |
src/commands/backlinks.ts (8/10) |
Folded into L4 validator | Keep as CLI-facing lint entry point |
src/core/operations.ts validators |
Reused in ResolverContext trust enforcement | None |
src/core/engine.ts BrainEngine (35 methods) |
ResolverContext.engine | Extend with getResolverRegistry() |
7.2 Replace (ad-hoc today)
| Existing | Replace with |
|---|---|
src/core/enrichment-service.ts (5/10) |
src/core/enrichment/orchestrator.ts (L2) |
src/core/embedding.ts (monolithic) |
src/core/resolvers/builtin/embedding/openai.ts |
src/core/transcription.ts (monolithic) |
src/core/resolvers/builtin/transcription/{groq,openai}.ts |
src/commands/integrations.ts recipe format |
Unified Resolver plugin format (§3.5) |
src/core/data-research.ts recipe format |
Same unified format |
src/commands/autopilot.ts hard-coded daemon loop |
Wraps a set of ScheduledResolvers |
7.3 Extend
src/core/engine.ts: addgetResolverRegistry(),getWriter(),getScheduler(). Engine becomes the runtime's root container.src/core/operations.ts:OperationContextinherits fromResolverContext(or vice-versa). Trust flags unified.src/core/types.ts: addcompleteness: numbertoPage,sourcedBy: string[]for provenance.
8. Migration Path (phased, shippable)
Each phase ships independently, passes full E2E, is feature-flagged, and is reversible. No big-bang.
Phase 0 — Foundation (human: ~1 wk / CC: ~4 h)
- Define
Resolver<I,O>,ResolverContext,ResolverRegistry,ResolverResult(§3.2–3.4). - Add
src/core/resolvers/index.tswiring + tests for registry (register/get/list). - No behavioral change; ship as
v0.11.0-alphawith feature flag.
Phase 1 — Three reference resolvers (human: ~1 wk / CC: ~4 h)
- Port
src/core/embedding.ts→resolvers/builtin/embedding/openai.ts. - Implement
resolvers/builtin/brain-local/slug-lookup.ts(wrapsengine.resolveSlugs). - Implement
resolvers/builtin/url-reachable.ts(HEAD-check). - Prove the interface: old callers swap to
registry.resolve('openai_embedding', ...).
Phase 2 — BrainWriter + Slug Registry (human: ~1.5 wk / CC: ~6 h)
- L4 core:
BrainWriter.transaction,Scaffolder,SlugRegistrywith conflict detection. - Pre-write validators: citation, link, back-link, triple-HR.
- Migrate
src/commands/publish.ts+src/commands/backlinks.tsto route through BrainWriter. - Now Wintermute's "Philip Leung" hallucination is structurally impossible — LLM output passes through JSON-Schema validator before reaching Scaffolder.
Phase 3 — gbrain integrity command (human: ~0.5 wk / CC: ~2 h)
- Ship the originally-scoped user-facing feature on top of the new foundation.
- Uses Resolver SDK:
x_handle_to_tweet+url_reachable. - Uses BrainWriter: all auto-repairs go through validated writes.
--auto --confidence 0.8mode as user approved in cherry-pick #1.- User-visible value ships in Phase 3, not Phase 7.
Phase 4 — Enrichment Orchestrator (human: ~2 wk / CC: ~8 h)
- L2 core:
EnrichmentOrchestrator,BudgetLedger,CompletenessScorer,EntityGraph.cascadeFrom. - Migrate
src/core/enrichment-service.tscallers (deprecate the old file after). - Completeness score in frontmatter on every write (dogfooding cascades).
Phase 5 — Scheduler (human: ~2 wk / CC: ~8 h)
- L3 core:
Scheduler,ScheduledResolver,DurableState, circuit breaker, quiet-hours enforcer. - Migrate
src/commands/autopilot.tsto a ScheduledResolver set. - Ship
gbrain schedule list|run|pause|tailCLI for observability.
Phase 6 — Port 5–8 Wintermute resolvers (human: ~1.5 wk / CC: ~6 h)
perplexity_query,text_to_entities,mistral_ocr_pdf,x_search_all,x_user_to_tweets,gmail_query_to_threads,calendar_date_to_events.- Each ships as YAML + TS module under
resolvers/builtin/— proof of the plugin format.
Phase 7 — Wintermute Claw Adoption Integration (human: ~1 wk / CC: ~4 h)
- Write
docs/wintermute/ADOPTION.mdshowing Wintermute how to replace its 69 bespoke scripts with calls togbrain registry.resolve(...). - Ship a
gbrain claw-bridgesubcommand that proxies Wintermute's current script invocations to the resolver registry — zero-edit adoption path. - This is the test of the north star. If Wintermute can stand up a 1-line shim and drop
scripts/x-api-client.mjs, the abstraction succeeded.
Total: human: ~10 weeks / CC: ~42 hours / calendar with single implementer: ~3–4 weeks.
9. Critical Files
New directories / files
src/core/
runtime/
index.ts # RuntimeContext (engine, storage, config, logger, metrics, budget)
registry.ts # ResolverRegistry
factory.ts # createResolver()
resolvers/
interface.ts # Resolver<I, O>
fail-improve-wrapper.ts # auto-wraps every resolver in FailImproveLoop
builtin/
x-api/
handle-to-tweet.ts
handle-to-tweet.yaml
perplexity/
query.ts
query.yaml
brain-local/
slug-lookup.ts
url-reachable.ts
embedding/
openai.ts # refactored from src/core/embedding.ts
transcription/
groq.ts
openai.ts
enrichment/
orchestrator.ts # EnrichmentOrchestrator
tiers.ts # TIER_CONFIG
budget.ts # BudgetLedger
completeness.ts # CompletenessScorer + per-type rubrics
cascade.ts # EntityGraph
scheduling/
scheduler.ts # Scheduler + ScheduledResolver
schedule.ts # Schedule type, cron expr parser
state.ts # DurableState primitives
quiet-hours.ts # TZ-aware enforcement
stagger.ts # deterministic slot assignment
output/
writer.ts # BrainWriter
scaffold.ts # Scaffolder (typed URL builders)
slug-registry.ts # SlugRegistry (conflict detection)
validators/
citation.ts
link.ts
back-link.ts
triple-hr.ts
src/commands/
integrity.ts # ships in Phase 3, replaces Feynman Phase A/B
schedule.ts # gbrain schedule list|run|pause|tail (Phase 5)
docs/wintermute/
ADOPTION.md # written in Phase 7
Replaced / removed
src/core/enrichment-service.ts— folded intoenrichment/orchestrator.tssrc/core/embedding.ts— moved intoresolvers/builtin/embedding/openai.tssrc/core/transcription.ts— moved intoresolvers/builtin/transcription/
Extended
src/core/engine.ts— addgetResolverRegistry(),getWriter(),getScheduler()src/core/operations.ts— unify with ResolverContext; every operation validator reusable by resolverssrc/core/types.ts— addcompleteness: number,sourcedBy: string[],lastVerified: Date
10. Testing Strategy
Contract tests
Every Resolver implementation tested against the interface spec. Table-driven: run the same suite against openai_embedding, x_handle_to_tweet, etc. Ensures plugin authors can't ship broken resolvers.
Property tests
- Idempotency: running a ScheduledResolver twice with the same state produces the same output and doesn't double-write.
- Atomicity: a BrainWriter transaction that throws mid-flight leaves the brain bit-for-bit identical to pre-transaction.
- Deterministic scaffolds: given the same resolver outputs, the Scaffolder produces byte-identical citations/links.
Integration tests
EnrichmentOrchestratorend-to-end against PGLite (in-memory, no API keys) with mocked resolver registry.Schedulerwith fake clock + quiet-hours scenarios.- BrainWriter transaction rollback on validator failure.
Chaos tests
- Kill the process mid-enrichment; next run must resume cleanly.
- Simulate API timeout mid-transaction; transaction must roll back completely.
- Corrupted state file; scheduler must escalate, not silently skip.
Regression tests vs. Wintermute behavior
For each Wintermute pattern we port (e.g. X-handle → tweet URL), a regression test proves the new resolver produces the same answer on real-world inputs from the brain audit. This is the "Wintermute would adopt" proof.
11. Open Questions (flagged for CEO re-review)
- Scope shape. Is this the right four-layer decomposition, or are some layers better left to Wintermute (e.g. Scheduling lives above GBrain, not in it)?
- Phase 3 user-value break. Does Phase 3 (user-visible
gbrain integrity) ship early enough, or do we need an even smaller MVP? - LLM-as-resolver. Should
text_to_entitiesbe a Resolver, or does that blur the "code vs LLM" line the invariant relies on? - Plugin format. YAML + TS module (§3.5) vs. pure TS module with decorator-style metadata. Latter is more type-safe; former is more discoverable.
- Cross-resolver transactions. Do we support "atomic fetch-from-Perplexity + write-to-brain" at the L2 layer? Current design says yes; implementation is tricky (Perplexity call isn't rollbackable).
- Wintermute bridge scope. Phase 7
gbrain claw-bridge— is that worth a phase of its own, or should adoption be documentation-only? - Completeness rubric coverage. Do we define rubrics for all 9 PageTypes upfront, or ship people/company/meeting first and extend incrementally?
- Budget config UX. Hard daily cap is strict; should we also expose a soft-cap warning mode, and how is the cap set (env var? config file? prompt on first use?)
- Backwards compat.
src/commands/publish.tsandsrc/commands/backlinks.tshave been running cleanly for weeks. Refactoring through BrainWriter carries migration risk. Acceptable? - Existing TODOS alignment.
TODOS.mdhas P0 "Runtime MCP access control" and P2 security hardening. The new RuntimeContext.remote flag interacts with both — do we fold MCP access control into Phase 0 or keep separate?
12. Verification (the "Wintermute would adopt" test)
The design succeeds iff:
- A user can add a new resolver by dropping a YAML + TS module in
~/.gbrain/resolvers/without editing GBrain source. - Wintermute can delete
scripts/x-api-client.mjsand replace all callers with 1-lineawait registry.resolve('x_handle_to_tweet', ...). - No brain page can be written with a bare tweet reference, a missing back-link, or an unverified URL (validators catch it pre-commit).
- Running
gbrain integrity --auto --confidence 0.8over a real brain fixes ≥1,000 of the 1,424 known bare-tweet citations without human review. - Full E2E test suite passes on both PGLite + Postgres engines.
- The Knowledge Runtime ships across 7 phases with each phase individually shippable and reversible.