mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 11:22:34 +00:00
b44a31cc0defe418b2ea9ec2d4cfcf5658d9c8a9
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
c89aa909c7 |
feat: Knowledge Runtime — Resolver SDK + BrainWriter + integrity + Budget + scheduler polish (v0.13.0) (#210)
* docs: Knowledge Runtime design doc (draft) — 4-layer architecture + reduced-scope delta
Captures the Knowledge Runtime design thinking from the CEO review session:
Resolver SDK, Enrichment Orchestrator, Scheduler, Deterministic Output Builder.
The original 7-phase plan was drafted before v0.12.0 (knowledge graph layer)
and v0.11.x (Minions agent runtime) shipped. Cross-referenced against what's
already merged on master, roughly 60% of the 4-layer vision is already in
production under different names:
- Minions = scheduler + plugin contract (L1 + L3)
- Knowledge graph auto-link = deterministic output at L4 + orchestrator at L2
- BrainBench v1 benchmarks already validate the graph layer
The doc is kept as a draft design reference; the actual build-out will scope
down to the real delta (typed Resolver interface, BrainWriter API + validators,
BudgetLedger, CompletenessScorer, quiet-hours + stagger). See the CEO review
notes for the reduced plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolvers): Resolver SDK pass 1 — interface + registry (PR 1/5)
Adds the typed plugin interface that unifies external-lookup calls (X API,
Perplexity, HEAD check, brain-local slug resolution) behind a single shape:
registry.resolve('x_handle_to_tweet', { handle, keywords }, ctx)
→ { value, confidence, source, fetchedAt, raw? }
Zero behavior change — the registry is empty by default. Builtins
(url_reachable, x_handle_to_tweet) land in the next pass. ScheduledResolver
wrapping via Minions lands in PR 5.
New files:
- src/core/resolvers/interface.ts — Resolver<I,O>, ResolverResult<O>,
ResolverContext (engine, storage, config, logger, requestId, remote,
deadline, signal), ResolverError (not_found, already_registered,
unavailable, timeout, rate_limited, auth, schema, aborted, upstream)
- src/core/resolvers/registry.ts — ResolverRegistry (register/get/has/
list/resolve/clear/size) + getDefaultRegistry() for process-wide use
- src/core/resolvers/index.ts — barrel export
Design rules enforced by types:
- Every result carries confidence (0.0-1.0) + source attribution
- LLM-backed resolvers return confidence<1.0 by convention
- ctx.remote propagates the trust boundary (mirrors OperationContext.remote)
- AbortSignal threads through for cooperative cancellation
Smoke: imports + runs, list()/get()/resolve() behave as typed.
Dependency-free beyond types and storage/engine type imports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(fail-improve): optional AbortSignal — Resolver SDK pass 2 (PR 1/5)
Extends FailImproveLoop.execute with an optional `opts.signal` that threads
through the deterministic-first / LLM-fallback flow. Needed by the Resolver
SDK so long-running lookups can be cooperatively cancelled when a caller
aborts (deadline hit, Minion job timeout, user ctrl-c).
Additive and backwards-compatible:
- execute() signature widens callbacks to (input, signal?) => ...; existing
two-arg callbacks are structurally compatible and ignore the extra arg.
- opts is optional; callers that omit it get pre-extension behavior.
- Aborts throw a DOM-style AbortError (name='AbortError'), matching what
fetch() throws, so downstream `err.name === 'AbortError'` branches work
unchanged.
- Aborted runs are NOT logged to the failure JSONL — not informative and
would pollute pattern analysis.
Abort check fires in three places:
- Before the deterministic call (pre-flight)
- Between deterministic miss and LLM call (mid-flight)
- Inside llmFallbackFn if the implementation respects signal itself
Smoke tests: 5 scenarios (existing sig, llm fallback, pre-abort, mid-flight
abort, signal threaded to fallback) — all pass. Existing test/fail-improve.test.ts
(13 tests, 27 expects) unchanged and passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolvers): url_reachable + x_handle_to_tweet — SDK pass 3 (PR 1/5)
Two reference resolver implementations that validate the interface against
real-world requirements: a deterministic free-cost check and a rate-limited
paid-backend lookup.
src/core/resolvers/builtin/url-reachable.ts
HEAD-check a URL, follow redirects (max 5), detect dead links. Reused
isInternalUrl() from the wave-3 SSRF hardening; re-validates every redirect
hop against the same filter. Falls back from HEAD to GET on 405/501.
Composes caller's AbortSignal with a per-request timeout via
AbortSignal.any (with manual-propagation fallback). Confidence=1 when the
backend answers; confidence=0 only on transport failure (DNS/connect/timeout).
src/core/resolvers/builtin/x-api/handle-to-tweet.ts
Find a tweet by handle + free-text keyword hint. Used by the upcoming
`gbrain integrity --auto` loop to repair the 1,424 bare-tweet citations
in Garry's brain. Confidence buckets align with the three-bucket contract:
- >=0.8 auto-repair (single strong match, or dominant in small candidate set)
- 0.5-0.8 review queue (ambiguous but promising)
- <0.5 skip (many candidates or weak match)
Scoring: normalized keyword-token overlap against tweet text, with margin
boost for dominant matches. Strict handle regex (X's username rules).
Retries on 429 up to 2x with Retry-After honor. Terminal 401/403 surfaces
as auth ResolverError so the caller stops hammering. Bearer token read
from ctx.config.x_api_bearer_token or X_API_BEARER_TOKEN env — never logged.
Smoke: registry accepts both, SSRF blocks localhost + file://, available()
returns false when token missing, schema validator rejects bad handles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(resolvers): tests + gbrain resolvers CLI — SDK pass 4 (PR 1/5 complete)
Closes out PR 1. 43 new tests in test/resolvers.test.ts covering registry
contract, both reference builtins, all three confidence buckets, and every
ResolverError subcode.
test/resolvers.test.ts
- ResolverRegistry: register, duplicate-id rejection, get/has, list with
cost+backend filters, resolve, unavailable propagation, clear, default
singleton lifecycle.
- url_reachable: available(), SSRF guard on localhost + RFC1918 + 169.254
metadata + file:// scheme, empty-url schema error, 200/404 status
propagation, HEAD→GET fallback on 405, redirect chain, per-hop SSRF
re-validation, network failure → reachable=false, AbortSignal mid-flight.
- x_handle_to_tweet: token gate via env AND via ctx.config, invalid/long
handle schema errors, zero-candidate + single-strong + single-weak +
many-ambiguous confidence buckets (gates >=0.5 url emission), 401/403
auth error, 500 upstream error, 429 retry-then-rate_limited, X operator
stripping (prompt injection defense).
src/commands/resolvers.ts
- `gbrain resolvers list [--cost | --backend | --json]` pretty table
or JSON.
- `gbrain resolvers describe <id>` schema + availability detail.
- registerBuiltinResolvers() is idempotent; ready to be called from
future entry points (gbrain integrity, MCP server).
src/cli.ts wires `resolvers` into CLI_ONLY + dispatches to runResolvers.
Full suite: 1343 pass / 0 fail / 141 skip (E2E without DATABASE_URL).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(output): BrainWriter + Scaffolder + SlugRegistry — PR 2 pass 1/4
Lands the transactional writer library that the rest of the Knowledge
Runtime sits on top of. No callers routed through it yet — publish.ts /
backlinks.ts / put_page migrations are pass 4 and PR 2.5.
src/core/output/scaffold.ts
Deterministic URL / citation / link builders. Callers pass typed inputs
(handle + tweetId, account + messageId, slug + display text) and get
canonical markdown bytes out. LLM-generated URLs never touch disk.
- tweetCitation({handle, tweetId, dateISO?})
- emailCitation({account, messageId, subject, dateISO?})
- sourceCitation(resolverResult, {url?, label?})
- entityLink({slug, displayText, relativePrefix?})
- timelineLine({dateISO, summary, citation?})
ScaffoldError with codes for invalid_handle / invalid_tweet_id /
invalid_slug / invalid_message_id / invalid_date / empty.
src/core/output/slug-registry.ts
Solves the "Marc Benioff vs Marc-Benioff both slug to marc-benioff" bug.
create() probes engine.getPage and either returns the desired slug or
disambiguates (alice-smith → alice-smith-2). isFree() + suggestDisambiguators()
for interactive UX. Errors: collision, disambiguator_exhausted, invalid_slug.
src/core/output/writer.ts
BrainWriter.transaction(fn, ctx) wraps engine.transaction. The `fn`
callback receives a WriteTx with createEntity / appendTimeline /
setCompiledTruth / setFrontmatterField / putRawData / addLink (the last
creates both forward + reverse back-link atomically). On commit, per-page
validators run against all touchedSlugs. Strict mode throws on
error-severity findings, rolling back the outer tx. Lint mode (default for
PR 2 rollout) returns the report but commits regardless. Pages with
`validate: false` frontmatter skip validators entirely (grandfather hook
for PR 2 migration).
Integration smoke against PGLite: createEntity → disambiguator (2nd call
with same desired slug), addLink writes both forward + back-link,
strict-mode validator failure rolls back the transaction bit-identically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(output): 4 pre-commit validators + tests — PR 2 pass 2/4
Lands the validator suite that BrainWriter runs before committing a
transaction. Paragraph-level deterministic checks, markdown-aware, skip
legacy pages via validate:false frontmatter.
src/core/output/validators/citation.ts
Every factual paragraph in compiled_truth carries at least one citation
marker: [Source: ...] or a linked URL. Splits paragraphs on blank lines,
strips fenced code / inline code / HTML comments before checking.
Ignores headings, key-value lines ("**Status:** Active"), table rows,
pure wikilink bullets (## See Also), and short labels without a factual
verb. Deterministic — no LLM, no semantic judgment.
src/core/output/validators/link.ts
Every [text](path) wikilink resolves to a page that exists (unless it's
an external http(s) URL, which this validator doesn't check; that's
url_reachable's job in PR 3). Strips relative prefix and .md extension.
Batches engine.getPage lookups per unique target. mailto/anchor/other
schemes flagged as warning. Links inside fenced code blocks are skipped.
src/core/output/validators/back-link.ts
Iron Law: if page X → page Y, then Y → X. Reads engine.getLinks(ctx.slug),
and for each target checks engine.getLinks(target) for a reverse edge.
Missing reverses flagged as warning (runAutoLink is the authoritative
enforcer on put_page; this is defense-in-depth for pages edited outside
the main write path).
src/core/output/validators/triple-hr.ts
Catches hygiene issues on the compiled_truth / timeline split: bare `---`
in compiled_truth would re-split on round-trip through parseMarkdown;
headings in the timeline section signal authoring mistakes. Both warn
(not error) — legacy pages legitimately use thematic breaks.
src/core/output/validators/index.ts
registerBuiltinValidators(writer) wires all four.
test/writer.test.ts
57 tests: Scaffolder (all 5 helpers + error paths), SlugRegistry (create,
disambiguator, collision throw, invalid-slug, isFree, suggestDisambiguators),
BrainWriter (happy path, disambiguate, addLink + reverse, strict rollback,
lint proceeds with report, off skips validators, validate:false grandfather,
setCompiledTruth, setFrontmatterField merge, registered validators list),
citation validator (all 11 shape cases), link validator (normalizeToSlug
including ../../, external URL skip, mailto warning, code-fence skip),
back-link validator (no outbound, missing reverse → warning, bidirectional
clean), triple-hr validator (clean, bare --- warning, fenced --- skipped,
heading in timeline warning, ## Timeline header allowed).
Full suite: 1400 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(migrations): v0.13.0 grandfather validate:false — PR 2 pass 3/4
Adds the TS migration that makes BrainWriter's strict-mode rollout safe:
every existing page gets `validate: false` in frontmatter so the new
citation / link / back-link / triple-HR validators skip legacy content.
gbrain integrity --auto (PR 3) clears the flag per-page once real citations
are repaired.
src/commands/migrations/v0_13_0_add_validate_false.ts
Four-phase orchestrator following the v0_12_0 pattern:
A. connect — loadConfig + createEngine. Does NOT write config (prior
learning: gbrain init --migrate-only semantics; never
flip Postgres users to PGLite via bare init).
B. snapshot — engine.getAllSlugs() upfront (prior learning:
listpages-pagination-mutation; OFFSET iteration is
self-invalidating when each write bumps updated_at).
C. grandfather — per slug, skip if frontmatter.validate already set,
else append-log pre-mutation snapshot to
~/.gbrain/migrations/v0_13_0-rollback.jsonl and
putPage with validate:false merged in. Batched 100
at a time so interruption losses are bounded.
D. verify — SQL count of pages with validate=false ≥ expectedTouched.
Idempotent: second run is a no-op. Reversible: rollback log is
append-only JSONL; future `gbrain apply-migrations --rollback v0.13.0`
replays it. Safe on empty brains (returns complete with 0 touched).
src/commands/migrations/index.ts
Registers v0_13_0 after v0_12_0 in semver order.
test/migrations-v0_13_0.test.ts
Registry integration (v0.13.0 present, semver-after-v0.12.0, pitch
metadata well-formed), orchestrator handles no-config gracefully,
dryRun skips the connect phase.
test/apply-migrations.test.ts
Updated two assertions that hard-coded the v0.12.0 skippedFuture list
to also include v0.13.0 (now skippedFuture when installed < 0.13.0).
Full suite: 1405 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(integrity): gbrain integrity — bare-tweet repair + dead-link scan (PR 3)
Ships the user-visible milestone for the Knowledge Runtime delta: a
command that finds brain-integrity issues and repairs them through the
BrainWriter + Resolver SDK infrastructure from PRs 1 and 2.
Targets the two quantified pain points from brain/CITATIONS.md:
- 1,424 of 3,115 people pages have bare tweet references without URLs
- An unknown fraction of existing URL citations have rotted
Subcommands:
gbrain integrity check Read-only report, optional --json
gbrain integrity auto Three-bucket repair loop
gbrain integrity review Print review-queue path + count
gbrain integrity reset-progress Clear the progress file
Three-bucket contract (matches x_handle_to_tweet resolver's confidence
scoring):
>=0.8 → auto-repair via BrainWriter transaction. Appends a timeline
entry on the page with a Scaffolder-built tweet citation (URL
from the API response, never from LLM text).
0.5-0.8 → append to ~/.gbrain/integrity-review.md with all candidates
sorted by match score, for batch human review.
<0.5 → log reason to ~/.gbrain/integrity.log.jsonl and skip.
Resumable: every processed slug hits ~/.gbrain/integrity-progress.jsonl
so an interrupted run resumes from the last slug. --fresh clears it.
Bare-tweet detection patterns (regex, deterministic, skip code fences
and already-cited lines):
- "tweeted about"
- "in/on a (recent|viral) tweet"
- "wrote a tweet/post"
- "posted on X"
- "via X" (but not "via X/handle" — already cited)
- possessive "his/her/their tweet"
External-link detection extracts all [text](https?://...) pairs (code
fences skipped) for optional dead-link probing via url_reachable.
Dead links are surfaced, not auto-repaired — no "correct" replacement
exists without human judgment.
Wiring: runIntegrity dispatches subcommands, registers builtin resolvers
into the default registry, connects to the brain engine, and uses
BrainWriter in strict-off mode (integrity is the repair path, not the
write-gate path).
Unit tests: 21 cover bare-tweet regex (all 9 phrase shapes + code-fence
skip + URL-already-present skip + per-line dedup), external-link
extraction (http+https, line numbers, fenced skip), frontmatter handle
extraction (x_handle, twitter, twitter_handle, x; preference order;
leading @ strip; null paths). End-to-end auto flow verified manually
via the resolver SDK tests + BrainWriter tests it composes.
src/cli.ts wires `integrity` into CLI_ONLY + dispatches to runIntegrity.
Full suite: 1426 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(enrichment): BudgetLedger + CompletenessScorer — PR 4
Two layer-2 primitives that slot under the resolver SDK and BrainWriter:
cost-aware spend caps and evidence-weighted per-page completeness scoring.
Schema migration v11 adds two tables:
budget_ledger (scope, resolver_id, local_date) PK — midnight rollover by
date column means a new calendar day upserts a new row; no rollover
thread, no race.
budget_reservations (reservation_id) — TTL-bounded held reservations
(default 60s) so process death between reserve() and commit() doesn't
strand money.
Rollback plan: DROP TABLE. Budget data is regenerable from resolver call
logs; no durable product value lives in the ledger.
src/core/enrichment/budget.ts
BudgetLedger.reserve({resolverId, estimateUsd, capUsd?, ttlSeconds?})
serializes concurrent reserves on {scope, resolver_id, local_date} via
SELECT ... FOR UPDATE. Returns {kind:'held', reservationId, ...} or
{kind:'exhausted', reason, spent, pending, cap} — never over-spends.
commit(id, actualUsd) moves money from reserved_usd to committed_usd and
marks the reservation status='committed'. rollback(id) zeros out the
reservation without touching committed. Commit-after-commit throws
already_finalized; rollback-after-commit is a no-op (callers don't need
to guard). commit-unknown-id throws reservation_not_found.
cleanupExpired() sweeps held reservations past expires_at and rolls them
back; reserve() opportunistically reclaims the target row's expired
reservations before acquiring its own lock.
IANA timezone config via opts.tz (default America/Los_Angeles); midnight
rollover is naturally expressed as a date column + Intl.DateTimeFormat
with en-CA locale (YYYY-MM-DD). DST is handled by the formatter.
src/core/enrichment/completeness.ts
Seven per-type rubrics (person, company, project, deal, concept, source,
media) + default. Each rubric's dimension weights sum to 1.0, checked at
module load. scorePage(page) returns {score, dimensionScores, rubric}
where score is 0.000–1.000.
Person rubric dimensions: has_role_and_company, has_source_urls,
has_timeline_entries, has_citations, has_backlinks, recency_score,
non_redundancy. The last two are the explicit fix for the two pathologies
called out in the codex review of the earlier design: stale pages that
never decay (30-day re-enrich forever) and Wilco-style repeated blocks
that pass Wintermute's length heuristic.
Pure functions. No engine calls — BrainWriter invokes scorePage after a
transaction and caches the result in frontmatter.completeness.
test/enrichment.test.ts — 23 tests:
BudgetLedger: under-cap held, over-cap exhausted, commit moves money,
rollback clears, commit-rollback no-op, commit-commit throws, commit-
unknown throws, invalid input, empty state null, scope isolation,
parallel reserves respect cap (10 parallel, cap 1.0, est 0.3 each →
≤ 3 held; state.reservedUsd ≤ 1.0), cleanupExpired reclaims TTL=0.
CompletenessScorer: all 8 rubrics sum to 1.0, empty person scores <0.3,
fully-enriched person >0.8, dimension scores exposed, role detection,
company/concept/source/media/default routing, recency decay with age,
non_redundancy penalizes repeated lines.
Full suite: 1449 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): quiet-hours + stagger + claim-time gate — PR 5
Closes the scheduler gap per CEO plan: Minions v7 shipped a durable
runtime but nothing about when jobs should NOT run. This wires
quiet-hours enforcement at claim time (the codex correction — dispatch-
time is wrong because a queued job can become claimable after its window
opens) plus deterministic stagger slots to prevent cron-boundary storms.
Schema migration v12 adds two columns to minion_jobs:
quiet_hours JSONB — {start, end, tz, policy} window config
stagger_key TEXT — partitioning key for deterministic offset
Plus a partial index on stagger_key for later slot-assignment queries.
src/core/minions/quiet-hours.ts
evaluateQuietHours(cfg, now?) → 'allow' | 'skip' | 'defer'. Pure,
deterministic, no engine. Handles straight-line and wrap-around windows
(e.g. 22→7 spans midnight). IANA timezone via Intl.DateTimeFormat;
unknown tz fails open (allow) — safer than hard-blocking every job.
'skip' policy drops the event; 'defer' (default) re-queues for later.
src/core/minions/stagger.ts
staggerMinuteOffset(key) → 0–59, FNV-1a hash. Same key → same slot.
Pure; no module-level state. Used by scheduled resolvers that want to
avoid cron-boundary collisions ("10 jobs all fire at minute 0").
src/core/minions/worker.ts
MinionWorker.tick now consults evaluateQuietHours on every claimed job.
Verdict 'defer' → UPDATE status='delayed', delay_until = now() + 15m
(prevents immediate re-claim loops when the claim query re-runs).
Verdict 'skip' → UPDATE status='cancelled', error_text='skipped_quiet_hours'.
Both paths clear lock_token and require lock_token match in the WHERE
clause so a concurrent stall recovery can't race us.
test/minions-quiet-hours.test.ts — 25 tests:
evaluateQuietHours: null/undefined/invalid config paths (allow fail-open),
straight-line in/out + exclusive-end, wrap-around in (before midnight +
after), skip vs defer policy, timezone-offset propagation (winter PST
vs summer PDT), localHour parity with Date.getUTCHours.
staggerMinuteOffset: deterministic same key → same offset, different
keys spread across buckets (10 keys → ≥5 unique buckets), empty/non-
string edge cases.
Schema v12: quiet_hours and stagger_key columns exist on minion_jobs,
idx_minion_jobs_stagger_key index present.
Full suite: 1474 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(output): post-write validator lint hook — PR 2.5
Minimal integration of BrainWriter validators into the main write path,
feature-flag-gated and non-blocking. The CEO plan explicitly scoped PR 2.5
as a pre-soak landing step: the hook plugs in now, observability lands,
but strict-mode rejection is deferred to a follow-on release gated on the
7-day soak + BrainBench regression ≤1pt.
src/core/output/post-write.ts
runPostWriteLint(engine, slug, opts?) invokes the four BrainWriter
validators (citation, link, back-link, triple-hr) against a freshly
written page and returns a PostWriteLintResult. Skips cleanly when:
- config `writer.lint_on_put_page` is not truthy (default OFF; opts.force overrides)
- the page is not found (shouldn't happen in normal put_page flow)
- the page has frontmatter.validate === false (grandfathered)
Findings are logged to:
- ~/.gbrain/validator-lint.jsonl (capped at 20 findings per line)
- engine.logIngest (ingest_log table) for durable agent-inspectable history
Validator-level exceptions are swallowed so a buggy validator never
breaks put_page.
src/core/operations.ts put_page handler
After importFromContent + runAutoLink, imports runPostWriteLint and
invokes it. Result returns writer_lint: {error_count, warning_count} or
{skipped: reason}. Try/catch wraps the whole hook so an import or
runtime error never blocks the main write.
Enable locally:
gbrain config set writer.lint_on_put_page true
Then every put_page emits a writer_lint summary + appends structured
findings to the ingest log for analysis before the strict-mode flip.
test/post-write-lint.test.ts — 11 tests:
Flag reader (default off, true/1/on, other values false, explicit false)
Hook behavior (flag-off skip, page-not-found skip, validate:false
grandfather skip, force=true overrides flag, dirty page yields citation
error, clean page yields zero findings).
Full suite: 1485 pass / 0 fail / 141 skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(migrations-v0_13_0): drop flaky no-config assertion
The 'does not succeed when no brain is configured' test assumed loadConfig
would return null when HOME is empty, but it also reads DATABASE_URL from
the environment. When .env.testing sources DATABASE_URL into the shell
(normal E2E lifecycle), the orchestrator connects successfully and runs
to completion — the test's assertion was unreachable.
The dry-run path is still covered by the remaining test in the same
describe block; registry integration and semver ordering are covered by
the sibling describe.
Full suite with DATABASE_URL live: 1574 pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(minions): wire quiet_hours + stagger_key into MinionJobInput + queue.add
Codex adversarial review caught that PR 5 (claim-time quiet-hours gate) was
cosmetic: the schema v12 column existed, the worker read it via
`readQuietHoursConfig(job)`, but `MinionJobInput` never accepted it,
`queue.add()` never inserted it, and `rowToMinionJob()` never mapped it out.
Result: every scheduled job saw `quiet_hours: null`, so the gate was a
no-op. Stagger_key had the same broken wiring.
- MinionJob (types.ts): add `quiet_hours` and `stagger_key` fields.
- MinionJobInput: add matching optional fields so callers can submit them.
- rowToMinionJob: parse both columns (JSONB handled the same way as `data`).
- MinionQueue.add: include both columns in the INSERT (idempotent + normal
paths), bound as $19/$20. The `$19::jsonb` cast matches the JSONB column
shape; the wire format is the same native-JS object path that fixed the
JSONB double-encode bug in v0.12.1.
After this, `await queue.add('x', {}, { quiet_hours: {start:22,end:7,
tz:"America/Los_Angeles",policy:"defer"} })` actually stores the window
and the worker's claim-time gate defers the job inside it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(minions): route quiet-hours 'skip' through cancelJob to rollup parents
Codex flagged that handleQuietHoursDefer with verdict='skip' directly set
status='cancelled' via raw UPDATE — bypassing MinionQueue.cancelJob, which
means:
- Parent jobs in 'waiting-children' never get rolled up.
- Descendant jobs don't cascade-cancel.
- Child-done inbox notification is skipped.
Result: a parent waiting on a child that fell inside quiet hours with
policy='skip' stays stuck forever.
Fix: release the lock, then delegate to queue.cancelJob(job.id) which
handles the recursive CTE + parent rollup + inbox posting correctly.
Falls back to a direct UPDATE only if cancelJob errors — even then, the
status transition is status-guarded to avoid stomping terminal states.
Defer path unchanged (no parent rollup needed since the job hasn't reached
a terminal state).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(budget): commit() re-checks cap + rejects negative actuals
Codex caught two cap-bypass bugs in BudgetLedger.commit():
1. reserve({estimateUsd: 0.01, capUsd: 1.0}) + commit(id, 100) silently
charged $100 to a $1-cap bucket. Cap is an advertised invariant that
the code was not enforcing.
2. Negative actuals (commit(id, -5)) were accepted, letting callers
artificially reduce committed_usd below the real spend. Refunds need
a dedicated API, not a side-channel on commit.
Fix:
- Reject non-finite AND negative actualUsd at entrypoint.
- Lock the ledger row FOR UPDATE during commit (same serialization as
reserve).
- Compute effective cap headroom = cap - other_committed - other_reserved
(excluding this reservation from the reserved pool since we're about to
finalize it).
- When actualUsd would exceed available, clamp committed_usd to max
available and throw BudgetError with the overage reported. The
reservation is still marked 'committed' (API call already happened;
don't retry-loop), but the cap is honored.
After this, a $1/day cap actually means $1/day.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(integrity): --dry-run no longer writes progress, poisoning resume
Codex caught that 'gbrain integrity auto --dry-run' appended progress
entries (status='repaired', 'reviewed', 'skipped', 'error') despite doing
no actual writes. The follow-on real run with default --resume would then
skip those slugs — the dry-run silently consumed the work queue.
Fix: gate every appendProgress() call in cmdAuto on !dryRun. Dry-run
still logs to the skip log / review queue (so the user sees what WOULD
happen), but the progress file stays untouched.
Behavior:
--dry-run → buckets counted + summary printed + review-queue
+ log populated, but progress file unchanged.
(default) → progress file tracks every processed slug, so
Ctrl-C + re-run resumes from the right place.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.13.0.0)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(resolvers): DNS-rebinding defense + X rate-limit header parity
Two non-blocking codex findings on PR #210 rolled into one bisectable
commit because their tests share an import line.
url_reachable: hostname-string SSRF guard is vulnerable to DNS rebinding
(attacker-controlled DNS returns a public IP at validate time and
169.254.169.254 at fetch time). Add checkDnsRebinding() that resolves
the hostname via dns.lookup({all:true}) and rejects any result whose
A/AAAA record lands in a private range (v4 via isPrivateIpv4, v6
loopback/link-local/unique-local/IPv4-mapped). Applied on the initial
URL and on every redirect target. Null on DNS failure so genuine
network problems surface via fetch.
x_handle_to_tweet: rate-limit backoff only honored Retry-After and
ignored X's proprietary x-rate-limit-reset header. computeBackoffMs()
parses both (Retry-After = seconds or HTTP-date; x-rate-limit-reset =
epoch seconds), takes MAX, and clamps to [2s, 60s]. Exported for
testability; callers use it uniformly on every 429.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(writer): advisory lock on desiredSlug prevents cross-process TOCTOU
BrainWriter's createEntity checks engine.getPage(slug) and falls back
to putPage(), which upserts. Two putPage('people/alice') calls from
separate processes (a Claude Code session + a Minions worker, say) can
both read "free" from SlugRegistry and both call putPage, silently
overwriting each other with no disambiguation.
Take a transaction-scoped advisory lock keyed on hashtext(desiredSlug)
before the registry check. Concurrent writers for the same slug now
serialize at the DB level: the second observes the first's commit and
disambiguates to alice-2. PGLite is single-process so this is a
harmless no-op there. Wrapped in try/catch so engines/test doubles
that don't support advisory locks fall through to the existing
within-process check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(validators): empty [Source:] no longer satisfies citation check
Regex /\[Source:[^\]]*\]/ matched decorative markers like [Source:]
and [Source: ] that carry zero provenance. Tighten to require at
least one non-whitespace character before the closing bracket. The
inline URL form ](https://...) already requires a scheme+host so it
stays as-is.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(auto-link): advisory lock serializes concurrent reconciliation
runAutoLink wraps getLinks + addLink/removeLink in a transaction, but
row-level locks alone don't prevent the union-of-writes race: two
concurrent put_page calls on the same slug can both read the same
existingKeys BEFORE either mutates a row, then proceed to add links
the other side's rewrite no longer mentions.
Take a transaction-scoped advisory lock on hashtext("auto_link:" ||
slug) at the start of the reconciliation. Concurrent writers on the
same slug now fully serialize; writers on different slugs still run
in parallel. No-op on engines without advisory locks (PGLite).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: expand coverage on abort-signal threading + integrity CLI dispatch
fail-improve: four new AbortSignal cases — pre-start abort, between
deterministic and LLM, signal forwarded into both callbacks, and
LLM-thrown AbortError propagates without logging a failure entry.
integrity: three new CLI dispatch cases — --help, no-subcommand (help),
and unknown subcommand (stderr + exit 1). Non-engine paths so they
exercise routing without spinning up a DB.
Coverage-only; no source changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): fold integrity sample scan into default health check
Expose scanIntegrity(engine, opts) as a pure library function — same
logic cmdCheck uses — and call it from doctor in non-fast mode with
a 500-page sampling limit. Surfaces bare-tweet phrase count and
external-link count as an 'integrity' check, warn-status when bare
tweets are present with a one-liner pointing at 'gbrain integrity
check' for the full report and 'integrity auto' for repair.
Read-only: no network, no writes, no resolver calls. Pages with
validate:false frontmatter are skipped (grandfathered). --fast mode
skips it entirely so the existing health-snapshot contract holds.
Users no longer need to remember three separate commands (doctor,
lint, integrity check) to audit brain health — doctor surfaces the
integrity signal by default, full scan stays available for deep dives.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(put_page): auto-extract timeline entries alongside auto-link
put_page already chunks, embeds, reconciles tags, and extracts
auto-links on every write. Timeline extraction has lived in a
separate command (gbrain extract timeline) that users had to remember
to run. Fold it into the write path: after the page commits, parse
timeline entries from compiled_truth + timeline body and insert via
addTimelineEntriesBatch. ON CONFLICT DO NOTHING keeps it idempotent
across re-writes.
Mirrors auto-link shape: best-effort post-hook, skipped for remote
(MCP) callers, gated by auto_timeline config (default TRUE). Response
includes auto_timeline: { created } alongside auto_links.
Side effect: a one-shot `gbrain put` now produces a complete page —
chunks, embeddings, links, AND timeline — instead of three commands
the user has to chain manually.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(migrate): verify target health after engine migration
After a PGLite↔Postgres migration, the user was left to run 'gbrain
doctor' themselves to confirm the target is good. Not great, because
the failure modes (partial copy, missing embeddings, schema drift)
all surface at next CLI use when the migration itself looks like it
succeeded.
Add verifyTarget() — inline doctor-lite that checks page count
matches the source, embedding coverage is above 90%, and schema
version is at latest. Prints a 3-line status table at the end of
migrate and points at 'gbrain doctor' for the full check. Non-fatal:
warns on discrepancies instead of failing the command so the user
sees the full picture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(bench): add v0.13 knowledge runtime benchmark deltas
Two new benchmark scripts + one consolidated markdown comparing this
branch against master (
|
||
|
|
d8613366a5 |
Minions v7 + v0.11.1 canonical migration + skillify (#130)
* feat: add minion_jobs schema, migration v5, and executeRaw to BrainEngine Foundation for the Minions job queue system. Adds: - minion_jobs table (20 columns) with CHECK constraints, partial indexes, and RLS. Inspired by BullMQ's job model, adapted for Postgres. - Migration v5 creates the table for existing databases. - executeRaw<T>() method on BrainEngine interface for raw SQL access, needed by the Minions module for claim queries (FOR UPDATE SKIP LOCKED), token-fenced writes, and atomic stall detection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions job queue — queue, worker, backoff, types BullMQ-inspired Postgres-native job queue built into GBrain. No Redis. No external dependencies. Postgres transactions replace Lua scripts. - MinionQueue: submit, claim (FOR UPDATE SKIP LOCKED), complete/fail (token-fenced), atomic stall detection (CTE), delayed promotion, parent-child resolution, prune, stats - MinionWorker: handler registry, lock renewal, graceful SIGTERM, exponential backoff with jitter, UnrecoverableError bypass - MinionJobContext: updateProgress(), log(), isActive() for handlers - 8-state machine: waiting/active/completed/failed/delayed/dead/ cancelled/waiting-children Patterns stolen from: BullMQ (lock tokens, stall detection, flows), Sidekiq (dead set, backoff formula), Inngest (checkpoint/resume). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 43 tests for Minions job queue Full coverage of the Minions module against PGLite in-memory: - Queue CRUD (9): submit, get, list, remove, cancel, retry, duplicate - State machine (6): waiting→active→completed/failed, retry→delayed→waiting - Backoff (4): exponential, fixed, jitter range, attempts_made=0 edge - Stall detection (3): detect stalled, counter increment, max→dead - Dependencies (5): parent waits, fail_parent, continue, remove_dep, orphan - Worker lifecycle (5): register, start-without-handlers, claim+execute, non-Error throws, UnrecoverableError bypass - Lock management (3): renewal, token mismatch, claim sets lock fields - Claim mechanics (4): empty queue, priority ordering, name filtering, delayed promotion timing - Cancel & retry (2): cancel active, retry dead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions CLI commands and MCP operations Wire Minions into the GBrain CLI and MCP layer: CLI (gbrain jobs): submit <name> [--params JSON] [--follow] [--dry-run] list [--status S] [--queue Q] [--limit N] get <id> — detailed view with attempt history cancel/retry/delete <id> prune [--older-than 30d] stats — job health dashboard work [--queue Q] [--concurrency N] — Postgres-only worker daemon 6 MCP operations (contract-first, auto-exposed via MCP server): submit_job, get_job, list_jobs, cancel_job, retry_job, get_job_progress Built-in handlers: sync, embed, lint, import. --follow runs inline. Worker daemon blocked on PGLite (exclusive file lock). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for Minions job queue CLAUDE.md: added Minions files to key files, updated operation count (36), BrainEngine method count (38), test file count (45), added jobs CLI commands. CHANGELOG.md: added Minions entry to v0.10.0 (background jobs, retry, stall detection, worker daemon). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions v2 — agent orchestration primitives (pause/resume, inbox, tokens, replay) Adds the foundation for Minions as universal agent orchestration infrastructure. GBrain's Postgres-native job queue now supports durable, observable, steerable background agents. The OpenClaw plugin (separate repo) will consume these via library import, not MCP, for zero-latency local integration. ## New capabilities - **Concurrent worker** — Promise pool replaces sequential loop. Per-job AbortController for cooperative cancellation. Graceful shutdown waits for all in-flight jobs via Promise.allSettled. - **Pause/resume** — pauseJob clears the lock and fires AbortSignal on active jobs. Handlers check ctx.signal.aborted and exit cleanly. resumeJob returns paused jobs to waiting. Catch block skips failJob when signal.aborted. - **Inbox (separate table)** — minion_inbox table for sidechannel messages. sendMessage with sender validation (parent job or admin). readInbox is token-fenced and marks read_at atomically. Separate table avoids row bloat from rewriting JSONB on every send. - **Token accounting** — tokens_input/tokens_output/tokens_cache_read columns. updateTokens accumulates; completeJob rolls child tokens up to parent. USD cost computed at read time (no cost_usd column — pricing too volatile). - **Job replay** — replayJob clones a terminal job with optional data overrides. New job, fresh attempts, no parent link. ## Handler contract additions MinionJobContext now provides: - `signal: AbortSignal` — cooperative cancellation - `updateTokens(tokens)` — accumulate token usage - `readInbox()` — check for sidechannel messages - `log()` — now accepts string or TranscriptEntry ## MCP operations added pause_job, resume_job, replay_job, send_job_message — all auto-generate CLI commands and MCP server endpoints. ## Library exports package.json exports map adds ./minions and ./engine-factory paths so plugins can `import { MinionQueue } from 'gbrain/minions'` for direct library use. ## Instruction layer (the teaching) - skills/minion-orchestrator/SKILL.md — when/how to use Minions, decision matrix, lifecycle management, anti-patterns - skills/conventions/subagent-routing.md — cross-cutting rule: all background work goes through Minions - RESOLVER.md — trigger entries for agent orchestration - manifest.json — registered ## Schema migration v6 Additive: 3 token columns, paused status, minion_inbox table with unread index. Full Postgres + PGLite support. No backfill needed. ## Tests 65 tests (was 43): pause/resume (5), inbox (6), tokens (4), replay (4), concurrent worker context (3), plus all existing coverage. ## What's NOT in this commit Deferred to follow-up PRs: - LISTEN/NOTIFY subscribe (needs real Postgres E2E) - Resource governor (depends on concurrent worker stress testing) - Routing eval harness (needs API keys + benchmark data) - OpenClaw plugin (separate @gbrain/openclaw-minions-plugin repo) See docs/designs/MINIONS_AGENT_ORCHESTRATION.md for full CEO-approved design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): migration v7 — agent_parity_layer schema Adds columns on minion_jobs (depth, max_children, timeout_ms, timeout_at, remove_on_complete, remove_on_fail, idempotency_key) plus the new minion_attachments table. Three partial indexes for bounded scans: idx_minion_jobs_timeout, idx_minion_jobs_parent_status, and uniq_minion_jobs_idempotency. Check constraints enforce non-negative depth and positive child cap / timeout. Additive migration — existing installs pick it up via ensureSchema on next use. No user action required. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): extend types for v7 parity layer Extends MinionJob with depth/max_children/timeout_ms/timeout_at/ remove_on_complete/remove_on_fail/idempotency_key. Extends MinionJobInput with the same options plus max_spawn_depth override. Adds MinionQueueOpts (maxSpawnDepth default 5, maxAttachmentBytes default 5 MiB). Adds AttachmentInput/Attachment shapes and ChildDoneMessage in the InboxMessage union. rowToMinionJob updated to pick up the new columns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): attachments validator New module validateAttachment() gates every attachment write. Rejects empty filenames, path traversal (.., /, \), null bytes, oversized content (5 MiB default, per-queue override), invalid base64, and implausible content_type headers. Returns normalized { filename, content_type, content (Buffer), sha256, size } on success. The DB also enforces UNIQUE (job_id, filename) as defense-in-depth for concurrent addAttachment races — JS-only checks are not sufficient. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): queue v7 — depth, child cap, timeouts, cascade, idempotency, child_done Wraps completeJob and failJob in engine.transaction() so parent hook invocations (resolveParent, failParent, removeChildDependency) fold into the same transaction as the child update. A process crash between child and parent can't strand the parent in waiting-children anymore. Adds v7 behaviors: - Depth tracking. add() computes depth = parent.depth + 1 and rejects past maxSpawnDepth (default 5). - Per-parent child cap. add() takes SELECT ... FOR UPDATE on the parent, counts non-terminal children, rejects when count >= max_children. NULL max_children = no cap. - Per-job wall-clock timeout. claim() populates timeout_at when timeout_ms is set. New handleTimeouts() dead-letters expired rows with error_text='timeout exceeded'. Terminal — no retry. - Cascade cancel. cancelJob() walks descendants via recursive CTE with depth-100 runaway cap. Returns the root row. Re-parented descendants (parent_job_id NULL) are naturally excluded. - Idempotency. add() uses INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING; falls back to SELECT when RETURNING is empty. Same key always yields the same job id. - child_done inbox. completeJob inserts {type:'child_done', child_id, job_name, result} into the parent's inbox in the same transaction as the token rollup, guarded by EXISTS so terminal/deleted parents skip without FK violation. New readChildCompletions(parent_id, lock_token, since?) helper; token-fenced like readInbox. - removeOnComplete / removeOnFail. Deletes the row after the parent hook fires, so parent policy sees consistent state. - Attachment methods. addAttachment validates via validateAttachment then INSERTs; UNIQUE (job_id, filename) backs the JS dup check. listAttachments, getAttachment, deleteAttachment round out the API. Fixes pre-existing inverted status bug: add() now puts children in waiting/delayed (not waiting-children) and atomically flips the parent to waiting-children in the same transaction. Tests no longer need manual UPDATE workarounds. Two correctness fixes: - Sibling completion race. Under READ COMMITTED, two grandchildren completing concurrently each saw the other as still-active in the pre-commit snapshot and neither flipped the parent. Fixed by taking SELECT ... FOR UPDATE on the parent row at the start of completeJob and failJob transactions, serializing siblings on the parent lock. - JSONB double-encode. postgres.js conn.unsafe(sql, params) auto- JSON-encodes parameters. Calling JSON.stringify(obj) first stored a JSON string literal (jsonb_typeof=string) and broke payload->>'key' queries silently. Removed JSON.stringify from three call sites (child_done inbox post, updateProgress, sendMessage). PGLite tolerated both forms so unit tests missed it — real-PG E2E caught it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): worker — timeout safety net + handleTimeouts tick Worker tick now calls handleStalled() first, then handleTimeouts() — stall requeue wins over timeout dead-letter when both could fire in the same cycle. handleTimeouts() guards on lock_until > now() so stalled jobs take the retryable path. launchJob schedules a per-job setTimeout(timeout_ms) that fires ctx.signal as a best-effort handler interrupt. The timer is always cleared in .finally so process exit isn't delayed by a dangling timer. Handlers that respect AbortSignal stop cleanly; handlers that ignore it still get dead-lettered by the DB-side handleTimeouts. Removed post-completeJob and post-failJob parent-hook calls from the worker — those are now inside the queue method transactions. Worker becomes simpler and crash-safer. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(minions): 33 new unit tests for v7 parity layer Covers depth cap, per-parent child cap, timeout dead-letter, cascade cancel (including the re-parent edge case), removeOnComplete / removeOnFail, idempotency (single + concurrent), child_done inbox (posted in txn + survives child removeOnComplete + since cursor), attachment validation (oversize, path traversal, null byte, duplicates, base64), AbortSignal firing on pause mid-handler, catch-block skipping failJob when aborted, worker in-flight bookkeeping, token-rollup guard when parent already terminal, and setTimeout safety-net cleanup. Existing tests updated to remove the inverted-status manual UPDATE workarounds that the add() fix made obsolete. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(e2e): Minions v7 concurrency + OpenClaw resilience coverage minions-concurrency.test.ts spins two MinionWorker instances against the test Postgres, submits 20 jobs, and asserts zero double-claims (every job runs exactly once). This is the only test that actually proves FOR UPDATE SKIP LOCKED under real concurrency — PGLite runs on a single connection and can't exercise the race. minions-resilience.test.ts covers the six OpenClaw daily pains: 1. Spawn storm caps enforce under concurrent submit. 2. Agent stall → handleStalled() requeues; handleTimeouts() skips (lock_until guard). 3. Forgotten dispatches recoverable via child_done inbox. 4. Cascade cancel stops grandchildren mid-flight. 5. Deep tree fan-in (parent → 3 children → 2 grandchildren each) completes with the full inbox chain. 6. Parent crash/recovery resumes from persisted state. helpers.ts extends ALL_TABLES with minion_attachments, minion_inbox, and minion_jobs (FK dependents first) so E2E teardown doesn't leak rows between runs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: release v0.11.0 — Minions v7 agent orchestration primitives Bumps VERSION / package.json to 0.11.0. Adds CHANGELOG entry covering depth tracking, max_children, per-job timeouts, cascade cancel, idempotency keys, child_done inbox, removeOnComplete/Fail, attachments, migration v7, plus the two correctness fixes (sibling completion race and JSONB double-encode). TODOS.md captures the four v7 follow-ups: per-queue rate limiting, repeat/cron scheduler, worker event emitter, and waitForChildren convenience helpers. 1066 unit + 105 E2E = 1171 tests passing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): unify JSONB inserts, tighten nullish coalescing Three non-blocker cleanups from post-ship review of v0.11.0: - queue.ts add() and completeJob(): pre-stringifying with JSON.stringify while other sites pass raw objects with $n::jsonb casts. postgres.js double-encodes if you stringify first — works on PGLite (text→JSONB auto-cast), fails silently on real PG. Unify on raw object + explicit $n::jsonb cast. - queue.ts readChildCompletions: since clause used sent_at > $2 relying on PG's implicit text→TIMESTAMPTZ coercion. Explicit $2::timestamptz is safer and clearer. - types.ts rowToMinionJob: parent_job_id used || which coerces 0 to null. Harmless today (SERIAL IDs start at 1) but ?? is semantically correct. All 110 unit tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): updateProgress missed $1::jsonb cast in unification Residual from |
||
|
|
ce15062694 |
feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39)
* docs: break SKILLPACK into 17 individual guides The 1,281-line SKILLPACK monolith is now 17 individually linkable guides in docs/guides/, organized by category: core patterns, data pipelines, operations, search, and administration. GBRAIN_SKILLPACK.md becomes a structured index with categorized tables linking to each guide. The URL stays stable for backward compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add integration guides, architecture docs, and ethos New documentation directories: - docs/integrations/ — "Getting Data In" landing page, credential gateway, meeting webhooks. Includes recipe format documentation. - docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search) - docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide - docs/designs/ — "Homebrew for Personal AI" 10-star vision document Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add gbrain integrations command + voice-to-brain recipe New CLI command: gbrain integrations (list/show/status/doctor/stats/test) - Standalone command, no database connection needed - Uses gray-matter directly for recipe parsing (not parseMarkdown) - --json flag on every subcommand for agent-parseable output - Bare command shows senses/reflexes dashboard - Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl First recipe: recipes/twilio-voice-brain.md - Phone calls create brain pages via Twilio + OpenAI Realtime - Opinionated defaults: caller screening, brain-first lookup, quiet hours - Outbound call smoke test (GBrain calls the user to prove it works) - Validate-as-you-go credential testing - Twilio signature validation for webhook security Migration file for v0.7.0 with agent-readable changelog. 13 unit tests covering parseRecipe, CLI routing, and recipe validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Getting Data In to README, update CLAUDE.md and manifest README: voice calls in intro bullet list, new "Getting Data In" section with integration table (voice, email, X, calendar) and recipe philosophy. CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/, docs/integrations/, docs/architecture/, docs/ethos/). manifest.json: bump to v0.7.0, add recipes_dir field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: v0.7.0 CHANGELOG, TODOS, VERSION bump CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain, gbrain integrations command, SKILLPACK breakout, and new documentation. TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL, community recipe submission, always-on deployment recipes). VERSION + package.json: bump to 0.7.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite voice recipe with agent instructions and verified links Major improvements to recipes/twilio-voice-brain.md: - Agent preamble: explains WHY sequential execution matters (each step depends on the previous), defines 4 stop points where the agent MUST pause and verify, tells agent to never say "something went wrong" but instead explain the exact error and fix - User actions are now specific: exact URLs for every credential (Twilio console, OpenAI API keys page, ngrok dashboard), what buttons to click, what fields to copy, common failure modes - All URLs verified via web search against current 2026 documentation: Twilio SID/token at twilio.com/console, OpenAI keys at platform.openai.com/api-keys, ngrok token at dashboard.ngrok.com/get-started/your-authtoken - Cost estimate corrected: OpenAI Realtime is $0.06/min input + $0.24/min output (was understated), total ~$20-22/mo for 100 min - Validate-as-you-go: each credential tested immediately with exact curl commands, failure messages explain what went wrong and how to fix - Smoke test flow: tells user exactly what to say, verifies ALL three outputs (messaging notification + brain page + search result) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add "Homebrew for Personal AI" essay (markdown is code) New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution corollary to "Thin Harness, Fat Skills." Argues that markdown skill files are simultaneously documentation, specification, package, and source code. The agent is the package manager. The git repo is the app store. Referenced from SKILLPACK index and CLAUDE.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite agent instructions as command language, promote skills The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide. Every step is an imperative command with exact verification criteria and explicit stop-on-failure behavior. No FYI, no suggestions, just rails. Key changes: - 11-step setup with STOP points after each step - Exact user instructions for Supabase connection string (what to click, what NOT to give the agent, what the string looks like) - "Verify: run X. You must see Y. If not: Z" after every step - Skills table now links to both skill files AND guide docs - Integration recipes table simplified (no "coming soon" placeholders) - Docs section reorganized: for agents / for humans / reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 4 codex findings + add email-to-brain recipe Codex review found 4 issues, all fixed: 1. getStatus() returned "configured" if ANY secret was set (e.g. just OPENAI_API_KEY). Now requires ALL required secrets before marking configured. Prevents false "configured" status and spurious doctor runs. 2. Twilio health check hit unauthenticated endpoint (always 401). Now uses authenticated curl with SID:token, matching the setup validation. 3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md. 4. Compiled binary can't find recipes/ via import.meta.dir. Added GBRAIN_RECIPES_DIR env var override + global bun install path fallback. Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern with ClawVisor credential gateway, validate-as-you-go, agent instructions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add email, X, calendar, and meeting sync recipes Four new integration recipes extracted from production wintermute patterns: - recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector pattern (code pulls emails with baked-in links, agent does judgment), noise filtering, signature detection, digest generation - recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search, deletion detection (diffs previous run, verifies 404), engagement velocity tracking, rate limit awareness - recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical backfill (years of data), daily markdown files with attendees + locations, attendee enrichment for brain pages - recipes/meeting-sync.md: Circleback API, transcript import with speaker labels, attendee detection + filtering, entity propagation to people/ company pages, action item extraction, idempotent by source_id All recipes follow the same format: agent preamble with sequential execution rules, validate-as-you-go credentials, exact URLs for API key setup, stop-on-failure verification, and heartbeat logging. Updated README, SKILLPACK index, and integrations landing page with all 5 recipes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes Both recipes now offer two auth options: - Option A: ClawVisor (recommended, handles OAuth + token refresh) - Option B: Google OAuth2 directly (no extra service, you manage tokens) Option B includes step-by-step instructions for Google Cloud Console: exact URLs, which buttons to click, which scopes to add, how to enable the API, and the OAuth flow for token exchange. This removes ClawVisor as a hard dependency for getting started. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add implementation guides with pseudocode and test suggestions Every recipe now includes an "Implementation Guide" section with: - Production-tested pseudocode the agent can follow to build each collector - Edge cases and failure modes discovered in real deployment - Non-obvious implementation details (why the 48h staleness heuristic, why Gmail links need authuser, why SSE responses need double-parsing) - Test suggestions: what the agent should verify after setup email-to-brain: noise filtering algorithm, signature detection patterns, Gmail link generation (authuser is critical), sent-mail dedup x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness, API verification), engagement velocity thresholds (50 min for 2x, 100 absolute jump), atomic writes, stdout contract, rate limit handling calendar-to-brain: smart chunking (monthly for sparse years, weekly for dense), attendee filtering (rooms, groups, distros), merge-with-existing (only replace ## Calendar section), date/time parsing edge cases meeting-sync: SSE double-JSON parsing, idempotency double-check (grep + filename), auto-tagging from meeting names, git commit after sync Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: 6 new guides from production patterns (wintermute extraction) New guides extracted and generalized from production deployment: - repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge). Strict boundary rules, decision tree, hard rule: never write knowledge to the agent repo. - sub-agent-routing.md: Model routing table by task type. Signal detector pattern (spawn Sonnet on every message). Research pipeline pattern (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization. - skill-development.md: 5-step cycle (concept, prototype, evaluate, codify, cron). MECE discipline (no overlapping skills). Quality bar checklist. "If you ask twice, it should already be a skill." - idea-capture.md: Originality distribution rating (0-100 across 4 populations). Depth test ("could someone unfamiliar understand WHY?"). Deep cross-linking mandate. Notability filtering. - quiet-hours.md: Hold notifications 11pm-8am local time. Held messages directory pattern. Timezone-aware delivery. Morning briefing pickup. - diligence-ingestion.md: 9-step pipeline for data room materials. Detection patterns (PDF filenames, spreadsheet tabs, user language). Index.md template with bull/bear case. Company page enrichment. All PII scrubbed. Patterns generalized for any user. SKILLPACK index updated with 6 new entries. CLAUDE.md references added. All 37 SKILLPACK links verified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: upgrade all guides to operational playbooks with pseudocode Every guide now follows the playbook structure: - Goal: one sentence, what this achieves - What the User Gets: without this / with this - Implementation: pseudocode with actual gbrain commands - Tricky Spots: production-tested gotchas - How to Verify: test steps the agent runs after setup Guides upgraded (15 files): - brain-agent-loop: on_message() loop with read/write/sync pseudocode - brain-first-lookup: 4-step lookup cascade with exact commands - brain-vs-memory: routing algorithm for 3 knowledge layers - compiled-truth: page structure + rewrite vs append rules - content-media: 3 ingest patterns (YouTube, social, PDFs) - cron-schedule: full schedule table + dream cycle pseudocode - enrichment-pipeline: 7-step protocol with tier classification - entity-detection: spawn pattern + detection prompt + notability filter - executive-assistant: 3 workflow algorithms (triage, prep, post-inbox) - meeting-ingestion: 6-step transcript-to-brain flow - operational-disciplines: 5 executable discipline blocks - originals-folder: detection + exact-phrasing capture + cross-linking - search-modes: decision tree for keyword vs hybrid vs direct - source-attribution: citation format + hierarchy + conflict resolution - Plus Goal/What User Gets headers on 6 newer guides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add WebRTC to voice recipe + ngrok Hobby setup guide Voice recipe updates: - Added WebRTC endpoint (POST /session, GET /call, POST /tool) for browser-based calling with RNNoise noise suppression - WebRTC pseudocode with the 4 non-obvious gotchas from production (voice under audio.output.voice, no turn_detection, no session.update on connect, trigger greeting via data channel) - Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier - Fixed domain means URLs never change, Twilio never breaks New guide: docs/mcp/NGROK_SETUP.md - How to set up ngrok Hobby for both MCP and voice agent - Fixed domain setup, watchdog pattern, AI client configuration - Claude Desktop requires Settings > Integrations (not JSON config) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add dependency graph + ngrok-tunnel + credential-gateway recipes Recipes now have real dependencies via the `requires` field: - voice-to-brain requires ngrok-tunnel (needs public URL for Twilio) - email-to-brain requires credential-gateway (needs Gmail access) - calendar-to-brain requires credential-gateway (needs Calendar access) - x-to-brain and meeting-sync are standalone (direct API keys) Two new infrastructure recipes: - ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby ($8/mo) for a domain that never changes. Includes watchdog pattern. - credential-gateway: secure Google service access via ClawVisor (recommended) or direct OAuth2. One setup, all Google recipes use it. Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure, not MCP-specific. README and integrations landing page show dependency chains. When agent installs voice-to-brain, it sets up ngrok-tunnel first. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add infra category, fix dashboard alignment, show dependencies DX audit found two bugs in gbrain integrations dashboard: 1. Column alignment broken — IDs > 18 chars ran into descriptions with no space. Fixed: pad to 22 chars. 2. ngrok-tunnel and credential-gateway showed as SENSES but they're infrastructure. Added 'infra' category. Dashboard now shows three sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES. 3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)" Also added 'requires' field to JSON output for agent consumption. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add frontier model requirement disclaimer to README GBrain's markdown-is-code approach requires models capable of interpreting intent and implementing from architecture descriptions. Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models will struggle with the recipe format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add PGLite → Supabase upgrade path to README Clarify the database progression: start with PGLite (Postgres as WASM, zero infrastructure, pgvector built in, nothing to install). Graduate to Supabase or self-hosted Postgres when you need connection pooling, concurrency, and remote MCP access from Claude Desktop, Cowork, ChatGPT, Perplexity Computer, or any MCP-compatible agent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: revert PGLite mention (coming in next branch) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify) Every guide now has exactly these sections in this order: - ## Goal (one sentence) - ## What the User Gets (without this / with this) - ## Implementation (pseudocode with gbrain commands) - ## Tricky Spots (3-5 numbered gotchas) - ## How to Verify (3-5 numbered test steps) 11 guides restructured from non-standard headings: - deterministic-collectors, live-sync, upgrades-auto-update (full rewrites) - entity-detection, diligence-ingestion, idea-capture, quiet-hours, repo-architecture, skill-development, sub-agent-routing (restructured) 23/23 guides now pass consistency audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: restructure README around the #1 blocker (getting data in) The README was leading with Postgres and database architecture. Most users are stuck at step zero: "I have an agent but it doesn't know anything about my life." New structure: 1. The Problem — your agent doesn't know your life 2. Getting Data In — integration recipes, front and center 3. The Compounding Thesis — why this matters 4. How this happened — credibility, origin story 5. When you need Postgres — scale, not starting point Postgres is de-emphasized from a full section to two paragraphs: "You don't need Postgres to start" and "When you need Postgres" (1,000+ files, remote MCP access, multiple AI clients). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move Install to top of README, remove duplicate section Install now appears right after Getting Data In (line 38), not buried at line 295. The user sees: Problem → Getting Data In → Install. Removed the duplicate Install section (262 lines) that was lower in the README. The agent instructions block, CLI quickstart, and all content is now in the single Install section near the top. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move agent install block to first thing in README "Start here: paste this into your agent" is now the first section, right after the one-line pitch. No scrolling, no context, no preamble. User opens the README, sees the paste block, copies it into OpenClaw or Hermes, and the agent takes over. Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: compress install block from 11 steps to 5 The agent install block was 102 lines and 11 steps. Now it's 40 lines and 5 steps. Same coverage, half the text. Changes: - Merged "prove keyword search" + "embed" + "prove hybrid search" into one SEARCH step (the user doesn't care about the intermediate) - Merged skillpack, sync, auto-update, integrations, verification into one GO LIVE step with sub-items (post-install polish, not install) - Shortened database instructions (one line instead of 5 sub-steps) - Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now just "Do not skip steps. Verify each step.") The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: gitignore all .env files, not just specific ones CSO audit found .gitignore covered .env.testing and .env.production but not bare .env. A user creating .env with database credentials could accidentally commit it. Fix: .env and .env.* are now gitignored. .env.*.example files are explicitly un-ignored so templates remain tracked. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: scrub PII from essay and recipe examples - 510-MY-GARRY phone mnemonic → "Your Phone Number" - "Garry → Authenticated Mode" → "Owner → Authenticated Mode" - "Telegram" → "secure channel" in auth example - @garrytan → @yourhandle in X recipe example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |