2 Commits

Include renames

Author	SHA1	Message	Date
3 people	c22ca84772	feat: v0.13 frontmatter relationship indexing — YAML becomes typed graph edges (#231) ... * feat(schema): links provenance + engine plumbing (v0.13) Adds link_source, origin_page_id, origin_field columns with UNIQUE NULLS NOT DISTINCT constraint + CHECK constraint. New indexes on link_source + origin_page_id. migrate.ts v11 handles idempotent upgrade path for existing brains. Both engines: addLink/addLinksBatch threads new columns (4→7 col unnest). removeLink gains linkSource filter. getLinks/getBacklinks return new columns. New engine method findByTitleFuzzy(name, dirPrefix?, minSim?) uses pg_trgm % operator + similarity(). Drives the v0.13 resolver's fuzzy-match step with zero LLM/embedding cost. * feat(graph): frontmatter edge extraction + slug resolver (v0.13) Canonical FRONTMATTER_LINK_MAP: field → type + direction + dir-hint for 10 frontmatter patterns (company/companies, key_people, investors, attendees, partner, lead, founded, sources, source, related/see_also). Direction semantics: "incoming" means resolved value is the FROM side so subject-of-verb reads naturally (pedro → meeting, not backwards). makeResolver(engine, {mode}) — two-mode resolver: batch (migration): slug → dir-hint → pg_trgm. NEVER hits search. live (put_page): + optional search fallback with expand=false (dodges hidden Haiku per operations-query learning). Per-run cache: same name → single DB lookup. extractFrontmatterLinks handles arrays-of-objects (investors: [{name: 'Sequoia', role: 'lead'}]), skips bad types silently, tracks unresolved names for the summary report. extractPageLinks is now async. LinkCandidate gains fromSlug, linkSource, originSlug, originField. Returns {candidates, unresolved}. 22 new tests: field-map coverage, direction semantics, source vs sources, resolver fallback chain (batch + live), cache hit, bad types skipped, context enrichment, FRONTMATTER_LINK_MAP integrity. * feat(auto-link): bidirectional reconciliation + unresolved response put_page auto-link post-hook now handles incoming-direction frontmatter edges. Reconciliation splits candidates into out (fromSlug === slug) and in (fromSlug !== slug — frontmatter fields like key_people on a company page emit person → company edges). Safe reconciliation via origin_page_id scoping: we only touch link_source='frontmatter' edges where origin_slug = the page being written. Markdown + manual edges survive untouched. Edges created by OTHER pages' frontmatter also survive. put_page response extends auto_links with unresolved: Array<{field, name}>. Agents writing attendees: [Pedro, Alex] where Alex doesn't resolve see it in the response and can queue for enrichment. Additive — existing agents unaffected. extract.ts: delete the local 5-field extractFrontmatterLinks + local inferLinkType. FS-source now calls canonical link-extraction.ts via a synthetic resolver backed by the allSlugs Set. --include-frontmatter flag (default OFF in v0.13 for back-compat; migration explicitly enables for the one-time backfill). Top-20 unresolved names summary when active. * feat(migration): v0.13.0 orchestrator 3-phase orchestrator (schema → backfill → verify → record) follows the v0_12_2.ts pattern. Phase A triggers migrate.ts v11 via gbrain init --migrate-only. Phase B runs: gbrain extract links --source db --include-frontmatter to backfill frontmatter edges for every existing page. Uses the batch-mode resolver (pg_trgm only, no LLM calls, zero API cost). Ignores auto_link=false config — migration is canonical, the auto_link flag controls per-write post-hook not one-time schema work. Idempotent + resumable via ON CONFLICT DO NOTHING + origin_page_id scoping. Wall-clock budget: 2-5 min on 46K-page brains. Registered in migrations/index.ts. apply-migrations test updated to include v0.13.0 in skippedFuture for older installed versions. * feat(release): upgrade-errors.jsonl trail + doctor surfacing upgrade.ts catches post-upgrade subprocess failures as best-effort today (line 65 comment: "post-upgrade is best-effort, don't fail the upgrade"). When that chain silently fails, users end up with half-upgraded brains and no signal. v0.13: on post-upgrade failure, append a structured record to ~/.gbrain/upgrade-errors.jsonl with ts, phase, versions, error message, and a paste-ready recovery hint. doctor.ts reads the jsonl and surfaces the latest entry with a warn-status check. User runs gbrain doctor, sees exactly what failed, pastes the recovery command, files an issue if needed. Applies to every future release — doctor grows with the codebase without per-release edits. The CHANGELOG pattern ("To take advantage of v[version]" block) mirrors this in user-facing form. * chore: bump version and changelog (v0.13.0) v0.13.0 — Frontmatter Relationship Indexing. Adds the "To take advantage of v[version]" block pattern to CHANGELOG format (CLAUDE.md documents the requirement going forward). Pairs with the upgrade-errors.jsonl + doctor surfacing to close the "half-upgraded brain, no signal" loop. UPGRADING_DOWNSTREAM_AGENTS.md gets a v0.13 section: no-action- required verdict for most skills, optional diffs for meeting- ingestion / enrich / idea-ingest if they want to consume auto_links.unresolved. skills/migrations/v0.13.0.md is the user-facing upgrade skill. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.13): adversarial review P0s Codex + Claude adversarial review caught 4 critical issues in the v0.13 implementation. Fixing before ship. 1. findByTitleFuzzy SET LOCAL was a no-op. postgres.js auto-commits each sql`` so SET LOCAL pg_trgm.similarity_threshold committed before the `%` operator ran against it. Resolver used server default (0.3, not 0.55) → way too many fuzzy matches, wrong links on a 46K-page brain. Switched to inline `similarity(title, $1) >= $N` which has no transaction scoping. Added `ORDER BY sim DESC, slug ASC` for deterministic tie-breaking (prevents reconciliation churn on re-runs). 2. v11 migration now checks Postgres ≥ 15 before applying UNIQUE NULLS NOT DISTINCT. Old Supabase projects on PG14 would have dropped the old unique constraint and failed to add the new one, corrupting the uniqueness invariant. The check raises a clear error with the actual PG version, leaving the old constraint in place. 3. v11 migration now backfills NULL link_source → 'markdown' for pre-v0.13 legacy rows. Without this, reconciliation's existKey comparison treats NULL and 'markdown' as equivalent but the unique constraint sees them as distinct (NULLS NOT DISTINCT only collapses NULL with NULL, not NULL with 'markdown'). Result was duplicate edges accumulating forever. Treating legacy as markdown is the accurate best-guess — pre-v0.13 auto-link only emitted markdown edges. 4. v0_13_0.ts orchestrator now uses process.execPath, not a bare `gbrain` on PATH. After `gbrain upgrade` rewrites the binary, alias shadowing / PATH caching / multiple installs could resolve a stale `gbrain` binary. process.execPath is always the binary that loaded this migration module. Phase C verify clarified: reports page + link counts and points to Phase B's own stdout as the authoritative signal for backfill results (extract.ts already prints `Links: created N from M pages`). * docs: scrub real names from public docs + add privacy rule to CLAUDE.md Public artifacts (CHANGELOG, skills, docs) should never reveal real contacts, companies, funds, or private agent-fork names from any user's brain. When a doc copies a query like `gbrain graph diana-hu` or names a fork like `Wintermute`, that real name gets indexed, cross-referenced, and distributed with every release. CLAUDE.md gains a "Privacy rule: scrub real names from public docs" section with: - What counts as public (CHANGELOG, README, docs/, skills/, PR bodies, commit messages, code comments) - Name mapping table (agent forks → your agent fork; example person → alice-example; example fund → fund-a; etc.) - Distinction between illustrative API examples with household brands (Stripe, Brex) and queries that reveal real relationships Applied the rule to v0.13 scope: - CHANGELOG v0.13 entry: Pedro/Diana/Wintermute/Sequoia/Benchmark/a16z all replaced with alice/charlie/fund-a/acme/agent-fork placeholders - skills/migrations/v0.13.0.md: same - docs/UPGRADING_DOWNSTREAM_AGENTS.md: Wintermute references scrubbed throughout (pre-v0.13 and v0.13 sections) - CLAUDE.md: "Brain skills (from Wintermute)" → "(ported from an upstream agent fork)", internal Wintermute provenance notes genericized, "Garry finds fragile upgrade paths" → "the gbrain maintainers find fragile upgrade paths" in the template Pre-v0.13 historical CHANGELOG entries (v0.10-v0.12) left alone — those are shipped releases; rewriting changes public history. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 07:05:27 +08:00
3 people	b7e3005b5b	fix: sync pipeline, extract, features, autopilot (v0.10.1) (#129) ... * feat: migrate 8 existing skills to conformance format Add YAML frontmatter (name, version, description, triggers, tools, mutating), Contract, Anti-Patterns, and Output Format sections to all existing skills. Rename Workflow to Phases. Ingest becomes thin router delegating to specialized ingestion skills (Phase 2). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add RESOLVER.md, conventions directory, and output rules RESOLVER.md is the skill dispatcher modeled on Wintermute's AGENTS.md. Categorized routing table: Always-on, Brain ops, Ingestion, Thinking, Operational, Setup, Identity. Conventions directory extracts cross-cutting rules (quality, brain-first lookup, model routing, test-before-bulk). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add skills conformance and resolver validation tests skills-conformance.test.ts validates every skill has YAML frontmatter with required fields, Contract, Anti-Patterns, and Output Format sections, and manifest.json coverage. resolver.test.ts validates routing table categories, skill path existence, and manifest-to-resolver coverage. 50 new tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add 9 brain skills from Wintermute (Phase 2) Generalized from Wintermute's battle-tested skills: - signal-detector: always-on idea+entity capture on every message - brain-ops: brain-first lookup, read-enrich-write loop, source attribution - idea-ingest: links/articles/tweets with author people page mandatory - media-ingest: video/audio/PDF/book with entity extraction (absorbs video/youtube/book) - meeting-ingestion: transcripts with attendee enrichment chaining - citation-fixer: audit and fix citation formatting - repo-architecture: filing rules by primary subject - skill-creator: create skills with conformance standard + MECE check - daily-task-manager: task lifecycle with priority levels All Garry-specific references generalized. Core workflows preserved. Updated RESOLVER.md and manifest.json. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add operational infrastructure + identity layer (Phase 3) Operational skills: - daily-task-prep: morning prep with calendar context and open threads - cross-modal-review: quality gate via second model with refusal routing - cron-scheduler: schedule staggering, quiet hours, wake-up override, idempotency - reports: timestamped reports with keyword routing - testing: skill validation framework (conformance checks) - soul-audit: 6-phase interview generating SOUL.md, USER.md, ACCESS_POLICY.md, HEARTBEAT.md - webhook-transforms: external events to brain signals with dead-letter queue Identity layer: - SOUL.md template (agent identity, generated by soul-audit) - USER.md template (user profile, generated by soul-audit) - ACCESS_POLICY.md template (4-tier access control) - HEARTBEAT.md template (operational cadence) - cross-modal.yaml convention (review pairs, refusal routing chain) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md with 24 skills, RESOLVER.md, conventions, templates GBrain is now a GStack mod for agent platforms. Updated architecture description, key files listing (16 new skill files, RESOLVER.md, conventions, templates), skills section (24 skills organized by resolver categories), and testing section (new conformance and resolver tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add GStack detection + mod status to gbrain init (Phase 4) After brain initialization, gbrain init now reports: - Number of skills loaded (from manifest.json) - GStack detection (checks known host paths, uses gstack-global-discover if available) - GStack install instructions if not found - Resolver and soul-audit pointers Also adds installDefaultTemplates() for SOUL.md/USER.md/ACCESS_POLICY.md/HEARTBEAT.md deployment, and detectGStack() using gstack-global-discover with fallback to known paths (DRY: doesn't reimplement GStack's host detection logic). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: v0.10.0 release documentation - CHANGELOG: 24 skills, signal detector, RESOLVER.md, soul-audit, access control, conventions, conformance standard, GStack detection in init - README: updated skill section with 24 skills, resolver, conventions - TODOS: added runtime MCP access control (P1) - VERSION: 0.9.2 → 0.10.0 - package.json + manifest.json version bumped Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add skill table to CHANGELOG v0.10.0 16-row table detailing every new skill, what it does, and why it matters. Written to sell the upgrade, not document the implementation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore package.json version after merge conflict resolution Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: zero-based README rewrite for GStackBrain v0.10.0 Lead with GStack mod identity. 24 skills table organized by category. Install block references RESOLVER.md and soul-audit. GBrain+GStack relationship explained. Removed redundancy (733 -> 406 lines). All essential content preserved: install, recipes, architecture, search, commands, engines, voice, knowledge model. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: extract install block to INSTALL_FOR_AGENTS.md, simplify README The 30-line copy-paste install block becomes one line: "Retrieve and follow INSTALL_FOR_AGENTS.md" Benefits: agent always gets latest instructions (no stale copy-paste), README stays clean, install details live where agents read them. README now leads with what GBrain does ("gives your agent a brain") instead of GStack relationship. Removed "requires frontier model" note. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 3 bugs in init.ts from merge conflict resolution 1. llstatSync typo (merge corruption) → lstatSync 2. __dirname undefined in ESM module → fileURLToPath polyfill 3. require('fs') in ESM → use imported readFileSync All three would crash gbrain init at runtime. Caught by /review. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add checkResolvable shared core function for resolver validation Shared function at src/core/check-resolvable.ts validates that all skills are reachable from RESOLVER.md, detects MECE overlaps (with whitelist for always-on/router skills), finds gaps in frontmatter triggers, and scans for DRY violations. Returns structured ResolvableIssue objects with machine-parseable fix objects alongside human-readable action strings. Three call sites: bun test, gbrain doctor, skill-creator skill. Cleans up test/resolver.test.ts: removes stale 9-line skip list, imports from production check-resolvable.ts instead of reimplementing parsing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: expand doctor with resolver validation, filesystem-first architecture Doctor now runs filesystem checks (resolver health, skill conformance) before connecting to DB. New --fast flag skips DB checks. Falls back to filesystem-only when DB is unavailable. Adds schema_version: 2 to JSON output, composite health score (0-100), and structured issues array with action strings for agent parsing. Resolver health check calls checkResolvable() and surfaces actionable fix instructions. Link integrity check uses engine.getHealth() dead_links count. CLI routing split: doctor dispatched before connectEngine() so filesystem checks always run. Fixes Codex-identified blocker where doctor required DB. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add adaptive load-aware throttling and fail-improve loop backoff.ts: System load checking (CPU via os.loadavg, memory via os.freemem), exponential backoff with 20-attempt max guard, active hours multiplier (2x slower during waking hours), concurrent process limit (max 2). Windows-safe: defaults to "proceed" when os.loadavg returns zeros. fail-improve.ts: Deterministic-first, LLM-fallback pattern with JSONL failure logging. Cascade failure handling: when both paths fail, throws LLM error and logs both. Log rotation at 1000 entries. Call count tracking for deterministic hit rate metrics. Auto-generates test cases from successful LLM fallbacks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add transcription service and enrichment-as-a-service transcription.ts: Groq Whisper (default) with OpenAI fallback. Files >25MB segmented via ffmpeg. Provider auto-detection from env vars. Clear error messages for missing API keys and unsupported formats. enrichment-service.ts: Global enrichment service callable from any ingest pathway. Entity slug generation (people/jane-doe, companies/acme-corp), mention counting via searchKeyword, tier auto-escalation (Tier 3→2→1 based on mention frequency and source diversity), batch enrichment with backoff throttling, regex-based entity extraction from text. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add data-research skill with recipe system, extraction, dedup, tracker New skill: data-research — one parameterized pipeline for any email-to- structured-data workflow (investor updates, donations, company metrics). 7-phase pipeline: define recipe, search, classify, extract (with extraction integrity rule), archive, deduplicate, update tracker. data-research.ts: Recipe validation, MRR/ARR/runway/headcount regex extraction (battle-tested patterns), dedup with configurable tolerance, markdown tracker parsing/appending, quarterly/monthly date windowing, 6-phase HTML email stripping with 500KB ReDoS cap. Registers data-research in manifest.json (25th skill) and RESOLVER.md. Fixes backoff test robustness for high-load systems. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.10.0 infrastructure additions CLAUDE.md: added 6 new core files (check-resolvable, backoff, fail-improve, transcription, enrichment-service, data-research), 6 new test files, updated skill count to 25, test file count to 34. README.md: updated skill count to 25, added data-research to skills table. CHANGELOG.md: added Infrastructure section documenting resolver validation, doctor expansion, adaptive throttling, fail-improve loop, voice transcription, enrichment service, and data-research skill. TODOS.md: anonymized personal references. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: doctor.ts use ES module imports, harden backoff test Replace require('fs') with ES module import in doctor.ts for consistency with the rest of the file. Backoff test made resilient to parallel test execution leaking module-level state. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: sync --watch routing, dead_links parity, doctor command, embed --slugs - Move sync to CLI_ONLY so --watch flag reaches runSync() (was routed through operation layer which only calls performSync single-pass) - Hide sync_brain from CLI help (MCP still exposes it) - Fix performFullSync missing sync state persistence (C1) - Align Postgres dead_links query to match PGLite (count dangling links, not empty-content chunks) (C3) - Fix doctor recommending nonexistent 'gbrain embed refresh' (C4) - Refactor doctor outputResults to not call process.exit directly - Add --slugs flag to embed for targeted page embedding - Add sync auto-extract + auto-embed after performSync - Add noExtract to SyncOpts - Route extract, features, autopilot in CLI_ONLY - Update help text with new commands Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: extract, features, and autopilot commands - gbrain extract <links|timeline|all> — batch extraction of links and timeline entries from brain markdown files. Broad regex for all .md links (C7: filters external URLs). Frontmatter field parsing (company, investors, attendees). Directory-based link type inference. JSONL progress on stderr for agents. Sync integration hooks (extractLinksForSlugs, extractTimelineForSlugs). - gbrain features [--json] [--auto-fix] — scan brain usage, pitch unused features with the user's own numbers. Priority 1 (data quality): missing embeddings, dead links. Priority 2 (unused features): zero links, zero timeline, low coverage, unconfigured integrations, no sync. Embedded recipe metadata for binary-safe integration detection. Persistence in ~/.gbrain/feature-offers.json. Doctor teaser hook. Upgrade hook. - gbrain autopilot [--repo] [--interval N] — self-maintaining brain daemon. Pipeline: sync → extract → embed. Health-based adaptive scheduling (brain_score >= 90 doubles interval, < 70 halves it). --install/--uninstall for launchd (macOS) and crontab (Linux). Signal handling. Consecutive error tracking (stops at 5). Log to ~/.gbrain/autopilot.log. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: hook features scan into post-upgrade flow After gbrain post-upgrade completes, automatically run gbrain features to show the user what's new and what to fix. Best-effort (doesn't fail the upgrade). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: brain_score (0-100) in BrainHealth Weighted composite score computed in getHealth() for both Postgres and PGLite: embed_coverage: 0.35, link_density: 0.25, timeline_coverage: 0.15, no_orphans: 0.15, no_dead_links: 0.10 Returns 0 for empty brains. Agents use brain_score as a health gate. Autopilot uses it for adaptive scheduling (>=90 slows down, <70 speeds up). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: extract and features unit tests 25 tests covering: - extractMarkdownLinks: relative links, external URL filtering, edge cases - extractLinksFromFile: slug resolution, frontmatter parsing, directory-based type inference (works_at, deal_for, invested_in) - extractTimelineFromContent: bullet format, header format with detail, em/en dash handling, empty content - features: module exports, brain_score calculation weights, CLI routing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: instruction layer for extract, features, autopilot Agent-facing tools are invisible without instruction-layer coverage. - RESOLVER.md: add routing for extract, features, autopilot - maintain/SKILL.md: add link graph extraction, timeline extraction, autopilot check sections Without these, agents reading skills/ will never discover or run the new commands. This is the #1 DX finding from the devex review. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.10.1) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: sync CLAUDE.md with v0.10.1 additions Add extract.ts, features.ts, autopilot.ts to key files. Add extract.test.ts, features.test.ts to test list. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: adversarial review fixes — 7 issues - #3: autopilot extract step was a no-op (imported but never called) - #6: PGLite orphan_pages query aligned with Postgres (check both inbound+outbound) - #8: embedPage throws instead of process.exit (was killing sync/autopilot) - #9: dead-links set auto_fixable=false (needs repo path we may not have) - #10: JSON auto-fix output was dead code (unreachable !jsonMode check) - #14: autopilot lock file prevents concurrent instances - #20: --dir without value no longer crashes extract Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: fix command injection + plaintext API key in daemon install - #1: Crontab install used echo pipe with shell-interpolated values. Now uses a temp file via crontab(1) and single-quote escaping on all interpolated paths. No shell expansion possible. - #2: OPENAI_API_KEY was baked as plaintext into the launchd plist (readable by any local process, backed up by Time Machine). Now uses a wrapper script (~/.gbrain/autopilot-run.sh) that sources ~/.zshrc at runtime. No secrets in plist or crontab. - #16: extract.ts used a custom 20-line YAML parser that only handled single-line key:value pairs. Multi-line arrays (attendees list with - items) were silently ignored. Now uses the project's gray-matter parser via parseMarkdown() from src/core/markdown.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-14 21:40:48 -10:00