admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-31 04:07:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
93 Commits 567 Branches 1 Tag
9a549342163fe3c3c13f28f6ea9c8d4d8539e430
Commit Graph
12 Commits
Author SHA1 Message Date
Garry Tan 9a54934216 test(bench): durability under SIGKILL — Minions vs OpenClaw --local 
Headline bench for the claim: when the orchestrator dies mid-dispatch,
Minions rescues via PG state + stall detection; OpenClaw --local loses
in-flight work outright.

Minions side: seed 10 active+expired-lock rows (exact state a SIGKILLed
worker leaves) then run a rescue worker. Expect 10/10 completed.
OpenClaw side: spawn 10 `openclaw agent --local` in parallel, SIGKILL
each at 500ms, count pre-kill delivered output. Expect 0/10 — no
persistence layer, nothing to recover.

Budget: ~$0 (Minions handlers sleep 10ms; OC calls die at 500ms so
partial LLM billing is negligible).
 2026-04-18 08:23:30 +08:00
Garry Tan c90cb471d9 test(bench): add harness for Minions vs OpenClaw subagent dispatch 
Shared harness (openclawDispatch + minionsHandler) using matching
claude-haiku-4-5 calls on both sides so the delta measures queue+
dispatch overhead on top of identical LLM work. Includes
statsFromResults (p50/p95/p99) and formatStats helpers. Uses
`openclaw agent --local` embedded mode; does not test gateway
multi-agent fan-out (documented in the harness header).
 2026-04-18 08:21:28 +08:00
Garry TanandClaude Opus 4.7 76a530cf44 test(e2e): Minions v7 concurrency + OpenClaw resilience coverage 
minions-concurrency.test.ts spins two MinionWorker instances against the
test Postgres, submits 20 jobs, and asserts zero double-claims (every job
runs exactly once). This is the only test that actually proves FOR UPDATE
SKIP LOCKED under real concurrency — PGLite runs on a single connection
and can't exercise the race.

minions-resilience.test.ts covers the six OpenClaw daily pains:
1. Spawn storm caps enforce under concurrent submit. 2. Agent stall →
handleStalled() requeues; handleTimeouts() skips (lock_until guard).
3. Forgotten dispatches recoverable via child_done inbox. 4. Cascade
cancel stops grandchildren mid-flight. 5. Deep tree fan-in
(parent → 3 children → 2 grandchildren each) completes with the full
inbox chain. 6. Parent crash/recovery resumes from persisted state.

helpers.ts extends ALL_TABLES with minion_attachments, minion_inbox, and
minion_jobs (FK dependents first) so E2E teardown doesn't leak rows
between runs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-18 06:32:41 +08:00
Garry TanGitHubClaude Opus 4.7
 		7bbfc3e36a security: fix wave 3 — 9 vulns (file_upload, SSRF, recipe trust, prompt injection) (#174) 
* feat(engine): add cap parameter to clampSearchLimit (H6)

clampSearchLimit(limit, defaultLimit, cap = MAX_SEARCH_LIMIT) — third arg
is a caller-specified cap so operation handlers can enforce limits below
MAX_SEARCH_LIMIT. Backward compatible: existing two-arg callers still cap
at MAX_SEARCH_LIMIT.

This fixes a Codex-caught semantics bug: the prior signature took (limit,
defaultLimit) where the second arg was misread as a cap. clampSearchLimit(x, 20)
was actually allowing values up to 100, not 20.

* feat(integrations): SSRF defense + recipe trust boundary (B1, B2, Fix 2, Fix 4, B3, B4)

- B1: split loadAllRecipes into trusted (package-bundled) and untrusted
  (cwd/recipes, $GBRAIN_RECIPES_DIR) tiers. Only package-bundled recipes
  get embedded=true. Closes the fake trust boundary that let any cwd-local
  recipe bypass health-check gates.
- B2: hard-block string health_checks for non-embedded recipes (was previously
  only blocked when isUnsafeHealthCheck regex matched, which the cwd recipe
  exploit bypassed). Embedded recipes still get the regex defense.
- Fix 2: gate command DSL health_checks on isEmbedded. Non-embedded
  recipes cannot spawnSync.
- Fix 4 + B3 + B4: gate http DSL health_checks on isEmbedded; for embedded
  recipes, validate URLs via new isInternalUrl() before fetch:
  - Scheme allowlist (http/https only): blocks file:, data:, blob:, ftp:, javascript:
  - IPv4 range check covering hex/octal/decimal/single-integer bypass forms
  - IPv6 loopback ::1 + IPv4-mapped ::ffff: (canonicalized hex hextets handled)
  - Metadata hostnames (AWS, GCP, instance-data) blocked
  - fetch with redirect: 'manual' + per-hop re-validation up to 3 hops

Original PRs #105-109 by @garagon. Wave 3 collector branch reimplemented
the fixes after Codex outside-voice review found that PRs #106/#108 alone
did not actually gate cwd-local recipes (B1) and that PR #108 missed
redirect-following SSRF (B3) and non-http schemes (B4).

* feat(file_upload): path/slug/filename validation + remote-caller confinement (Fix 1, B5, H5, M4, Fix 5)

- Fix 1 + B5 + H1: validateUploadPath uses realpathSync + path.relative
  to defeat symlink-parent traversal. lstatSync alone (the original PR #105
  approach) only catches final-component symlinks; a symlinked parent dir
  still followed to /etc/passwd. Now the entire path chain is resolved.
- H5: validatePageSlug uses an allowlist regex (alphanumeric + hyphens,
  slash-separated segments). Closes URL-encoded traversal (%2e%2e%2f),
  Unicode lookalikes, backslashes, control chars implicitly.
- M4: validateFilename allowlist regex. Rejects control chars, backslash,
  RTL override (\u202E), leading dot/dash. Filename flows into storage_path
  so this matters for every storage backend.
- Fix 5: clamp list_pages and get_ingest_log limits at the operation layer
  via new clampSearchLimit cap parameter (list_pages caps at 100,
  get_ingest_log at 50). Internal bulk commands bypass the operation
  layer and remain uncapped.
- New OperationContext.remote flag distinguishes trusted local CLI from
  untrusted MCP callers. file_upload uses strict cwd confinement when
  remote=true (default), loose mode when remote=false (CLI). MCP stdio
  server sets remote=true; cli.ts and handleToolCall (gbrain call) set
  remote=false.

Original PR #105 by @garagon. Issue #139 reported by @Hybirdss.

* feat(search): query sanitization + structural prompt boundary (Fix 3, M1, M2, M3)

- M1: restructure callHaikuForExpansion to use a system message that declares
  the user query as untrusted data, plus an XML-tagged <user_query> boundary
  in the user message. Layered defense with the existing tool_choice constraint
  (3 layers vs 1).
- Fix 3 (regex sanitizer, defense-in-depth): sanitizeQueryForPrompt strips
  triple-backtick code fences, XML/HTML tags, leading injection prefixes,
  and caps at 500 chars. Original query is still used for downstream search;
  only the LLM-facing copy is sanitized.
- M2: sanitizeExpansionOutput validates the model's alternative_queries array
  before it flows into search. Strips control chars, caps length, dedupes
  case-insensitively, drops empty/non-string items, caps to 2 items.
- M3: console.warn on stripped content NEVER logs the query text — privacy-safe
  debug signal only.

Original PR #107 by @garagon. M1/M2/M3 are wave 3 hardening per Codex review.

* chore: bump version and changelog (v0.10.2)

Security wave 3: 9 vulnerabilities closed across file_upload, recipe trust
boundary, SSRF defense, prompt injection, and limit clamping. See CHANGELOG
for full details.

Contributors:
- @garagon (PRs #105-109)
- @Hybirdss (Issue #139)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: sync documentation with v0.10.2 security wave 3

- CLAUDE.md: document OperationContext.remote, new security helpers
  (validateUploadPath, validatePageSlug, validateFilename, isInternalUrl,
  parseOctet, hostnameToOctets, isPrivateIpv4, getRecipeDirs,
  sanitizeQueryForPrompt, sanitizeExpansionOutput), updated clampSearchLimit
  signature, recipe trust boundary, new test files
- docs/integrations/README.md: replace string-form health_check example
  with typed DSL (string checks now hard-block for non-embedded recipes);
  add recipe trust boundary subsection
- docs/mcp/DEPLOY.md: document file_upload remote-caller cwd confinement,
  symlink rejection, slug/filename allowlists

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-16 23:03:15 -07:00
Garry TanGitHubClaude Opus 4.6YIING994shut0sh
 		d547a64600 feat: search quality boost — compiled truth ranking + detail parameter (v0.8.1) (#64) 
* feat: search quality boost — compiled truth ranking, detail parameter, cosine re-scoring

Compiled truth chunks now rank 2x higher in hybrid search via RRF
normalization + source boost. New --detail flag (low/medium/high)
controls timeline inclusion. Cosine re-scoring blends query-chunk
similarity before dedup for query-specific ranking.

Also: remove DISTINCT ON from keyword search (dedup handles per-page
capping), add chunk_id + chunk_index to SearchResult, add
getEmbeddingsByChunkIds to BrainEngine interface.

Inspired by Ramp Labs' "Latent Briefing" paper (April 2026).

* feat: RRF normalization, source-aware dedup, detail param in operations

RRF scores normalized to 0-1 before 2.0x compiled truth boost.
Source-aware dedup guarantees compiled truth chunk per page.
Detail parameter added to query operation, dedupResults added to
bare search operation. Debug logging via GBRAIN_SEARCH_DEBUG=1.

* chore: bump version and changelog (v0.8.1)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: CJK word count in query expansion

CJK text is not space-delimited. A query like "向量搜索优化" was counted
as 1 word and silently skipped expansion. Now counts characters for CJK
queries instead of space-separated tokens.

Co-Authored-By: YIING99 <yiing99@users.noreply.github.com>

* feat: retrieval evaluation harness — P@k, R@k, MRR, nDCG@k + gbrain eval

Full IR evaluation framework: precisionAtK, recallAtK, mrr, ndcgAtK
metrics with runEval() orchestrator. gbrain eval CLI with single-run
table and A/B comparison mode (--config-a / --config-b) for parameter
tuning. HybridSearchOpts now accepts rrfK and dedupOpts overrides.

Co-Authored-By: 4shut0sh <4shut0sh@users.noreply.github.com>

* test: search quality tests — RRF boost, dedup guarantee, cosine similarity, E2E benchmark

42 new tests across 3 files:
- test/search.test.ts: RRF normalization, compiled truth 2x boost, dedup key
  collision prevention, cosine similarity edge cases, CJK word count detection
- test/dedup.test.ts: source-aware compiled truth guarantee, layer interactions,
  custom maxPerPage, empty/single result edge cases
- test/e2e/search-quality.test.ts: full pipeline against PGLite with basis vector
  embeddings — chunk_id/chunk_index fields, detail parameter filtering,
  getEmbeddingsByChunkIds, keyword multi-chunk, vector ordering

Also: export rrfFusion + cosineSimilarity for unit testing, fix PGLite
getEmbeddingsByChunkIds to parse string vectors from pgvector.

* test: search quality benchmark with A/B comparison (baseline vs PR#64)

Benchmark measures P@1, MRR, nDCG@5, and source accuracy across 8 queries
against 5 seeded pages. Key finding: boost helps entity lookups but
over-corrects temporal queries. Validates the --detail parameter as the
right control mechanism. Output at docs/benchmarks/2026-04-13.md.

* feat: query intent classifier — auto-selects detail level, 100% source accuracy

Zero-latency heuristic classifier detects query intent from text patterns:
- "Who is Pedro?" → entity → detail=low (compiled truth only)
- "When did we last meet?" → temporal → detail=high (no boost, natural ranking)
- "Variant fund announcement" → event → detail=high
- General queries → detail=medium (default with boost)

The key insight: skip the 2.0x compiled truth boost for detail=high queries.
Temporal/event queries want natural ranking where timeline entries can win.

Benchmark results (source accuracy = does the top chunk match expected type):
- Baseline: 100% (already good, no boost needed)
- Boost only: 71.4% (boost over-corrects temporal queries)
- Boost + intent classifier: 100% (best of both worlds)

35 unit tests for the classifier. 590 total tests pass.

* feat: query intent classifier — auto-selects detail level, 100% source accuracy

Heuristic classifier detects query intent from text patterns (zero latency,
no LLM call). Maps temporal queries ("when did we last meet") to detail=high,
entity queries ("who is X") to detail=low, events to detail=high.

Benchmark results (29 pages, 20 queries, graded relevance):
- Baseline: P@1=0.947, MRR=0.974, source accuracy=89.5%
- Boost only: P@1=0.895, MRR=0.939, source accuracy=63.2% (over-correction)
- Boost + intent: P@1=0.947, MRR=0.974, source accuracy=89.5% (fully recovered)

The intent classifier eliminates the boost's over-correction on temporal queries
while preserving its benefits for entity lookups. 35 unit tests for the classifier.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: search quality benchmark with A/B comparison (baseline vs PR#64)

Rich benchmark: 29 pages, 58 chunks, 20 queries with graded relevance.
Now measures CHUNK-LEVEL quality, not just page-level retrieval.

Key findings (C. Boost+Intent vs A. Baseline):
- Unique pages in top-10: 7.2 → 8.7 (+21% broader coverage)
- Compiled truth ratio: 51.6% → 66.8% (+15pp more signal)
- CT-first rate: 100% (compiled truth leads for entity queries)
- Timeline accessible: 100% (temporal queries still find dates)
- Source accuracy: 89.5% maintained (intent classifier prevents regression)

The boost alone (B) causes -26pp source accuracy regression.
Intent classifier (C) recovers it fully.

* docs: clean benchmark report — ELI10 search quality analysis for PR#64

Replaces two drafts with one clean report. Explains what changed, why it
matters, and what the numbers mean. All fictional data, no private info.

Key findings: 21% more page coverage per query, 29% more compiled truth
in results. Intent classifier prevents boost from burying timeline for
temporal queries. Full per-query breakdown with before/after comparison.

* chore: remove auto-generated benchmark file (clean version is 2026-04-14-search-quality.md)

* docs: update project documentation for search quality boost

CLAUDE.md: added search/intent.ts, search/eval.ts, commands/eval.ts to key
files. Added 5 new test files (search, dedup, intent, eval, e2e/search-quality).
Updated test count from 23+4 to 28+5. Added docs/benchmarks/ to key files.

README.md: updated search pipeline diagram with intent classifier, RRF
normalization, compiled truth boost, cosine re-scoring, and 5-layer dedup.
Added --detail flag explanation and benchmark instructions.

CHANGELOG.md: added search quality entries to v0.9.3 (intent classifier,
--detail flag, gbrain eval, CJK fix). Credited @4shut0sh and @YIING99.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: headline benchmark gains in changelog

* docs: add community attribution rule to CHANGELOG voice section

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: YIING99 <yiing99@users.noreply.github.com>
Co-authored-by: 4shut0sh <4shut0sh@users.noreply.github.com>
 2026-04-13 21:03:40 -10:00
Garry TanGitHubClaude Opus 4.6Gus
 		f82978d38d security: fix wave 2 — 5 vulns + typed health check DSL (v0.9.3) (#95) 
* security: path traversal, query bounds, marker injection fixes

LocalStorage: contained() method validates all paths stay within storage root.
file-resolver: resolveFile validates filePath within brainRoot, marker prefix
rejects ../, absolute paths, bare '..'. file_list: LIMIT 100 on slug-filtered
branch + FILE_LIST_LIMIT constant for both branches.

Co-Authored-By: Gus <garagon@users.noreply.github.com>

* security: symlink hardening in all file walkers

All 4 walkers in files.ts (collectFiles, findRedirects, findAndClean, scan)
plus init.ts counter now use lstatSync + isSymbolicLink skip. Tests import
production collectFiles instead of reimplementing it. node_modules skipped.
CLI file list and verify queries bounded with LIMIT.

Co-Authored-By: Gus <garagon@users.noreply.github.com>

* feat: typed health check DSL + recipe migration

4 DSL types: http, env_exists, command, any_of. Replaces raw execSync
on recipe YAML. All 7 first-party recipes migrated from shell strings
to typed objects. String health_checks still accepted with deprecation
warning + metachar validation for non-embedded recipes. isUnsafeHealthCheck
blocks shell injection for user-created recipes.

Co-Authored-By: Gus <garagon@users.noreply.github.com>

* chore: bump version and changelog (v0.9.3)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: E2E test for file_list LIMIT enforcement against real Postgres

Inserts 150 file rows for one slug, verifies file_list returns at most
100 (both slug-filtered and unfiltered branches). Proves the LIMIT
works at the database level, not just in unit tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Gus <garagon@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-13 07:49:13 -10:00
Garry TanGitHubClaude Opus 4.6
 		3e21e9b69b feat: GBrain v0.6.0 — Remote MCP Server + 12 Bug Fixes (#28) 
* fix: 7 bug fixes from Issue #9 and #22

- fix(mcp): use ListToolsRequestSchema/CallToolRequestSchema instead of string literals (Issue #9, PR #25)
- fix(mcp): handleToolCall reads dry_run from params instead of hardcoding false (#22 Bug #11)
- fix(search): keyword search returns best chunk per page via DISTINCT ON, not all chunks (#22 Bug #8)
- fix(search): dedup layer 1 keeps top 3 chunks per page instead of collapsing to 1 (#22 Bug #12)
- fix(engine): transaction uses scoped engine via Object.create, no shared state mutation (#22 Bug #2)
- fix(engine): upsertChunks uses UPSERT instead of DELETE+INSERT, preserves existing embeddings (#22 Bug #1)
- fix(slugs): validateSlug normalizes to lowercase, pathToSlug lowercases consistently (#22 Bug #4)
- schema: add unique index on content_chunks(page_id, chunk_index) for UPSERT support
- schema: add access_tokens and mcp_request_log tables via migration

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: embed schema.sql at build time, remove fs dependency from initSchema

initSchema() previously read schema.sql from disk at runtime via readFileSync,
which broke in compiled Bun binaries and Deno Edge Functions. Now uses a
generated schema-embedded.ts constant (run `bun run build:schema` to regenerate).

- Removes fs and path imports from postgres-engine.ts and db.ts
- Adds scripts/build-schema.sh for one-source-of-truth generation
- Adds build:schema npm script

Fixes Issue #22 Bug #6.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: 5 more bug fixes from Issue #22

- fix(file_upload): call storage.upload() in all 3 paths (operation, CLI upload, CLI sync) with rollback semantics (#22 Bug #9)
- fix(import): use atomic index counter for parallel queue instead of array.shift() race, preserve checkpoint on errors (#22 Bug #3)
- fix(s3): replace unsigned fetch with @aws-sdk/client-s3 for proper SigV4 auth, supports R2/MinIO via forcePathStyle (#22 Bug #10)
- fix(redirect): verify remote file exists before deleting local copy, skip files not found in storage (#22 Bug #5)
- deps: add @aws-sdk/client-s3

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: remote MCP server via Supabase Edge Functions

Deploy GBrain as a serverless remote MCP endpoint on your existing Supabase
instance. One brain, accessible from Claude Desktop, Claude Code, Cowork,
Perplexity Computer, and any MCP client. Zero new infrastructure.

New files:
- supabase/functions/gbrain-mcp/index.ts — Edge Function with Hono + MCP SDK
- supabase/functions/gbrain-mcp/deno.json — Deno import map
- src/edge-entry.ts — curated bundle entry point (excludes fs-dependent modules)
- src/commands/auth.ts — standalone token management (create/list/revoke/test)
- scripts/deploy-remote.sh — one-script deployment
- .env.production.example — 3-value config template

Changes:
- config.ts: lazy-evaluate CONFIG_DIR (no homedir() at module scope)
- schema.sql: add access_tokens + mcp_request_log tables
- package.json: add build:edge script

Auth: bearer tokens via access_tokens table (SHA-256 hashed, per-client, revocable)
Transport: WebStandardStreamableHTTPServerTransport (stateless, Streamable HTTP)
Health: /health endpoint (unauth: 200/503, auth: postgres/pgvector/openai checks)
Excluded from remote: sync_brain, file_upload (may exceed 60s timeout)

Setup: clone, fill .env.production, run scripts/deploy-remote.sh, create token, done.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: per-client MCP setup guides

- docs/mcp/DEPLOY.md — deployment walkthrough, auth, troubleshooting, latency table
- docs/mcp/CLAUDE_CODE.md — claude mcp add command
- docs/mcp/CLAUDE_DESKTOP.md — Settings > Integrations (NOT JSON config!)
- docs/mcp/CLAUDE_COWORK.md — remote + local bridge paths
- docs/mcp/PERPLEXITY.md — Perplexity Computer connector setup
- docs/mcp/CHATGPT.md — coming soon (requires OAuth 2.1, P0 TODO)
- docs/mcp/ALTERNATIVES.md — Tailscale Funnel + ngrok self-hosted options

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.6.0)

GBrain v0.6.0: Remote MCP server via Supabase Edge Functions + 12 bug fixes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add Remote MCP Server section to README

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: make document-release mandatory in CLAUDE.md, add MCP key files

Post-ship requirements section: document-release is NOT optional. Lists every
file that must be checked on every ship. A ship without updated docs is incomplete.

Also adds remote MCP server files to Key files section.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: batch upsertChunks into single statement to prevent deadlocks

The per-chunk UPSERT loop caused deadlocks under parallel workers because
each INSERT ON CONFLICT acquired row-level locks sequentially. Multiple
workers upserting different pages could deadlock on the shared unique index.

Fix: batch all chunks into a single multi-row INSERT ON CONFLICT statement.
One round-trip, one lock acquisition. COALESCE preserves existing embeddings
when the new value is NULL.

Fixes CI failure: "E2E: Parallel Import > parallel import with --workers 4"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: advisory lock in initSchema() prevents deadlock on concurrent DDL

When multiple processes call initSchema() concurrently (e.g., test setup +
CLI subprocess, or parallel workers during E2E tests), the schema SQL's
DROP TRIGGER + CREATE TRIGGER statements acquire AccessExclusiveLock on
different tables, causing deadlocks.

Fix: pg_advisory_lock(42) serializes all initSchema() calls within the
same database. The lock is session-scoped and released in a finally block.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add explicit test timeouts for CLI subprocess E2E tests

CLI subprocess tests (Setup Journey, Doctor Command, Parallel Import)
spawn `bun run src/cli.ts` which takes several seconds to JIT compile +
connect. The Bun test framework default 5000ms per-test timeout is too
tight for CI. Added 30-60s timeouts matching each subprocess's own
timeout to prevent false failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: infinite recursion in config.ts exported getConfigDir/getConfigPath

The replace_all refactor created recursive functions: the exported
getConfigDir() called the private getConfigDir() which called itself.
Renamed exports to configDir()/configPath() to avoid shadowing.

Also adds scripts/smoke-test-mcp.ts — verified all 8 MCP tool calls
work against a real Postgres database.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-10 15:23:00 -10:00
Garry TanGitHubClaude Opus 4.6
 		27eb87f1f4 feat: slugify file paths with spaces and special characters (v0.5.1) (#29) 
* feat: slugify file paths with spaces and special characters

Apple Notes files (e.g., "2017-05-03 ohmygreen.md") now get clean,
URL-safe slugs instead of raw filenames with spaces. Spaces become
hyphens, special chars are stripped, accented chars normalize to ASCII.

Both import (inferSlug) and sync (pathToSlug) pipelines now use the
same slugifyPath() function, eliminating the case-preservation mismatch.

* feat: one-time migration to slugify existing page slugs

Extends Migration interface with optional TypeScript handler for
application-level data transformations. Adds version 2 migration that
renames all existing slugs to their slugified form, including link
rewriting. Collision handling via try/catch + warning.

* test: slugify unit tests, E2E tests, and updated expectations

22 new unit tests for slugifySegment and slugifyPath covering spaces,
special chars, unicode, dots, empty segments, and all 4 bug report
examples. Updated pathToSlug tests for new lowercase behavior. Updated
E2E tests for slugified Apple Notes slugs. Added 2 new E2E tests for
space-named file import and sync.

* chore: bump version and changelog (v0.5.1)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: fix changelog example to show directory with spaces

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 11:42:32 -07:00
Garry TanGitHubClaude Opus 4.6
 		e9f3c9c24d docs: live sync setup + verification runbook + API key loading (#24) 
* docs: add SKILLPACK Section 18 — Live Sync (MUST ADD)

Contract-first guide for keeping the vector DB in sync with the brain
repo. Documents the pooler prerequisite (Session mode required for
transactions), sync + embed primitives, four example approaches (cron,
--watch, webhook, git hook), isSyncable exclusions, silent skip warning,
and OpenClaw/Hermes cron registration examples.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add GBRAIN_VERIFY.md installation verification runbook

Six-check runbook: schema (doctor), skillpack loaded, auto-update,
live sync (coverage check + embed check + end-to-end push-and-search
test), embedding coverage, brain-first lookup protocol. Emphasizes
"sync ran" != "sync worked" — the real test is searching for corrected
text after a push.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add setup Phases H (Live Sync) and I (Verification)

Phase H: MUST ADD live sync setup — pooler prerequisite check, automatic
sync configuration (agent picks approach), sync+embed chaining, coverage
verification. Phase I: run GBRAIN_VERIFY.md end-to-end before declaring
setup complete.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add install steps 8-9 (live sync + verification)

Step 8: set up automatic sync with SKILLPACK Section 18 reference.
Step 9: run GBRAIN_VERIFY.md runbook. Add GBRAIN_VERIFY.md to docs
section.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add API key loading instructions to CLAUDE.md

Source ~/.zshrc before running Tier 2 tests so OPENAI_API_KEY and
ANTHROPIC_API_KEY are available. Without this, embedding and skills
tests skip silently.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version to v0.5.0

Live sync, verification runbook, API key loading instructions.
Version markers updated in SKILLPACK and RECOMMENDED_SCHEMA.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add anti-hand-roll rule to skill routing in CLAUDE.md

Explicitly prohibit manually running git commit + push + gh pr create
when /ship is available. /ship handles VERSION, CHANGELOG,
document-release, reviews, and coverage audit. Hand-rolling skips
all of these. Added "commit and ship" / "push and ship" variants
to the ship routing rule.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: changelog voice rule + rewrite 0.5.0 changelog to sell the upgrade

CLAUDE.md: add changelog voice guidance — lead with benefits, not
implementation details. Make users want to upgrade.

CHANGELOG: rewrite 0.5.0 entries from dry feature descriptions to
capability-focused bullets ("your brain never falls behind" not
"SKILLPACK Section 18 added").

SKILLPACK Section 17: update the auto-update message template to
instruct agents to sell the upgrade, not just summarize the diff.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add v0.5.0 migration directive for live sync + verification

Agents upgrading from v0.4.x will automatically: check their pooler
connection string, set up automatic sync, and run the verification
runbook. Without this migration file, upgrading agents would learn
about live sync (by re-reading Section 18) but wouldn't set it up.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: sharpen migration file guidance in CLAUDE.md

Replace vague "requires agent action" with concrete trigger list:
new setup steps existing users don't have, MUST ADD skillpack sections,
schema changes, deprecated commands, new verification steps, new crons.
Add the key test: "if an existing user upgrades and does nothing else,
will their brain work worse?"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: make Section 17 upgrade flow work for direct user requests

Section 17 was structured as a cron-initiated flow only. An agent
handling "upgrade gbrain" might just run the command and stop, missing
the post-upgrade steps where the value is (re-read skills, run
migrations, schema sync). Added explicit entry point for direct
upgrade requests. Made Steps 2-4 more concrete about where to find
files and why migrations can't be skipped.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add E2E sync tests — git-to-DB pipeline (11 tests)

Tests the full sync lifecycle against real Postgres+pgvector:
- First sync imports all pages from a git repo
- Second sync with no changes returns up_to_date
- Incremental sync picks up new files (add → commit → sync → verify)
- Incremental sync picks up modifications — THE CRITICAL TEST:
  corrected text appears in DB and keyword search after sync
- Incremental sync handles deletes
- Non-syncable files are excluded (README, .raw/, ops/)
- Sync state (last_commit, last_run) persisted to config
- Sync logged to ingest_log
- --full reimports everything
- --dry-run shows changes without applying

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: strengthen CLAUDE.md to always run ALL test tiers

Replace passive "source zshrc" suggestion with ALWAYS directive.
Explicitly state that "run all tests" means ALL tiers including
Tier 2 with API keys. Do not skip Tier 2 just because keys need
loading.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: Tier 2 E2E tests — correct openclaw CLI invocation

The tests used `openclaw -p` which doesn't exist. The correct command
is `openclaw agent --local --agent <id> --message <prompt>`. Also fixed
JSON output parsing (structured JSON goes to stderr, not stdout — use
non-JSON mode instead). Fixed ingest test to assert on agent response
text rather than test DB state (the agent writes to its own configured
DB, not the ephemeral test DB).

82 tests pass, 0 fail, 0 skip across all 5 E2E files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-10 07:23:59 -10:00
Garry TanGitHubClaude Opus 4.6
 		f541f045d2 feat: add gbrain check-update command and auto-update agent workflow (#15) 
* feat: add `gbrain check-update` command for auto-update notifications

Deterministic collector that checks GitHub Releases for new versions,
compares semver (minor+ only, skips patches), and fetches changelog diffs.
Exports `detectInstallMethod()` from upgrade.ts for reuse. Includes 15
unit tests covering version comparison, CLI wiring, and error handling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add E2E upgrade tests against real GitHub API

Exercises check-update CLI end-to-end: valid JSON output, human-readable
mode, help text, graceful no-releases handling, and version comparison
wiring. Skips gracefully when network is unavailable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add SKILLPACK Section 17 — auto-update notifications

Full agent playbook for the update lifecycle: check, notify, consent,
upgrade, skills refresh, schema sync, report. Includes standalone
self-update for skillpack-only users via version markers and raw
GitHub URL fetching. Adds version markers to both SKILLPACK and
RECOMMENDED_SCHEMA headers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add auto-update step 7 to install paste, setup Phase G, migrations dir

Adds step 7 to the OpenClaw install paste (default-on update checks).
Setup skill gets Phase G (conditional offer for manual installs) and
schema state tracking via ~/.gbrain/update-state.json. Creates
skills/migrations/ directory for version-specific upgrade directives.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md with E2E test DB lifecycle, migration conventions

Adds E2E test DB lifecycle instructions (spin up, run, tear down).
Documents version migration convention (skills/migrations/v[version].md)
and schema state tracking (~/.gbrain/update-state.json). Updates test
file counts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: broken semver comparison in extractChangelogBetween

The version range check compared minor versions without guarding on
major being equal, causing incorrect changelog entries to be captured
(e.g., v0.5.0 would match when upgrading from v1.2.0). Extracted
semverGt/semverLte helpers for correct comparisons. Added 5 tests
for extractChangelogBetween covering cross-major, same-version, and
malformed input cases.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.4.1)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-09 12:25:04 -10:00
Garry TanGitHubClaude Opus 4.6
 		912a321cfa GBrain v0.4.0 — production agent documentation + reference architecture (#10) 
* fix: widen validateSlug to accept any filename characters

Git is the system of record. Slugs are lowercased repo-relative paths.
The restrictive regex rejected spaces, parens, and special chars, blocking
5,861 Apple Notes files from importing. Now only rejects empty slugs,
path traversal (..), and leading slash.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: enable RLS on all tables with BYPASSRLS safety check

Without RLS, the Supabase anon key gives full read access to the DB.
Enable RLS on all 10 tables with no policies — the postgres role
(used by gbrain via pooler) has BYPASSRLS and is unaffected. Only
enables if the current role actually has BYPASSRLS privilege to
avoid locking ourselves out on non-Supabase setups.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: import resilience — 5MB limit, error suppression, structured progress

Raise MAX_FILE_SIZE from 1MB to 5MB for Apple Notes with attachments.
Track error patterns and suppress after 5 identical errors to prevent
5,861 identical warnings from killing the agent process. Replace \r
progress bar with structured log lines (rate, ETA) for agent parsing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: init detects IPv6-only Supabase URLs, adds pgvector check

Detect db.*.supabase.co direct URLs and warn about IPv6 failure.
On ECONNREFUSED/ETIMEDOUT to Supabase, suggest the Session pooler
connection string with exact dashboard click path. Check for pgvector
extension after connecting and fail with clear instructions if missing.
Update wizard hints to show pooler URL format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add pre-ship requirement for E2E tests

E2E tests against real Postgres+pgvector must pass before /ship or
/review. Adds the requirement to CLAUDE.md so all agents enforce it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: parallel import with per-worker engine instances

Refactor PostgresEngine to support instance-level DB connections instead
of only the module-global singleton. Each worker gets its own connection
with poolSize:2 (vs 10 for the main engine), so 8 workers = 16 connections.

Add --workers N flag to gbrain import. Workers pull from a shared queue
and use independent engine instances — no transaction context corruption.

The bottleneck is network round-trips to Supabase (one per page upsert).
Parallel workers cut import time proportionally.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: automatic schema migration runner

Migrations are embedded as string constants in migrate.ts (survives
Bun --compile). Each migration runs in a transaction for clean rollback
on failure. Runs automatically on initSchema() — no manual step needed
when a user updates the gbrain binary against an older DB.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: pluggable storage backend (S3 + Supabase Storage + local)

Add StorageBackend interface with three implementations:
- S3Storage: works with AWS S3, Cloudflare R2, MinIO (any S3-compatible)
- SupabaseStorage: uses Supabase Storage REST API with service role key
- LocalStorage: filesystem-based, for testing

Add file-resolver.ts with fallback chain: local file → .redirect
breadcrumb → .supabase marker → storage backend. Supports the
three-stage migration (mirror → redirect → clean).

Add yaml-lite.ts for parsing marker and breadcrumb files without
adding a YAML dependency.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: gbrain doctor command — health checks with --json output

Checks: connection, pgvector extension, RLS on all tables, schema
version, embedding coverage. Outputs structured JSON with --json flag
for agent parsing. Exit code 0 if healthy, 1 if issues found.

Agents should run gbrain doctor --json when any command fails.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite setup skill + README for agent-first DX

Setup skill: add Why Supabase, step-by-step project creation, explicit
agent instructions (nohup for large imports, doctor on failure, don't
ask for anon key), available init flags, file migration offer after
first import. Remove ClawHub references.

README: simplify to single OpenClaw install path, remove ClawHub, fix
squatted npm name to github:garrytan/gbrain, add Supabase settings
note about Session pooler.

Add Apple Notes test fixtures with spaces and parens in filenames for
E2E testing of the slug fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add RLS verification, schema health, and nohup hints to maintain skill

Maintenance skill now checks RLS status and schema version as part of
periodic health checks. Adds nohup pattern for large embedding refreshes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: import resume checkpoint + Supabase smart URL parsing

Import resume: saves checkpoint every 100 files to ~/.gbrain/import-checkpoint.json.
On restart with same directory and file count, skips already-processed files.
Use --fresh to ignore checkpoint and start over. Cleared on successful completion.

Supabase admin: extractProjectRef() parses any Supabase URL format (dashboard,
direct, pooler, project URL) to extract the project ref. discoverPoolerUrl()
uses the Management API to find the correct pooler connection string (including
the exact region prefix). checkRls() verifies RLS status via the API.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add 56 unit tests for all new code

8 new test files covering every feature added in this branch:
- slug-validation.test.ts: spaces, parens, unicode, path traversal (10 tests)
- yaml-lite.test.ts: parse + stringify, marker/redirect formats (9 tests)
- supabase-admin.test.ts: extractProjectRef for 4 URL formats (7 tests)
- migrate.test.ts: version export, runMigrations callable (2 tests)
- storage.test.ts: LocalStorage CRUD + createStorage factory (14 tests)
- file-resolver.test.ts: fallback chain, redirect, marker parsing (6 tests)
- import-resume.test.ts: checkpoint save/load/resume/fresh (6 tests)
- doctor.test.ts: module export, CLI registration (3 tests)

Total: 184 pass, 0 fail (up from 128).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: bulk chunk INSERT + E2E tests for all new features

Bulk INSERT: upsertChunks now builds a multi-row VALUES query instead
of inserting chunks one-by-one. Reduces DB round-trips by ~50x per page.

E2E tests added to mechanical.test.ts:
- Slug with special chars: import Apple Notes fixtures with spaces/parens,
  verify search finds them, verify idempotency
- RLS verification: check pg_tables.rowsecurity on all tables, verify
  current user has BYPASSRLS
- Doctor command: verify exit 0 on healthy DB, --json produces valid JSON
  with check structure
- Parallel import: --workers 2 produces same page count as sequential

Unit tests added:
- setup-branching.test.ts: IPv6 detection, defaultWorkers auto-tuning,
  smart URL parsing across all Supabase URL formats

Fixtures added:
- large/big-file.md (2.1MB) for testing raised file size limit
- apple-notes/ fixtures already existed

Total: 200 pass, 0 fail (up from 184).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: --json on init/import, file migration CLI, lifecycle tests

--json flag: init and import now support --json for structured output.
Agents get parseable JSON instead of human-readable text.

File migration CLI: implement mirror, unmirror, redirect, restore,
clean, and status subcommands for the three-stage file migration
lifecycle (local → mirrored → redirected → cloud-only).

File migration tests: full lifecycle test covering every transition
in the state machine (LOCAL → MIRROR → UNMIRROR → REDIRECT → RESTORE
→ CLEAN), including edge cases and file resolver at each stage.

Bulk chunk INSERT: upsertChunks now builds multi-row parameterized
VALUES query, reducing round-trips per page from ~50 to 1.

Total: 207 pass, 0 fail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: thorough E2E tests for parallel import concurrency

Replace the weak single-comparison parallel import test with 7 tests:
- Sequential baseline: capture page count, chunk count, and all slugs
- --workers 2: verify page count matches sequential
- Chunk count matches (no duplicates from concurrent writes)
- Page slugs match exactly
- No duplicate pages (SQL GROUP BY HAVING count > 1)
- No duplicate chunks (SQL GROUP BY page_id, chunk_index)
- --workers 4: also works correctly
- Re-import with workers is idempotent

These tests catch the exact bug Codex found (db.ts singleton causing
concurrent transaction corruption) by verifying data integrity after
parallel writes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add batch embedding queue as P1 TODO

Deferred during eng review (per-worker embedding is good enough for now).
Revisit after profiling real imports to confirm embedding is the bottleneck.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: E2E test failures — fixture counts, arg parsing, doctor exit code

Fix fixture count assertions: 13 → 16 pages (added apple-notes + large file),
companies 2 → 3 (ohmygreen), concepts 3 → 5 (notes, big-file).

Fix --workers arg parsing: the worker count value (e.g. "2") was being
picked up as the directory arg. Skip flag values when finding the dir.

Fix doctor exit code: warnings (like missing embeddings) should exit 0,
only actual failures exit 1. E2E tests import with --no-embed, so
embeddings are always WARN.

Fix E2E CLI tests: add initCli() before doctor and parallel import
tests so ~/.gbrain/config.json exists for the subprocess.

All E2E tests pass: 63 pass, 0 fail.
All unit tests pass: 207 pass, 0 fail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.4.0

New CHANGELOG entry for all post-0.3.0 features (doctor, storage backends,
parallel import, resume checkpoints, RLS, schema migrations, --json output).
Version bumped 0.3.0 → 0.4.0 across all manifests.

CLAUDE.md: test count 9→19, skill count 8→7, added key files.
CONTRIBUTING.md: fixture count 13→16, added missing source files.
README.md: added gbrain doctor to commands, fixed stale welcome PRs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add GBRAIN_SKILLPACK.md reference architecture

Production agent patterns from a real deployment with 14,700+ brain files.
Covers: entity detection on every message, brain-first lookup protocol,
7-step enrichment pipeline with tiered API spend, compiled truth + timeline,
source attribution with mandatory citations, meeting ingestion with entity
propagation, cron schedule with quiet hours and travel-aware timezone,
YouTube/media ingestion via Diarize.io, integration guides for ClawVisor,
Circleback webhooks, and Quo/OpenPhone SMS. Opens with the Vannevar Bush
memex framing and the originals folder for capturing intellectual capital.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite README opener with memex pitch and production architecture

Replace code-first opener with mimetic-desire pitch: Vannevar Bush memex
tagline, production brain numbers (10K+ files, 3K+ people, 13 years of
calendar), "ask it anything" examples, compounding thesis.

New sections: The Compounding Thesis (read-write loop), Architecture
(three-column diagram), What a Production Agent Looks Like (SKILLPACK
reference), How gbrain fits with OpenClaw (three-layer complement).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update skills with brain-first lookup, entity detection, heartbeat

setup: Phase D rewritten with brain-first lookup protocol (gbrain search
→ query → get → grep fallback), sync-after-write rule, memory_search
complement table.

query: token-budget awareness (chunks not full pages), source precedence
hierarchy (user > compiled truth > timeline > external).

ingest: entity detection on every message (scan, check brain, create or
enrich, commit and sync).

maintain: heartbeat integration (doctor, embed --stale, sync verification,
stale compiled truth detection).

briefing: gbrain-native context loading (search attendees before meetings,
search sender before email, daily deal/meeting/commitment queries).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add OpenClaw positioning to README opener

Make it clear up top that GBrain is built for OpenClaw agents and
works with any OpenClaw deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: credit Karpathy's Knowledge LLM vision, add origin story

GBrain started as Karpathy's LLM wiki idea built for real. Worked great
until the brain hit thousands of files and grep fell apart. GBrain is the
search layer that had to exist once the brain outgrew grep.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-09 10:17:13 -07:00
Garry TanGitHubClaude Opus 4.6
 		a86f995883 feat: GBrain v0.3.0 — contract-first architecture + ClawHub plugin (#7) 
* feat: contract-first operations.ts with OperationError, dry_run, importFromContent

30 shared operations as single source of truth for CLI and MCP.
- OperationError with typed error codes (page_not_found, invalid_params, etc.)
- dry_run support on all mutating operations
- importFromContent split from importFile with transaction wrapping
- Idempotency hash now includes ALL fields (title, type, frontmatter, tags)
- Config env var fallback: GBRAIN_DATABASE_URL > DATABASE_URL > config file

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: rewrite MCP server + CLI + tools-json from operations

server.ts: 233 -> ~80 lines. Tool definitions and dispatch generated from operations[].
cli.ts: shared operations auto-registered, CLI-only commands kept as manual dispatch.
tools-json: generated FROM operations[], eliminating the third contract surface.
Parity test verifies structural contract between operations, CLI, and MCP.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: delete 12 command files migrated to operations.ts

Handler logic for get, put, delete, list, search, query, health, stats,
tags, link, timeline, and version now lives in operations.ts.
Kept: init, upgrade, import, export, files, embed, sync, serve, call, config.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: init --non-interactive, upgrade verification, schema migration

- gbrain init --non-interactive --url <url> for plugin mode (no TTY required)
- Post-upgrade version verification in gbrain upgrade
- Drop storage_url from files table (storage_path is the only identifier)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: tool-agnostic skills + new setup skill

All 7 skills rewritten with intent-based language instead of CLI commands.
Works with both CLI and MCP plugin contexts.
New setup skill replaces install: auto-provision Supabase via CLI,
AGENTS.md injection, target TTHW < 2 min.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: ClawHub bundle plugin, CI workflows, v0.3.0

- openclaw.plugin.json with configSchema, MCP server config, skill listing
- GitHub Actions: test on push/PR, multi-platform release (macOS arm64 + Linux x64)
- Version bump 0.3.0, CHANGELOG, README ClawHub section, CLAUDE.md updated

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: idempotency hash mismatch + MCP dry_run passthrough

importFromContent now passes its all-fields hash through putPage via
content_hash on PageInput, so the stored hash matches the computed hash.
Previously the skip-if-unchanged check never fired because the hash
formulas differed.

MCP server now passes dry_run from tool params to OperationContext.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.3.0.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: schema loader handles PL/pgSQL $$ blocks

Delete the semicolon-based SQL splitter in db.ts which broke on
PL/pgSQL trigger functions containing semicolons inside $$ delimiter
blocks. Use single conn.unsafe(schemaSql) call instead — the postgres
driver handles multi-statement SQL natively. schema.sql already uses
IF NOT EXISTS / CREATE OR REPLACE for idempotency.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: E2E test infrastructure + realistic brain fixtures

Add test infrastructure for running E2E tests against real
Postgres+pgvector. Includes:
- test/e2e/helpers.ts: DB lifecycle, fixture import, timing, diagnostics
- 13 fixture files as a miniature realistic brain (people, companies,
  deals, meetings, concepts, projects, sources) following the
  compiled truth + timeline format from GBRAIN_RECOMMENDED_SCHEMA.md
- docker-compose.test.yml: local pgvector convenience (port 5433)
- .env.testing.example: template for test credentials
- package.json: add test:e2e script

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: E2E test suites + CI workflow

Tier 1 (mechanical.test.ts): 14 test suites covering all operations
against real Postgres — page CRUD, search with quality scoring, links,
tags, timeline, versions, admin, chunks, resolution, ingest log, raw
data, files, idempotency stress, setup journey (full CLI flow), init
edge cases, schema idempotency, schema diff guard, performance baselines.

Tier 1 (mcp.test.ts): MCP protocol test — spawns server, sends JSON-RPC,
verifies tools/list matches operations count.

Tier 2 (skills.test.ts): OpenClaw skill tests — ingest, query, health.
Skips gracefully when dependencies missing.

CI (.github/workflows/e2e.yml): Tier 1 on every PR (pgvector service),
Tier 2 nightly/manual with API key secrets.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: E2E test fixes + traverseGraph jsonb cast

- Fix traverseGraph query: cast json_agg to jsonb_agg so SELECT DISTINCT works
- Fix put_page tests to use importFromContent with noEmbed (no OpenAI key in Tier 1)
- Fix get_health assertion (page_count not total_pages)
- Fix raw_data test to handle JSONB string/object return
- Simplify MCP test to verify tool generation directly
- Add timeouts to CLI subprocess tests
- Use port 5434 for docker-compose (5433 often in use)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: update all project docs for E2E test suite

- CLAUDE.md: updated test count (9 unit + 3 E2E), added E2E test
  instructions, fixed skill count to 8
- CONTRIBUTING.md: updated project structure with test/e2e/, added E2E
  test instructions, rewrote "Adding a new command" to reflect
  contract-first architecture (add to operations.ts, done)
- README.md: fixed table count (10 not 9), added recommended schema doc
  to Docs section, added E2E instructions to Contributing section
- CHANGELOG.md: added E2E test suite, docker-compose, schema loader fix,
  and traverseGraph jsonb fix to v0.3.0 entry

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-08 23:26:11 -10:00