mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
f0825018ddeec8b69a8985eca4d75cd7d6f4bf18
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
08746b06d2 |
v0.22.9 feat: structured error code summary for sync --skip-failed (#501)
* feat: structured error code summary for sync --skip-failed (#500) When sync encounters per-file failures, the blocked/skip-failed messages now include a breakdown by error code (SLUG_MISMATCH, YAML_PARSE, etc.) instead of just a raw count. This makes it immediately obvious *why* files failed without requiring manual investigation. Changes: - Add classifyErrorCode() — maps error messages to ParseValidationCode - Add summarizeFailuresByCode() — groups failures into sorted code summary - SyncFailure now carries a 'code' field (backfilled on acknowledge) - acknowledgeSyncFailures() returns AcknowledgeResult {count, summary} - sync blocked + skip-failed messages show code breakdown - doctor sync_failures check shows code breakdown for both unacked and historical - 12 new tests for classifyErrorCode, summarizeFailuresByCode, and structured returns Before: Sync blocked: 2688 file(s) failed to parse. After: Sync blocked: 2688 file(s) failed to parse: SLUG_MISMATCH: 2685 YAML_DUPLICATE_KEY: 3 Closes #500 * fix: eng-review fixes for sync error-code classification - Reorder classifyErrorCode() so DB-layer errors (DB_DUPLICATE_KEY, STATEMENT_TIMEOUT) check BEFORE YAML patterns. Postgres "duplicate key value violates unique constraint" no longer mislabels as YAML_DUPLICATE_KEY. - Rewrite MISSING_OPEN/MISSING_CLOSE/EMPTY_FRONTMATTER/NULL_BYTES/NESTED_QUOTES regexes to match the canonical messages emitted by collectValidationErrors() in src/core/markdown.ts. Previous patterns (e.g. /missing.*open/i) never fired because the upstream throw site emits prose ("File is empty...", "No closing --- delimiter found"), not the code name. - Extract formatCodeBreakdown() helper that accepts either raw failures or pre-summarized {code, count}[] input. Replaces 3 duplicate inline builders in src/commands/sync.ts. - 15 new tests (37/37 pass on test/sync-failures.test.ts): - DB vs YAML duplicate-key disambiguation (3 cases) - Canonical-message coverage for the 5 frontmatter codes (7 cases) - acknowledgeSyncFailures() legacy-entry backfill branch (2 cases) - formatCodeBreakdown() dual-input shape (3 cases) - TODOS.md: file 3 follow-ups (P2 plumb structured ParseValidationCode; P0-at-ship CHANGELOG migration note for AcknowledgeResult; P3 concurrent- safe ack of sync-failures.jsonl). Eng-review plan: ~/.claude/plans/then-codex-synchronous-toucan.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.22.9) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: 16-core runner + 4-way matrix shard for test job The unit test suite ran 22m17s on ubuntu-latest (2-core/7GB) because: - 187 test files run with bun test parallelism bounded by core count - 23 of those files spin up a fresh PGLiteEngine + initSchema in beforeEach, paying ~22s WASM cold-start per test on the small runner This commit fixes the runner side: - runs-on: ubuntu-latest-16-cores (16 vCPU / 64 GB RAM) - strategy.matrix.shard splits 4 parallel jobs, each running ~40 of 158 unit test files. Single-file wall-time floor is ~3 min after the test refactor, so 4 shards × 16 cores hits the floor quickly without wasting cores past it. - pre-test gates (typecheck, check-jsonb, check-progress, check-wasm) only run on shard 1 — they're not test files and don't benefit from sharding. scripts/test-shard.sh partitions test files by stable FNV-1a hash mod N. Same file always lands in the same shard, so retries are reproducible. Pure shell, portable to bash 3.2 (macOS) and bash 5.x (CI). Excludes test/e2e/ which runs via bun run test:e2e separately and needs DATABASE_URL. Also: ignore .claude/ harness state files (scheduled_tasks.lock etc) instead of just .claude/skills/. Cost: ~$0.19/run vs $0 (public repo, default runner is free). At 50 PRs/month that's ~$10/month for ~5x faster CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: refactor top-3 PGLite-heavy files to share one engine per file Three test files were spinning up a fresh PGLiteEngine + connect + initSchema in beforeEach. PGLite WASM cold-start is ~22s on the small CI runner; doing this per test multiplied wall-time across the suite. The 3 files alone accounted for ~6.5 min of the 22m CI run (177s + 132s + 87s). Refactor: move PGLite setup to beforeAll (one engine per file), wipe data in beforeEach via the new test/helpers/reset-pglite.ts helper. The reset helper: - TRUNCATEs every public table CASCADE, including sources (so tests that register their own sources don't leak rows into the next test). - Re-seeds the default source row that pages.source_id's DEFAULT FKs against. Without this, the next page insert would fail FK validation. - Preserves schema_version so migration helpers don't think the brain is on v0. Files refactored: - test/extract-incremental.test.ts (8 tests, was 177s on CI) - test/brain-writer.test.ts (16 tests; only the scanBrainSources block uses PGLite, was 132s on CI) - test/sync.test.ts (37 tests; only the performSync dry-run block uses PGLite, was 87s on CI) All 61 tests still pass locally. The remaining 20 PGLite-heavy files use the same beforeEach anti-pattern; this commit only refactors the proven worst offenders. Sweep the rest in a follow-up if CI numbers indicate it's worth it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: fall back to ubuntu-latest for matrix shard The ubuntu-latest-16-cores label requires a provisioned larger-runner pool in repo/org settings. Without that setup, jobs queue indefinitely waiting for a runner that doesn't exist (verified: 4 shards stuck in 'queued' status with empty runner_name for 5+ min). Drop back to the default 2-core ubuntu-latest. The 4-way matrix shard still delivers ~5-6x speedup via parallelism alone — 4 jobs running in parallel, each handling ~40 of 158 unit test files. Cost stays $0 (default runner is free for public repos). If we ever provision a larger-runner pool, flip this label back to ubuntu-latest-16-cores. The matrix + sharder will use the bigger boxes unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
96178d726e |
fix(subagent): v0.16.3 — bind Anthropic SDK correctly + enable tsc in CI (#318)
* fix(subagent): bind Anthropic SDK messages.create() correctly The makeSubagentHandler was casting `new Anthropic()` directly to MessagesClient, but MessagesClient.create() maps to sdk.messages.create(), not sdk.create(). Every subagent job immediately died with: client.create is not a function Fix: wrap the SDK instance so .create() delegates to .messages.create() with proper `this` binding via .bind(sdk.messages). Discovered on first production run of gbrain agent against Supabase. Co-Authored-By: Wintermute <wintermute@openclaw.ai> * chore(ci): add typescript typecheck to test pipeline + clean up baseline errors Root cause infra gap that let the v0.16.0 subagent bug ship: CI ran only `bun test`, which transpiles types without checking them. Type errors only surfaced at runtime, in production. Changes: - Add `typescript` devDep and a `typecheck` npm script (`tsc --noEmit`). - Chain `bun run typecheck` into `bun run test` so developers get the same pipeline locally that CI runs. - Flip `.github/workflows/test.yml` to invoke `bun run test` (the npm script, including typecheck) instead of `bun test` (runner only). - Clean up 100+ pre-existing type errors across 30+ files so the first run of `tsc --noEmit` is green. Root causes were: - `databaseUrl` → `database_url` rename drift in test fixtures (9 files) - `PageType` union missing `'meeting'` / `'note'` entries that are already used in both src and tests (link-extraction.ts comments acknowledged the gap) - `GBrainConfig.storage` field never declared despite being read in files.ts and operations.ts - `ErrorCode` union missing `'permission_denied'` - `OrchestratorOpts` shape changed; test callers not updated - Dead-code comparisons in migration orchestrators against narrowed status types - postgres.js `Row`-callback type drift on several `.map()` calls - Buffer-as-BodyInit assignment in supabase.ts (real but non-fatal runtime bug; Uint8Array slice works and is type-correct) - Various `as X` single-step casts that now need `as unknown as X` per TS's stricter structural-conversion rules - Bump `beforeAll` hook timeout to 30s on four PGLite-heavy tests that were flaky under parallel test execution: wait-for-completion, extract-fs, e2e/search-quality, e2e/graph-quality. All pass in isolation; timeouts only happened when dozens of PGLite instances init'd simultaneously. The new CI pipeline now fails on any type error across src/ or test/, giving us the compile-time regression guard the subagent fix depends on. * fix(subagent): bind Anthropic SDK messages.create() correctly Shipped bug: v0.16.0 cast `new Anthropic()` to `MessagesClient`, but `.create()` lives at `sdk.messages.create`, not on the top-level client. Every subagent job in production died on first LLM call with `client.create is not a function`. Discovered on the first `gbrain agent run` against Supabase. Fix: assign `sdk.messages` directly to the `MessagesClient` slot. `sdk.messages` IS the object with a callable `.create()`; the original bug was picking the wrong entry point on the SDK. No helper, no wrapper, no `.bind()` — JS method-call semantics preserve `this` at the call site because `subagent.ts:336` invokes `client.create(...)` with `client === sdk.messages`. The one-line assignment also typechecks cleanly against the existing `MessagesClient` interface (SDK's first `create` overload: `(MessageCreateParamsNonStreaming, Core.RequestOptions?) => APIPromise<Message>` is assignable structurally). This gives us compile-time regression protection: anyone reverting to `new Anthropic()` would fail tsc because `Anthropic` has no top-level `.create`. (The companion chore commit puts `tsc --noEmit` in CI so this guard is enforced.) Also adds a `makeAnthropic?: () => Anthropic` dep-injection seam so the factory default construction branch is testable without real API calls. Regression test drives one handler turn through a fake SDK, asserting `sdk.messages.create` is actually called. If someone later reverts to `new Anthropic()`, both guards fire: tsc fails AND the test fails. Co-Authored-By: Wintermute <wintermute@garrytan.com> * chore(tests): add bunfig.toml + 60s hook timeouts to stabilize PGLite-heavy suites After turning on tsc in CI (previous commit), running the full `bun run test` suite in one shot triggered flaky `beforeEach/afterEach hook timed out` failures on 8+ test files. Every failure traced to PGLite WASM init contention when many test files spin up fresh PGLite instances in parallel; each one alone passes in isolation. - `bunfig.toml` sets the global test hook timeout to 60s (default is 5s), covering every test file without per-file edits. - Individual `beforeAll(fn, 60_000)` / `beforeEach(fn, 15_000)` calls on the 8 tests that flaked most stay in place as explicit safety nets so a future bunfig config change doesn't silently re-introduce the flake. Result: 1997 pass, 0 fail on `bun run test` (117 tests added since the prior baseline by picking up typecheck-gated passes). No infrastructure flake tolerated in CI. * chore: bump version and changelog (v0.16.3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Wintermute <wintermute@openclaw.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
eb218a96ad |
security: pin GitHub Actions, add gitleaks CI, harden permissions (v0.4.2) (#23)
* security: pin GitHub Actions to commit SHAs, add gitleaks CI - Pin all 5 actions (checkout, setup-bun, upload-artifact, download-artifact, action-gh-release) to commit SHAs across 3 workflow files - Add permissions: contents: read to test.yml and e2e.yml - Add gitleaks secret scanning job to test.yml - Pin openclaw install to v2026.4.9 in e2e.yml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: add .gitleaks.toml config Allowlists test fixtures, example env files, and skill documentation to prevent false positives from the gitleaks CI step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add GitHub Actions SHA maintenance rule to CLAUDE.md Instructs /ship and /review to check for stale SHA pins and update them, keeping action versions fresh without manual effort. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add S3 Sig V4 TODO from CSO audit Deferred from security audit. S3 storage backend accepts credentials but sends unsigned requests. Implement when S3 becomes a real deployment path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.4.2) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
a86f995883 |
feat: GBrain v0.3.0 — contract-first architecture + ClawHub plugin (#7)
* feat: contract-first operations.ts with OperationError, dry_run, importFromContent 30 shared operations as single source of truth for CLI and MCP. - OperationError with typed error codes (page_not_found, invalid_params, etc.) - dry_run support on all mutating operations - importFromContent split from importFile with transaction wrapping - Idempotency hash now includes ALL fields (title, type, frontmatter, tags) - Config env var fallback: GBRAIN_DATABASE_URL > DATABASE_URL > config file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: rewrite MCP server + CLI + tools-json from operations server.ts: 233 -> ~80 lines. Tool definitions and dispatch generated from operations[]. cli.ts: shared operations auto-registered, CLI-only commands kept as manual dispatch. tools-json: generated FROM operations[], eliminating the third contract surface. Parity test verifies structural contract between operations, CLI, and MCP. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: delete 12 command files migrated to operations.ts Handler logic for get, put, delete, list, search, query, health, stats, tags, link, timeline, and version now lives in operations.ts. Kept: init, upgrade, import, export, files, embed, sync, serve, call, config. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: init --non-interactive, upgrade verification, schema migration - gbrain init --non-interactive --url <url> for plugin mode (no TTY required) - Post-upgrade version verification in gbrain upgrade - Drop storage_url from files table (storage_path is the only identifier) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: tool-agnostic skills + new setup skill All 7 skills rewritten with intent-based language instead of CLI commands. Works with both CLI and MCP plugin contexts. New setup skill replaces install: auto-provision Supabase via CLI, AGENTS.md injection, target TTHW < 2 min. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: ClawHub bundle plugin, CI workflows, v0.3.0 - openclaw.plugin.json with configSchema, MCP server config, skill listing - GitHub Actions: test on push/PR, multi-platform release (macOS arm64 + Linux x64) - Version bump 0.3.0, CHANGELOG, README ClawHub section, CLAUDE.md updated Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: idempotency hash mismatch + MCP dry_run passthrough importFromContent now passes its all-fields hash through putPage via content_hash on PageInput, so the stored hash matches the computed hash. Previously the skip-if-unchanged check never fired because the hash formulas differed. MCP server now passes dry_run from tool params to OperationContext. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.3.0.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: schema loader handles PL/pgSQL $$ blocks Delete the semicolon-based SQL splitter in db.ts which broke on PL/pgSQL trigger functions containing semicolons inside $$ delimiter blocks. Use single conn.unsafe(schemaSql) call instead — the postgres driver handles multi-statement SQL natively. schema.sql already uses IF NOT EXISTS / CREATE OR REPLACE for idempotency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: E2E test infrastructure + realistic brain fixtures Add test infrastructure for running E2E tests against real Postgres+pgvector. Includes: - test/e2e/helpers.ts: DB lifecycle, fixture import, timing, diagnostics - 13 fixture files as a miniature realistic brain (people, companies, deals, meetings, concepts, projects, sources) following the compiled truth + timeline format from GBRAIN_RECOMMENDED_SCHEMA.md - docker-compose.test.yml: local pgvector convenience (port 5433) - .env.testing.example: template for test credentials - package.json: add test:e2e script Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: E2E test suites + CI workflow Tier 1 (mechanical.test.ts): 14 test suites covering all operations against real Postgres — page CRUD, search with quality scoring, links, tags, timeline, versions, admin, chunks, resolution, ingest log, raw data, files, idempotency stress, setup journey (full CLI flow), init edge cases, schema idempotency, schema diff guard, performance baselines. Tier 1 (mcp.test.ts): MCP protocol test — spawns server, sends JSON-RPC, verifies tools/list matches operations count. Tier 2 (skills.test.ts): OpenClaw skill tests — ingest, query, health. Skips gracefully when dependencies missing. CI (.github/workflows/e2e.yml): Tier 1 on every PR (pgvector service), Tier 2 nightly/manual with API key secrets. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: E2E test fixes + traverseGraph jsonb cast - Fix traverseGraph query: cast json_agg to jsonb_agg so SELECT DISTINCT works - Fix put_page tests to use importFromContent with noEmbed (no OpenAI key in Tier 1) - Fix get_health assertion (page_count not total_pages) - Fix raw_data test to handle JSONB string/object return - Simplify MCP test to verify tool generation directly - Add timeouts to CLI subprocess tests - Use port 5434 for docker-compose (5433 often in use) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: update all project docs for E2E test suite - CLAUDE.md: updated test count (9 unit + 3 E2E), added E2E test instructions, fixed skill count to 8 - CONTRIBUTING.md: updated project structure with test/e2e/, added E2E test instructions, rewrote "Adding a new command" to reflect contract-first architecture (add to operations.ts, done) - README.md: fixed table count (10 not 9), added recommended schema doc to Docs section, added E2E instructions to Contributing section - CHANGELOG.md: added E2E test suite, docker-compose, schema loader fix, and traverseGraph jsonb fix to v0.3.0 entry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |