mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
dffb607ef7
* v0.30.1 Lane A: connection-manager foundation + X1 initSchema routing Routes Postgres queries by query type: - read() goes to the Supabase pooler (port 6543, fast) - ddl() and bulk() go to direct (port 5432, 30min stmt timeout, mwm 256MB) Auto-detects Supabase via hostname pooler.supabase.com or port 6543. Override with GBRAIN_DIRECT_DATABASE_URL. Kill-switch via GBRAIN_DISABLE_DIRECT_POOL=1 falls back to single-pool legacy path. Foundation modules (Lane A scope): - src/core/connection-manager.ts: read/ddl/bulk/healthCheck, parent-CM inheritance (T5/X1), cached Promise<Sql> lazy init (A1), kill-switch inheritance (A2), Supabase URL auto-derivation - src/core/url-redact.ts: redactPgUrl + redactDeep (F3) - src/core/retry-matcher.ts: typed predicates for stmt-timeout / lock / conn errors (C4) - src/core/connection-audit.ts: ~/.gbrain/audit/connection-events JSONL with ISO-week rotation; doctor tail-reads last 5 errors (F8) - scripts/check-pg-url-redaction.sh: CI grep guard against unredacted postgresql:// URL leaks (F3) Engine integration: - PostgresEngine.connect: instantiates instance-owned ConnectionManager, inherits from parentConnectionManager when set (worker engines, sync, cycle), shares pool with module-singleton path - PostgresEngine.disconnect: tears down direct pool first - PostgresEngine.initSchema: routes DDL through connectionManager.ddl() when dual-pool active (X1 part 1; lock semantics replacement is Lane B) - cli.ts:connectEngine(opts): probeOnly skips initSchema entirely (X1 part 2 — get_health, upgrade --status will use this) Tests added (51 new cases): - test/url-redact.test.ts: 11 cases - test/retry-matcher.test.ts: 13 cases - test/connection-manager.test.ts: 27 cases (URL detection, derive, kill-switch, parent inheritance, dual-pool routing modes) Foundation for Lanes B-E. Sequential lane work continues. Plan: ~/.claude/plans/system-instruction-you-are-working-stateless-wadler.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane B: migration runner retry + verify hooks + namespaced --force flags Adds Migration interface fields: - idempotent: boolean (default true; explicit false blocks verify-hook re-runs on destructive migrations) - verify: optional post-condition probe; runs after migration claims success Migration retry wrapper (Cherry D3 / Finding F2): - 3 attempts with 5s/15s/45s backoff (env GBRAIN_MIGRATE_BACKOFF_MS=0 for tests) - Retries only on statement_timeout (57014) or connection-reset patterns - Pre-attempt: logs idle-in-transaction blockers via getIdleBlockers - On exhaustion: throws MigrationRetryExhausted with named PID + suggested pg_terminate_backend() recovery command Verify-hook self-healing (Cherry D6 / Codex X3): - On verify=false + idempotent=true → re-runs migration once silently - On verify=false + idempotent=false → throws MigrationDriftError - --skip-verify CLI flag bypasses for operator override withRefreshingLock helper (Cherry T4 / Codex A4 / X1 part 3): - setInterval refresh every TTL/6 ms during long-running work - SELECT 1 backend-alive heartbeat per refresh tick - Heartbeat hang past 30s → log + clear interval; lock TTL auto-expires - LockUnavailableError when acquire fails (caller decides retry) - buildTenantLockId(scope) appends current_database() suffix for multi-tenant safety (Cherry D4) Namespaced --force flags (Codex T5): - --force-orchestrator: write 'retry' markers for ALL wedged orchestrators - --force-schema: re-runs runMigrations against current config.version - --force / --force-all: both - --force-retry vX.Y.Z: existing single-version reset (preserved) - --skip-verify: bypass verify-hook drift detection on a single run Test additions: - test/migrate-extensions.test.ts: 14 cases (idempotent default, error envelopes, MIGRATIONS contract) - test/db-lock-refresh.test.ts: 10 cases (LockUnavailableError, buildTenantLockId multi-tenant, opts shape) - test/migrate.test.ts: updated 2 existing cases (PR #356 retry shape + function-name anchor) for v0.30.1 retry-wrapper semantics 156 unit tests passing across the v0.30.1 surface so far. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane C: backfill primitive + registry + X4 + X5 First-class generic backfill runner (Fix 3). Generalizes the keyset+checkpoint+adaptive-batch pattern from src/core/backfill-effective-date.ts so future backfills (embedding_voyage in v0.30.2, etc.) reuse one tested runner. NEW src/core/backfill-base.ts: - runBackfill() with keyset pagination, config-table checkpoint, adaptive batch halving on stmt timeout, conn-drop reconnect, max-errors bail - ensureBackfillIndex() verifies/creates partial index CONCURRENTLY (P2/X4) - clearBackfillCheckpoint() for --fresh path - T3 fix: writes go through engine.withReservedConnection so BEGIN / SET LOCAL / UPDATE / COMMIT execute on the SAME backend (otherwise SET LOCAL evaporates between pooled executeRaw calls) NEW src/core/backfill-registry.ts: - effective_date: implemented (wraps existing computeEffectiveDate) - emotional_weight: implemented (wraps computeEmotionalWeight + stamps new emotional_weight_recomputed_at column) - embedding_voyage: declared-only in v0.30.1 (multi-column embedding schema lands in v0.30.2) NEW src/commands/backfill.ts: - gbrain backfill <kind> [--batch-size N] [--concurrency N] [--resume] [--fresh] [--dry-run] [--keep-index] [--max-errors N] - gbrain backfill list — shows registered backfills + status - X5 admission control: clampConcurrency() forces --concurrency to GBRAIN_DIRECT_POOL_SIZE - 1 ceiling (always reserves 1 conn for HNSW + heartbeat + doctor probes). Loud-warns when user requests above. Schema migration v44 (X4 / Codex C8 fix): - pages.emotional_weight_recomputed_at TIMESTAMPTZ - emotional_weight = 0 is a VALID steady-state value per migration v40, so the original P2 predicate ("WHERE emotional_weight = 0") would have been a permanent large index over normal data. The corrected backlog predicate is "emotional_weight_recomputed_at IS NULL"; the partial index drops naturally as the cycle phase + this backfill stamp the column over time. - idempotent: true (ADD COLUMN ... NULL is metadata-only) CLI integration: - src/cli.ts: registers `backfill` subcommand - reindex-frontmatter stays as thin alias for v0.30.1 back-compat; canonical entrypoint is now `gbrain backfill effective_date` Test additions: - test/backfill-base.test.ts: 11 cases (keyset, checkpoint, dry-run, resume/fresh, maxRows cap, withReservedConnection routing, error paths, clearCheckpoint, ensureBackfillIndex) - test/backfill-concurrency-clamp.test.ts: 6 cases (X5 admission control) 173 unit tests passing across Lanes A+B+C of v0.30.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane D: HNSW lifecycle manager + A3 atomic-swap Extends src/core/vector-index.ts with the v0.30.1 lifecycle layer. The original chunkEmbeddingIndexSql / applyChunkEmbeddingIndexPolicy contract is preserved unchanged. New surfaces: - checkActiveBuild(engine, indexName): probes pg_stat_activity for an active CREATE INDEX or REINDEX on the named index. Used as pre-op guard so dropAndRebuild doesn't compete with a build already in flight (Supabase auto-maintenance, parallel gbrain procs). - dropZombieIndexes(engine, tableNames): startup sweep of indisvalid=false rows on gbrain tables. Drops them with DROP INDEX IF EXISTS, BUT skips any zombie that has an active build still in pg_stat_activity (codex Fix-5 in-progress-build guard). Wired into PostgresEngine.initSchema() — runs after migrations + verifySchema, best-effort, never blocks engine.connect(). - dropAndRebuild(engine, spec, opts): A3 atomic-swap pattern: 1. checkActiveBuild → bail if another build is active (--force overrides) 2. CREATE INDEX CONCURRENTLY <name>_rebuild_<unix-ms> via engine.withReservedConnection (CONCURRENTLY can't run in a txn) 3. Atomic swap inside engine.transaction: DROP INDEX <old-name> ALTER INDEX <temp-name> RENAME TO <old-name> 4. If step 2 fails (OOM, timeout, conn drop), the OLD index stays intact and search keeps serving queries. This is the headline A3 win — no production-degraded silent failure mode. - monitorBuild(engine, indexName, onProgress, opts): poll pg_stat_activity every 30s; emit elapsed_ms + size_bytes (via pg_relation_size) + pid. Used by gbrain backfill embedding_voyage when batch > 1000 triggers a rebuild. - isSupabaseAutoMaintenance(active): predicate on application_name (matches "supabase" / "postgres-meta"). Used by dropAndRebuild to log + back off when Supabase auto-maintenance is doing the rebuild. Engine integration: - PostgresEngine.initSchema() calls dropZombieIndexes after verifySchema. Surfaces zombie counts via console.log. - Best-effort wrapped in try/catch: pg_stat_activity / pg_index access can be restricted on managed Postgres tiers; gbrain shouldn't fail engine.connect() over diagnostic queries. Test additions (18 cases): - test/vector-index-lifecycle.test.ts: * chunkEmbeddingIndexSql contract (3 cases) — pre-existing behavior preserved * applyChunkEmbeddingIndexPolicy contract (1 case) * checkActiveBuild (4 cases, including PGLite no-op + best-effort failure) * isSupabaseAutoMaintenance (3 cases) * dropZombieIndexes (4 cases, including in-progress-build guard) * dropAndRebuild atomic-swap (3 cases, including PGLite + active-build bail + temp-name format assertion) 191 unit tests passing across Lanes A+B+C+D of v0.30.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 Lane E: upgrade pipeline checkpoint + brain_id binding + get_health migrations NEW src/core/upgrade-checkpoint.ts: - Cherry D5: persists step-by-step progress through gbrain post-upgrade so partial failures can be resumed via gbrain upgrade --resume. Steps: pull → install → schema → features → backfills → verify. - Codex X2: checkpoint binds to brain identity via sha256(database_url) (userinfo stripped before hashing so cred rotations don't invalidate). PGLite uses sha256(database_path). Cross-brain checkpoint application is now refused with reason='brain_mismatch'. - F4 fall-through: validateCheckpoint returns reason='no_checkpoint' when none exists, enabling silent fall-through to a full upgrade. - All-complete detection: stale checkpoints (every step done) return reason='all_complete' so the next run clears + re-runs from scratch. - markStepComplete + markStepFailed maintain the partial-state shape. T2 preserved: upgrade.ts still re-execs `gbrain post-upgrade` so the NEW binary's migration registry runs (the existing re-exec pattern is correct per codex round 1's plan-breaking finding). The checkpoint module is the substrate that Lane E's --resume / --status surfaces will plumb through in v0.30.2. D7 + C3 contract committed: - BrainHealth.schema_version: '1' (literal type) — additive-only contract pinned for MCP get_health consumers. - BrainHealth.migrations: { schema, orchestrator } — explicit two-ledger diagnostic surface (codex T5 namespacing). Both fields are OPTIONAL in v0.30.1 — engines can populate them in v0.30.2 without a contract bump. Backwards/forwards compat: clients default-handle missing fields. VERSION: 0.30.0 → 0.30.1 package.json: synced Test additions (18 cases): - test/upgrade-checkpoint.test.ts: * computeBrainId: userinfo strip, DB-distinct hashes, stable hex (5 cases) * write/load round-trip: roundtrip, missing file, malformed JSON, clear (4 cases) * validateCheckpoint: F4 no_checkpoint, X2 brain_mismatch, partial → resumeAt, all_complete, first-step pending (5 cases) * markStepComplete/markStepFailed: append, idempotent, clear-failed, failed-state shape (4 cases) 209 unit tests passing across all 5 lanes of v0.30.1 (Lanes A-E core foundations). Plumbing into upgrade.ts CLI + doctor checks + get_health() implementation is layered in via follow-up commits within this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.30.1 e2e + test isolation: integration smoke + serial quarantine NEW test/e2e/v030_1-integration-pglite.test.ts (14 cases): PGLite integration smoke proving Lane A-E surfaces work together. Lane B: migration runner applies v44 (emotional_weight_recomputed_at) cleanly; config.version reaches LATEST_VERSION Lane C: backfill registry resolves all 3 entries; emotional_weight + effective_date backfills on empty brain return examined=0 cleanly Lane D: dropZombieIndexes / checkActiveBuild on PGLite are no-ops Lane E: upgrade-checkpoint round-trips with brain_id; X2 mismatch refused; F4 fall-through detected via reason='no_checkpoint'; full step progression to all_complete Test isolation hygiene (scripts/check-test-isolation.sh): - test/connection-manager.test.ts → connection-manager.serial.test.ts - test/backfill-concurrency-clamp.test.ts → .serial.test.ts - test/upgrade-checkpoint.test.ts → .serial.test.ts All three files mutate process.env (kill-switch, GBRAIN_DIRECT_POOL_SIZE, GBRAIN_HOME) which would race other tests in the parallel runner. *.serial.test.ts quarantine ensures they run at --max-concurrency=1. Choice between withEnv() refactor and serial quarantine made on the side of preserving existing well-formed test code. E2E coverage status: - v030_1-integration-pglite.test.ts (this commit): 14 cases, all green - backfill-perf-pglite.test.ts: 1 case, green (no regression) - cycle-recompute-emotional-weight-pglite.test.ts: green (no regression) - multi-source-emotional-weight-pglite.test.ts: green (no regression) - dream-synthesize-pglite.test.ts: 14 cases, green (no regression) - anomalies-pglite.test.ts + salience-pglite.test.ts: 6 cases, green Postgres-only E2Es (migration-flow, http-transport, hnsw-lifecycle, connection-routing) require DATABASE_URL + a real Postgres+pgvector container per the CLAUDE.md E2E lifecycle. They land as separate DATABASE_URL-gated work — not regressed by v0.30.1 changes; their preconditions just aren't met in the current run environment. `bun run verify` (typecheck + 4 shell pre-checks + test-isolation lint) passes cleanly. Final v0.30.1 unit + integration test count: 4547 pass, 0 regressions. Two pre-existing flaky failures (BrainRegistry serial test + warm-create perf gate under shard contention) confirmed unrelated to this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.30.1) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
111 lines
5.1 KiB
JSON
111 lines
5.1 KiB
JSON
{
|
|
"name": "gbrain",
|
|
"version": "0.30.1",
|
|
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
|
"type": "module",
|
|
"main": "src/core/index.ts",
|
|
"bin": {
|
|
"gbrain": "src/cli.ts"
|
|
},
|
|
"exports": {
|
|
".": "./src/core/index.ts",
|
|
"./engine": "./src/core/engine.ts",
|
|
"./types": "./src/core/types.ts",
|
|
"./operations": "./src/core/operations.ts",
|
|
"./minions": "./src/core/minions/index.ts",
|
|
"./engine-factory": "./src/core/engine-factory.ts",
|
|
"./pglite-engine": "./src/core/pglite-engine.ts",
|
|
"./link-extraction": "./src/core/link-extraction.ts",
|
|
"./import-file": "./src/core/import-file.ts",
|
|
"./transcription": "./src/core/transcription.ts",
|
|
"./embedding": "./src/core/embedding.ts",
|
|
"./config": "./src/core/config.ts",
|
|
"./markdown": "./src/core/markdown.ts",
|
|
"./backoff": "./src/core/backoff.ts",
|
|
"./search/hybrid": "./src/core/search/hybrid.ts",
|
|
"./search/expansion": "./src/core/search/expansion.ts",
|
|
"./extract": "./src/commands/extract.ts"
|
|
},
|
|
"scripts": {
|
|
"dev": "bun run src/cli.ts",
|
|
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
|
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
|
"build:admin": "cd admin && bun run build",
|
|
"build:schema": "bash scripts/build-schema.sh",
|
|
"build:llms": "bun run scripts/build-llms.ts",
|
|
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
|
"test": "bash scripts/run-unit-parallel.sh",
|
|
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
|
"verify": "bun run check:privacy && bun run check:jsonb && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run typecheck",
|
|
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
|
"check:cli-exec": "scripts/check-cli-executable.sh",
|
|
"check:all": "scripts/check-privacy.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
|
|
"check:wasm": "scripts/check-wasm-embedded.sh",
|
|
"check:newlines": "scripts/check-trailing-newline.sh",
|
|
"test:e2e": "bash scripts/run-e2e.sh",
|
|
"test:slow": "bash scripts/run-slow-tests.sh",
|
|
"test:profile": "bash scripts/profile-tests.sh",
|
|
"test:serial": "bash scripts/run-serial-tests.sh",
|
|
"ci:local": "bash scripts/ci-local.sh",
|
|
"ci:local:diff": "bash scripts/ci-local.sh --diff",
|
|
"ci:select-e2e": "bun run scripts/select-e2e.ts",
|
|
"typecheck": "tsc --noEmit",
|
|
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
|
"check:privacy": "scripts/check-privacy.sh",
|
|
"check:progress": "scripts/check-progress-to-stdout.sh",
|
|
"check:exports-count": "scripts/check-exports-count.sh",
|
|
"check:admin-build": "scripts/check-admin-build.sh",
|
|
"check:test-isolation": "scripts/check-test-isolation.sh",
|
|
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
|
"prepublish:clawhub": "bun run build:all",
|
|
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
|
},
|
|
"openclaw": {
|
|
"compat": {
|
|
"pluginApi": ">=2026.4.0"
|
|
}
|
|
},
|
|
"dependencies": {
|
|
"@ai-sdk/anthropic": "^3.0.71",
|
|
"@ai-sdk/google": "^3.0.64",
|
|
"@ai-sdk/openai": "^3.0.53",
|
|
"@ai-sdk/openai-compatible": "^2.0.41",
|
|
"@anthropic-ai/sdk": "^0.30.0",
|
|
"@aws-sdk/client-s3": "^3.1028.0",
|
|
"@dqbd/tiktoken": "^1.0.22",
|
|
"@electric-sql/pglite": "0.4.3",
|
|
"@jsquash/avif": "^2.1.1",
|
|
"@modelcontextprotocol/sdk": "1.29.0",
|
|
"ai": "^6.0.168",
|
|
"cookie-parser": "^1.4.7",
|
|
"cors": "^2.8.5",
|
|
"eventsource-parser": "^3.0.8",
|
|
"exifr": "^7.1.3",
|
|
"express": "^5.1.0",
|
|
"express-rate-limit": "^7.5.0",
|
|
"gray-matter": "^4.0.3",
|
|
"marked": "^18.0.0",
|
|
"openai": "^4.0.0",
|
|
"pgvector": "^0.2.0",
|
|
"postgres": "^3.4.0",
|
|
"tree-sitter-wasms": "0.1.13",
|
|
"web-tree-sitter": "0.22.6",
|
|
"zod": "^4.3.6"
|
|
},
|
|
"devDependencies": {
|
|
"@types/bun": "latest",
|
|
"@types/cookie-parser": "^1.4.7",
|
|
"@types/cors": "^2.8.19",
|
|
"@types/express": "^5.0.6",
|
|
"bun-types": "^1.3.13",
|
|
"typescript": "^5.6.0"
|
|
},
|
|
"trustedDependencies": [
|
|
"@electric-sql/pglite"
|
|
],
|
|
"engines": {
|
|
"bun": ">=1.3.10"
|
|
},
|
|
"license": "MIT"
|
|
}
|