mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
8ab733471b
* feat(worker-pool): shared sliding pool + bounded semaphore + PGLite-clamp wrapper T1 + T2 of the v0.41.16.0 workers cathedral. New src/core/worker-pool.ts is the canonical primitive every --workers N bulk command in this wave (and future bulk commands) builds on. Atomic-claim invariant enforced by scripts/check-worker-pool-atomicity.sh (wired into bun run verify). BudgetExhausted bypass + AbortSignal composition baked into the helper so budget caps are a structural ceiling under concurrency, not a per-caller convention. The new resolveWorkersWithClamp wrapper composes existing autoConcurrency with PGLite-clamp + per-(command, requested) stderr dedup. Deliberately NOT a modification to shared autoConcurrency (silent today, used by sync + import); embed.ts keeps GBRAIN_EMBED_CONCURRENCY || 20 default per codex #13. 23 + 12 + 9 = 44 hermetic tests pin every contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: structural + dim-check regression suites for v0.41.16.0 wave - test/embed-helper-migration.test.ts (T3): asserts embed.ts's two sliding-pool sites are migrated to runSlidingPool, pre-migration shapes (let nextIdx = 0, Promise.all(Array.from(...))) are gone, GBRAIN_EMBED_CONCURRENCY || 20 default preserved, failureLabel threads page.slug. Per codex #16/#17 these are invariant assertions, not byte-equality on progress event ORDERING. - test/embedding-dim-check-facts.test.ts (T6): readFactsEmbeddingDim covers vector(N) + halfvec(N), halfvec-before-vector regex ordering pinned (codex #19), buildFactsAlterRecipe emits DROP INDEX + ALTER USING + CREATE INDEX (codex #18, not bare REINDEX), FactsEmbeddingDimMismatchError tagged class shape, assertFactsEmbeddingDimMatchesConfig PGLite skip + Postgres absent- column skip, doctor check + insert-cast wiring assertions. - test/extract-conversation-facts-workers.test.ts (T5): helper exports (extractConversationFactsLockId, PER_PAGE_LOCK_TTL_MINUTES), structural wiring (runSlidingPool, resolveWorkersWithClamp, withRefreshingLock, LockUnavailableError, delete-orphans-first before segment loop, preflight before pool, exit 3 when lock_skipped > 0), Minion handler round-trip. - test/extract-workers.test.ts (T7): --workers wiring on all 3 inner fs-walk loops (extractForSlugs, extractLinksFromDir, extractTimelineFromDir) + CLI parse + opts threading through runExtractCore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.41.16.0 → v0.41.17.0 (queue collision with PR #1510) PR #1510 (garrytan/dynamic-regex-conversation-formats) claimed v0.41.16.0 on master in parallel. Advancing this wave to v0.41.17.0 so both can land cleanly. Pure mechanical version bump: - VERSION + package.json → 0.41.17.0 - CHANGELOG.md header + "To take advantage of v0.41.17.0" block - TODOS.md section header + v0.41.18+ forward references - CLAUDE.md inline version tags - Regenerated llms-full.txt / llms.txt No code changes. The actual workers cathedral feature set is unchanged from the two prior commits in this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): search-image-column probes column dim at runtime CI shard 5 failed on `searchVector column routing (v0.27.1)` with: error: expected 1280 dimensions, not 1536 The test had a hardcoded `fakeText1536` helper that seeded chunks at 1536-d vectors. Master's default embedding model switched from OpenAI text-embedding-3-large (1536) to ZeroEntropy zembed-1 (1280) so a fresh PGLite brain on CI now sizes content_chunks.embedding at 1280; the test's 1536-d INSERT trips pgvector's CheckExpectedDim. Fix: probe `content_chunks.embedding` width via `readContentChunksEmbeddingDim(engine)` in `beforeAll`, store in `TEXT_DIM`, and build `fakeTextDefault(seed)` at that width. The test now passes regardless of which default ships (the model has flipped twice and may flip again). Local dev (1536 from older config) and CI fresh-install (1280 from new default) both pass. Image-side vectors stay at 1024 (matches Voyage multimodal-3 + the column's fixed width on the image side). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): bump PGLite hook timeout for shard-4 deep-process files facts-anti-loop.test.ts and ingest-capture.test.ts were timing out in CI shard 4 with "beforeEach/afterEach hook timed out" after the v0.41.16.0 master merge brought migration count to 99. When these files run deep in a shard process that has already created ~20 PGLite engines, the WASM cold-start + 95-migration replay legitimately exceeds bun's 5s default hook timeout (observed 5.6s and 7.3s locally when reproducing). Bun's --timeout=60000 from scripts/test-shard.sh covers TEST timeouts but NOT hook timeouts; those default to 5s and must be set per-hook via the optional 2nd arg to beforeAll/afterAll. Reproduced locally by running the first 21 shard-4 files via head -21 /tmp/shard4-list.txt | xargs bun test → 179 pass, 2 fail (both with hook-timeout error) After fix: → 198 pass, 0 fail (the 4 anti-loop + 15 ingest-capture tests recover) Full shard 4 with fix: 955 pass, 0 fail. Full shard 5 with fix: 1261 pass, 0 fail. Also added a defensive diagnostic to the two put_page tests: if facts_backstop is missing in the response payload, throw with the full payload + isError so future failures surface the actual handler error instead of a bare "expected {...} got undefined" assertion. No-op when the test passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
142 lines
7.0 KiB
JSON
142 lines
7.0 KiB
JSON
{
|
|
"name": "gbrain",
|
|
"version": "0.41.17.0",
|
|
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
|
"type": "module",
|
|
"main": "src/core/index.ts",
|
|
"bin": {
|
|
"gbrain": "src/cli.ts"
|
|
},
|
|
"exports": {
|
|
".": "./src/core/index.ts",
|
|
"./engine": "./src/core/engine.ts",
|
|
"./types": "./src/core/types.ts",
|
|
"./operations": "./src/core/operations.ts",
|
|
"./minions": "./src/core/minions/index.ts",
|
|
"./engine-factory": "./src/core/engine-factory.ts",
|
|
"./pglite-engine": "./src/core/pglite-engine.ts",
|
|
"./link-extraction": "./src/core/link-extraction.ts",
|
|
"./import-file": "./src/core/import-file.ts",
|
|
"./transcription": "./src/core/transcription.ts",
|
|
"./embedding": "./src/core/embedding.ts",
|
|
"./config": "./src/core/config.ts",
|
|
"./markdown": "./src/core/markdown.ts",
|
|
"./backoff": "./src/core/backoff.ts",
|
|
"./search/hybrid": "./src/core/search/hybrid.ts",
|
|
"./search/expansion": "./src/core/search/expansion.ts",
|
|
"./ai/gateway": "./src/core/ai/gateway.ts",
|
|
"./extract": "./src/commands/extract.ts",
|
|
"./ingestion": "./src/core/ingestion/index.ts",
|
|
"./ingestion/test-harness": "./src/core/ingestion/test-harness.ts"
|
|
},
|
|
"scripts": {
|
|
"dev": "bun run src/cli.ts",
|
|
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
|
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
|
"build:admin": "cd admin && bun run build && cd .. && bun run scripts/build-admin-embedded.ts",
|
|
"build:admin-embedded": "bun run scripts/build-admin-embedded.ts",
|
|
"build:schema": "bash scripts/build-schema.sh",
|
|
"build:llms": "bun run scripts/build-llms.ts",
|
|
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
|
"test": "bash scripts/run-unit-parallel.sh",
|
|
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
|
"verify": "bash scripts/run-verify-parallel.sh",
|
|
"check:source-config-leak": "scripts/check-source-config-leak.sh",
|
|
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
|
|
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
|
"check:system-of-record": "scripts/check-system-of-record.sh",
|
|
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
|
"check:cli-exec": "scripts/check-cli-executable.sh",
|
|
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh",
|
|
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
|
|
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
|
|
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
|
|
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
|
|
"check:wasm": "scripts/check-wasm-embedded.sh",
|
|
"check:newlines": "scripts/check-trailing-newline.sh",
|
|
"test:e2e": "bash scripts/run-e2e.sh",
|
|
"test:slow": "bash scripts/run-slow-tests.sh",
|
|
"test:heavy": "bash scripts/run-heavy.sh",
|
|
"test:profile": "bash scripts/profile-tests.sh",
|
|
"test:serial": "bash scripts/run-serial-tests.sh",
|
|
"ci:local": "bash scripts/ci-local.sh",
|
|
"ci:local:diff": "bash scripts/ci-local.sh --diff",
|
|
"ci:select-e2e": "bun run scripts/select-e2e.ts",
|
|
"typecheck": "tsc --noEmit",
|
|
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
|
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
|
"check:privacy": "scripts/check-privacy.sh",
|
|
"check:proposal-pii": "scripts/check-proposal-pii.sh",
|
|
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
|
"check:test-names": "scripts/check-test-real-names.sh",
|
|
"check:progress": "scripts/check-progress-to-stdout.sh",
|
|
"check:exports-count": "scripts/check-exports-count.sh",
|
|
"check:admin-build": "scripts/check-admin-build.sh",
|
|
"check:admin-embedded": "scripts/check-admin-embedded.sh",
|
|
"check:test-isolation": "scripts/check-test-isolation.sh",
|
|
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
|
|
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
|
|
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
|
|
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
|
|
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
|
"prepublish:clawhub": "bun run build:all",
|
|
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
|
},
|
|
"openclaw": {
|
|
"compat": {
|
|
"pluginApi": ">=2026.4.0"
|
|
},
|
|
"extensions": [
|
|
"./src/openclaw-context-engine.ts"
|
|
]
|
|
},
|
|
"dependencies": {
|
|
"@ai-sdk/anthropic": "^3.0.71",
|
|
"@ai-sdk/google": "^3.0.64",
|
|
"@ai-sdk/openai": "^3.0.53",
|
|
"@ai-sdk/openai-compatible": "^2.0.41",
|
|
"@anthropic-ai/sdk": "^0.30.0",
|
|
"@aws-sdk/client-s3": "^3.1028.0",
|
|
"@dqbd/tiktoken": "^1.0.22",
|
|
"@electric-sql/pglite": "0.4.3",
|
|
"@jsquash/avif": "^2.1.1",
|
|
"@jsquash/png": "^3.1.1",
|
|
"@modelcontextprotocol/sdk": "1.29.0",
|
|
"ai": "^6.0.168",
|
|
"chokidar": "^4.0.3",
|
|
"cookie-parser": "^1.4.7",
|
|
"cors": "^2.8.5",
|
|
"eventsource-parser": "^3.0.8",
|
|
"exifr": "^7.1.3",
|
|
"express": "^5.1.0",
|
|
"express-rate-limit": "^7.5.0",
|
|
"gray-matter": "^4.0.3",
|
|
"heic-decode": "^2.1.0",
|
|
"js-yaml": "^3.14.2",
|
|
"marked": "^18.0.0",
|
|
"openai": "^4.0.0",
|
|
"pgvector": "^0.2.0",
|
|
"postgres": "^3.4.0",
|
|
"tree-sitter-wasms": "0.1.13",
|
|
"web-tree-sitter": "0.22.6",
|
|
"zod": "^4.3.6"
|
|
},
|
|
"devDependencies": {
|
|
"@types/bun": "latest",
|
|
"@types/cookie-parser": "^1.4.7",
|
|
"@types/cors": "^2.8.19",
|
|
"@types/express": "^5.0.6",
|
|
"@types/js-yaml": "^3.12.10",
|
|
"bun-types": "^1.3.13",
|
|
"fast-check": "^4.8.0",
|
|
"typescript": "^5.6.0"
|
|
},
|
|
"trustedDependencies": [
|
|
"@electric-sql/pglite"
|
|
],
|
|
"engines": {
|
|
"bun": ">=1.3.10"
|
|
},
|
|
"license": "MIT"
|
|
}
|