admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
48e1000306a879051c3bbadd8fb36f93da290547
gbrain/package.json
T
Garry TanGitHubClaude Opus 4.7
 48e1000306 v0.41.23.0 feat: extract operator surfaces + pack-driven extractables (#1541) 
* Wave A: schema + receipts foundation for v0.42 extract operator surfaces

Foundation layer for the pack-driven extractables + receipt-as-brain-memory
+ operator-discoverability cathedral. Five atomic pieces ship together
because their schema + helpers + module dependencies are tight-coupled:

A1. Widen pack manifest's `extractable` from `boolean` to
    `boolean | ExtractableSpec`. ExtractableSpec carries prompt_template,
    fixture_corpus, eval_dimensions, benchmark_min_recall, and reserves
    verifier_path for v0.43+ pack-shipped verifier code (REFUSE at
    runtime in v0.42 per plan D-EXTRACT-37). Back-compat: every pre-v0.42
    pack with `extractable: true` continues parsing unchanged. Three new
    helpers: extractableSpecsFromPack(), getExtractableSpec(),
    refuseVerifierPathInV042().

A2. New page type `extract_receipt` in ALL_PAGE_TYPES. Source-boost map
    adds `extracts/` prefix at factor 0.3 — receipts surface in search
    when extraction-relevant but never dominate user content (D-EXTRACT-42).

A3. New module src/core/extract/receipt-writer.ts (~190 LOC) exporting
    writeReceipt(engine, input). Canonical slug shape
    extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N} per
    D-EXTRACT-17. Frontmatter belt+suspenders per D-EXTRACT-19: BOTH
    type:extract_receipt AND dream_generated:true stamped on every
    receipt, regardless of caller, so the eligibility predicate's
    anti-loop guards reject the receipt page from any future extraction
    sweep (single-flag bypass requires breaking two unrelated checks).
    Idempotent on resume — same run_id+round overwrites cleanly.

A4. Migration v104 creates extract_rollup_7d table (per-day rollup of
    extract events keyed on kind+source_id+day). Audit JSONL stays the
    SOURCE OF TRUTH per F-OUT-19; this table is a best-effort cache for
    doctor's <100ms read budget. Per-day rows mean the 7-day window
    auto-evicts on every read. v100 was deliberately skipped on master
    (renumbered out during a prior wave); v101/v102/v103 also taken;
    v104 is the next clean slot.

A5. Doctor `extract_health` check reads extract_rollup_7d for last 7
    days and emits per-kind aggregates: cost_7d_usd, eval_pass_count,
    eval_fail_count, halt_count, round_completed_count, halt_rate.
    3-state: OK when rollup empty (pre-v0.42 brain or fresh init), WARN
    when any per-kind halt rate > 10% (top-3 named in message), WARN
    when rollup_write_failures > 0 (audit JSONL is SoT but operator
    deserves to know the DB cache is degraded). Pre-v104 brains stay
    quiet — the missing-table error path is caught and treated as
    OK so doctor doesn't warn during the upgrade window.

Tests added:
- test/extractable-spec-widening.test.ts (22 cases) — back-compat with
  boolean shape, new struct parsing, verifier_path REFUSE contract.
- test/extract/receipt-writer.test.ts (12 cases) — slug shape, frontmatter
  belt+suspenders, idempotent resume, body human-readability.
- test/doctor-extract-health.test.ts (8 cases) — empty rollup OK, halt
  rate WARN, rollup_write_failures WARN, 7-day window inclusion at
  boundary, multi-kind top-3 message ordering.

Plus the canonical bootstrap-coverage test passes with the new v104
migration cleanly applied through both engines.

Plan: ~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md
Wave A scope. Wave B (hook receipts into existing extractors) follows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Wave B: hook receipts + rollup row into the 5 shipped extractors

Each LLM-backed extractor surface now records its run in two places when
something actually happened:

1. An extract receipt PAGE at extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}
   (queryable via gbrain search, citable, surfaces in cross-modal
   contradiction probes per the Wave A foundation). Only written when
   `total_rows > 0` so no-op runs don't bloat the brain.

2. An UPSERT row in extract_rollup_7d (DB-backed best-effort cache
   per F-OUT-19) so the doctor extract_health check from Wave A reads
   per-kind aggregates without scanning JSONL.

New module src/core/extract/rollup-writer.ts (~120 LOC) exports
upsertExtractRollup() with PostgreSQL ON CONFLICT DO UPDATE on the
(kind, source_id, day) PK. Concurrency-safe per F-OUT-14 design.
Failure path is best-effort — bumps rollup_write_failures in the
table itself, stderr-warns once per (kind, day, error-class), and
NEVER fails the parent extraction operation. JSONL remains source
of truth.

Wired into 5 extractors:
- extract-conversation-facts (kind: facts.conversation) — both
  success path AND BudgetExhausted halt path write receipt+rollup
  so partial runs are still observable.
- extract_atoms cycle phase (kind: atoms)
- synthesize_concepts cycle phase (kind: concepts, source_id: default
  because concepts are brain-global)
- propose_takes cycle phase (kind: takes.proposed) — scope-aware
  source_id from the read scope.
- extract_facts cycle phase (kind: facts.fence) — deterministic
  (no LLM cost) but still records reconcile activity so doctor sees
  the cycle is alive.

Receipt frontmatter belt+suspenders (D-EXTRACT-19) reused from
Wave A: every receipt stamps BOTH `type: extract_receipt` AND
`dream_generated: true` so the eligibility predicate's anti-loop
guards reject the receipt page from any future extraction sweep.

Test surgery in test/propose-takes.test.ts — one existing assertion
tightened from "no INSERTs" to "no INSERT INTO take_proposals" so
the new rollup UPSERT doesn't falsely fail the cache-hit case test.

Run regression: 85/85 tests pass across extract-conversation-facts,
extract-atoms-synthesize-concepts, extract-facts-phase, propose-takes.

Plan: ~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md
Wave B scope. Wave C (pack-author scaffolding + benchmark) follows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Wave C+D: pack-author scaffolding + operator surfaces for v0.42 extract

Wave C: pack-author authoring loop
  - scaffold-extractable mutation primitive declares a kind as extractable
    on a pack manifest in one verb (wires through updateTypeOnPack from
    the v0.41 mutate library); generates 5 placeholder fixtures + a
    pack-supplied prompt template stub
  - schema CLI wires gbrain schema scaffold-extractable <type> --pack <pack>
  - extract benchmark CLI loads a pack's fixture corpus through strict
    D-EXTRACT-21 path validation (rejects absolute paths, .. traversal,
    null bytes, symlinks resolving outside pack root); v0.42 ships as a
    stub reporter (LLM dispatch deferred to Wave E)

Wave D: operator surfaces
  - extract status CLI reads extract_rollup_7d for the last 7 days,
    sorts by (halt_rate desc, cost desc); kubectl-style right-aligned
    table, top-5 + "more rows" hint by default, --verbose shows all;
    stable schema_version: 1 JSON envelope for monitoring pipelines
  - extract --explain <kind> CLI prints the active pack's resolution
    chain: declaration source (pack-declared vs built-in cycle phase),
    prompt_template + fixture_corpus paths with existence checks,
    eval_dimensions, benchmark_min_recall, and the last 7d rollup
  - extract.ts gains a lifecycle-grouped help text (Extraction /
    Inspection / Status) per the original D3 plan goal

Tests:
  - test/schema-pack/scaffold-extractable.test.ts (15 cases) including
    explicit privacy-rule assertions guarding against real-name leakage
  - test/extract/benchmark.test.ts (17 cases) covering path validation
    rejections + JSONL fixture parsing
  - test/extract/status.test.ts (15 cases) over pure aggregation +
    formatting

Housekeeping:
  - test/extract/receipt-writer.test.ts refactored to the canonical
    PGLite block (beforeAll/afterAll/resetPgliteState in beforeEach)
    per CLAUDE.md test-isolation R3+R4; runtime drops from ~30s of
    99-migration replay per test to <6s for all 12 cases together

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.42.0.0: extract operator surfaces + pack-driven extractables

Bump VERSION + package.json to 0.42.0.0. CHANGELOG entry covers the
three-wave shipped scope (receipts + rollup + doctor check; receipts
hooked into all 5 shipped extractors; pack-author scaffolding +
benchmark stub-reporter; status + --explain dashboards + lifecycle
help). CLAUDE.md Key Files gains a v0.42 cluster annotation. llms.txt
regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.41.23.0: re-tag from 0.42.0.0 (patch-channel slot, no scope change)

VERSION + package.json + CHANGELOG header + CLAUDE.md cluster annotation
all moved from 0.42.0.0 to 0.41.23.0. Body text updated in-place: every
"v0.42" / "v0.43+" reference inside this entry's release notes now reads
"v0.41.23" or "follow-up release" as appropriate.

Same scope shipping — the three-wave extract operator surface stays
intact. Just lands in the patch-channel queue (.20/.21/.23 free; .22 is
PR #1542's type-unification cathedral) instead of the minor-channel bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: add extract_receipt to gbrain-base.yaml page_types (CI parity gate)

CI shard 5 caught the drift: test/regressions/gbrain-base-equivalence.test.ts
asserts every ALL_PAGE_TYPES seed has a matching page_type entry in the
gbrain-base.yaml pack. Wave A added `extract_receipt` to ALL_PAGE_TYPES
but didn't seed it in the base pack manifest.

Adds the entry under the `annotation` primitive with `extracts/` path
prefix (matches the source-boost demote site) and `extractable: false`
(receipts are written by the framework, never extracted from). Comment
documents the belt+suspenders D-EXTRACT-19 invariant so future readers
understand why receipts carry both `dream_generated: true` AND
`type: extract_receipt`.

Closes the CI gate without changing runtime behavior — the pack-aware
read paths already had the prefix demote wired in src/core/search/source-boost.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: bump gbrain-base page-type count 24→25 in schema-cli test

CI shard 4 caught the second drift from the same root cause as the
prior parity-gate fix: v0.41.23's `extract_receipt` addition bumped
gbrain-base.yaml from 24 to 25 page types. The schema-cli smoke test
was pinned at 24 (the count after v0.41.11.0 added `conversation` +
`atom`); update to 25 and note v0.41.23's contribution alongside the
prior version stamp.

Verified hermetic: running test/schema-cli.test.ts with a clean
GBRAIN_HOME tempdir produces 12/12 pass (the local-machine 'schema
active' fail is from a real ~/.gbrain pinning gbrain-base-v2; not a
shipped-code issue, doesn't repro on CI).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): pack-locator stub leak between shard 6 test files

CI shard 6 caught three flaky failures in test/onboard-pack-upgrade-checks.test.ts:
  - checkPackUpgradeAvailable > fires on gbrain-base brain with gbrain-base-v2
  - checkPackUpgradeAvailable > manual_only routing via render.ts allowlist (D17)
  - checkTypeProliferation > warns when distinct types exceed declared+5

Root cause: test/schema-pack-sync.test.ts calls
`__setPackLocatorForTests(...)` to stub the disk-loader, but doesn't
restore in afterAll. Bun's CI shard 6 loads multiple test files into
one process; when sync.test.ts runs before onboard-pack-upgrade-checks.test.ts,
the stubbed locator persists at module scope. `loadActivePack` for
gbrain-base / gbrain-base-v2 then returns null and:
  - findPackSuccessors returns [] → status='ok' instead of 'warn' (F1+F2)
  - declared falls back to 15 → fail threshold becomes 30, 32 > 30 → 'fail'
    instead of 'warn' (F3)

Local single-file runs pass because the locator starts at its default.

Two-layer fix:
  1. test/schema-pack-sync.test.ts afterAll calls
     `_resetPackLocatorForTests()` to undo the mutation (the canonical
     fix at the source).
  2. test/onboard-pack-upgrade-checks.test.ts beforeEach calls the same
     reset (defense-in-depth against any future test file in the shard
     that forgets to restore).

Reproduced locally: running the three shard-6 schema-pack files together
fails 3 tests pre-fix and passes 30/30 post-fix. Full shard 6 sweep
(77 files, 1232 tests) now green; bun run verify still 28/28.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): pglite-engine — dim-agnostic chunk-embedding test data

CI shard 6 caught two flaky failures in test/pglite-engine.test.ts:
  - PGLiteEngine: Chunks > getChunksWithEmbeddings returns embedding data
  - PGLiteEngine: stale chunk pagination > countStaleChunks counts chunks
    with NULL embedding only

Both failed with `expected 1280 dimensions, not 1536` at the upsert site.

Root cause: pglite-engine.ts:287 initSchema() reads embedding dim from
gw.getEmbeddingDimensions() if the gateway is configured (potentially
left in that state by another shard-6 test file in the same bun process),
falling back to DEFAULT_EMBEDDING_DIMENSIONS otherwise — which is 1280
since v0.36+ when the ZE default landed (zeroentropyai:zembed-1).
Pre-v0.36 defaults were OpenAI's 1536; my test data was pinned to that
stale literal.

The two outcomes that pass:
  - gateway happens to be configured for 1536-dim (e.g. master shard 6
    run 26515999465 — these tests passed at 20ms + 24ms with no
    "dimensions" error)
  - gateway happens to be configured for 1280-dim AND test data is 1280

The outcome that fails:
  - gateway configured for 1280-dim AND test data hardcoded to 1536

Fix: capture the actual column width after initSchema (probe
pg_attribute.atttypmod for content_chunks.embedding) and use that
captured `CHUNK_EMBED_DIM` constant at the three Float32Array sites.
Test data now matches whatever width the column was created at,
regardless of which shard-6 file ran first.

Local repro: full shard 6 (77 files, 1232 tests, ~6min) green; this
file standalone (100 tests) green; bun run verify 28/28.

Broader pattern: 9 other test files use the same Float32Array(1536)
literal. None land in shard 6 today (so they don't flake), but the
fix shape here can be lifted into a shared helper if the bug class
surfaces elsewhere — filed as a v0.42+ follow-up rather than a
preemptive sweep, since each file's setup shape is slightly different.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 08:27:05 -07:00

145 lines
7.3 KiB
JSON
Raw Blame History

{
"name": "gbrain",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
"bin": {
"gbrain": "src/cli.ts"
},
"exports": {
".": "./src/core/index.ts",
"./engine": "./src/core/engine.ts",
"./types": "./src/core/types.ts",
"./operations": "./src/core/operations.ts",
"./minions": "./src/core/minions/index.ts",
"./engine-factory": "./src/core/engine-factory.ts",
"./pglite-engine": "./src/core/pglite-engine.ts",
"./link-extraction": "./src/core/link-extraction.ts",
"./import-file": "./src/core/import-file.ts",
"./transcription": "./src/core/transcription.ts",
"./embedding": "./src/core/embedding.ts",
"./config": "./src/core/config.ts",
"./markdown": "./src/core/markdown.ts",
"./backoff": "./src/core/backoff.ts",
"./search/hybrid": "./src/core/search/hybrid.ts",
"./search/expansion": "./src/core/search/expansion.ts",
"./ai/gateway": "./src/core/ai/gateway.ts",
"./extract": "./src/commands/extract.ts",
"./ingestion": "./src/core/ingestion/index.ts",
"./ingestion/test-harness": "./src/core/ingestion/test-harness.ts"
},
"scripts": {
"dev": "bun run src/cli.ts",
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
"build:admin": "cd admin && bun run build && cd .. && bun run scripts/build-admin-embedded.ts",
"build:admin-embedded": "bun run scripts/build-admin-embedded.ts",
"build:schema": "bash scripts/build-schema.sh",
"build:llms": "bun run scripts/build-llms.ts",
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "bash scripts/run-unit-parallel.sh",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bash scripts/run-verify-parallel.sh",
"check:source-config-leak": "scripts/check-source-config-leak.sh",
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
"check:system-of-record": "scripts/check-system-of-record.sh",
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
"check:cli-exec": "scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
"test:slow": "bash scripts/run-slow-tests.sh",
"test:heavy": "bash scripts/run-heavy.sh",
"test:profile": "bash scripts/profile-tests.sh",
"test:serial": "bash scripts/run-serial-tests.sh",
"ci:local": "bash scripts/ci-local.sh",
"ci:local:diff": "bash scripts/ci-local.sh --diff",
"ci:select-e2e": "bun run scripts/select-e2e.ts",
"typecheck": "tsc --noEmit",
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:no-double-retry": "scripts/check-no-double-retry.sh",
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
"check:source-id-projection": "scripts/check-source-id-projection.sh",
"check:privacy": "scripts/check-privacy.sh",
"check:proposal-pii": "scripts/check-proposal-pii.sh",
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
"check:test-names": "scripts/check-test-real-names.sh",
"check:progress": "scripts/check-progress-to-stdout.sh",
"check:exports-count": "scripts/check-exports-count.sh",
"check:admin-build": "scripts/check-admin-build.sh",
"check:admin-embedded": "scripts/check-admin-embedded.sh",
"check:test-isolation": "scripts/check-test-isolation.sh",
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
"prepublish:clawhub": "bun run build:all",
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
},
"openclaw": {
"compat": {
"pluginApi": ">=2026.4.0"
},
"extensions": [
"./src/openclaw-context-engine.ts"
]
},
"dependencies": {
"@ai-sdk/anthropic": "^3.0.71",
"@ai-sdk/google": "^3.0.64",
"@ai-sdk/openai": "^3.0.53",
"@ai-sdk/openai-compatible": "^2.0.41",
"@anthropic-ai/sdk": "^0.30.0",
"@aws-sdk/client-s3": "^3.1028.0",
"@dqbd/tiktoken": "^1.0.22",
"@electric-sql/pglite": "0.4.3",
"@jsquash/avif": "^2.1.1",
"@jsquash/png": "^3.1.1",
"@modelcontextprotocol/sdk": "1.29.0",
"ai": "^6.0.168",
"chokidar": "^4.0.3",
"cookie-parser": "^1.4.7",
"cors": "^2.8.5",
"eventsource-parser": "^3.0.8",
"exifr": "^7.1.3",
"express": "^5.1.0",
"express-rate-limit": "^7.5.0",
"gray-matter": "^4.0.3",
"heic-decode": "^2.1.0",
"js-yaml": "^3.14.2",
"marked": "^18.0.0",
"openai": "^4.0.0",
"pgvector": "^0.2.0",
"postgres": "^3.4.0",
"tree-sitter-wasms": "0.1.13",
"web-tree-sitter": "0.22.6",
"zod": "^4.3.6"
},
"devDependencies": {
"@types/bun": "latest",
"@types/cookie-parser": "^1.4.7",
"@types/cors": "^2.8.19",
"@types/express": "^5.0.6",
"@types/js-yaml": "^3.12.10",
"bun-types": "^1.3.13",
"fast-check": "^4.8.0",
"typescript": "^5.6.0"
},
"trustedDependencies": [
"@electric-sql/pglite"
],
"engines": {
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.41.23.0"
}
Reference in New Issue View Git Blame Copy Permalink