mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
726dfff02c
* fix(conversation-parser): threshold-gated fallback + acceptance floor (closes #1533) `gbrain conversation-parser scan` reported `phase: no_match` on meeting pages where 175 of 226 lines (77.8%) were valid `imessage-slack` format. The 36 reformatted Circleback meetings could not flow through the conversation facts pipeline. Root cause: `scorePattern` only scans the first 10 non-blank lines. A meeting page's `## Summary` + blockquote + `## Transcript` preamble takes all 10 head slots, so every pattern scored 0 and the orchestrator short-circuited to `no_match` without ever seeing the transcript. Fix: two-tier scoring with threshold gates. 1. Fast path unchanged: chat-only pages match on line 1, scoring 1.0, skipping the fallback entirely. 2. Full-body fallback fires when `top.score < SCORING_HEAD_TRIGGER_THRESHOLD` (0.3). NOT `=== 0` — Codex P1 #1 caught the bug class where a stray head match (blockquote that accidentally matches an unrelated pattern at 0.1) would suppress the fallback. 0.3 leaves the fast path untouched while triggering on any preamble-dominated page. 3. Minimum acceptance floor `SCORING_MIN_ACCEPTANCE` (0.05) prevents essay false positives: a 300-line essay with one stray `**Name** (date time):` line scores ~0.003 — without the floor it would flip to `regex_match` with `messages.length = 1`. Closes Codex P1 #2. DRY refactor: extract `getNonBlankLines` + `scoreFromLines` so the quick_reject + regex loop lives in one place. New exported `scorePatternFull` for direct unit testing. Fallback pre-splits the body ONCE per pass to avoid 12 redundant splits. Plan + decisions + Codex consult absorption at: ~/.claude/plans/system-instruction-you-are-working-starry-frost.md Tests: 10 new cases in test/conversation-parser/parse.test.ts (87 pass). Highlights: - #1533 IRON-RULE regression pin (meeting page → regex_match, imessage-slack, 20 messages) - Stray-head-match guard (Codex P1 #1: irc-classic 0.1 in head does not suppress fallback; imessage-slack wins on full body) - Essay false-positive guard (Codex P1 #2: 1/301 score below acceptance floor stays no_match) - 300-line preamble + 50 chat lines hits fallback - Cap test reshaped (Codex P2 #6): pins behavior not constant value Once landed and a brain has `cycle.conversation_facts_backfill.enabled = true` (opt-in), the 36 Circleback meetings flow through the fact extractor automatically. Operators on the manual path run `gbrain extract-conversation-facts <source>` directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(conversation-parser): add bold-paren-time built-in pattern (closes user-facing half of #1533) Per /codex follow-up D-FOLLOWUP-1.B: the threshold-gated fallback fix (8d7a18ac) closed the bug CLASS — but the user's actual 112 Circleback meeting files at ~/git/brain/meetings/ use a transcript shape that no existing built-in pattern matches: **Participant 2** (00:00): Companies that we... ← (HH:MM) **Participant 1** (00:00:00): We found the... ← (HH:MM:SS) Without a pattern that matches this shape, even the fallback re-scoring all 12 candidates against the full body returned zero matches → no_match. Add `bold-paren-time` as the 13th built-in. Two sub-shapes covered via non-capturing optional seconds group; capture indexes stay identical. date_source: frontmatter so the page's `date:` provides the day anchor. Time semantics: Circleback timestamps are elapsed-time-from-meeting- start, not wall-clock. Parser treats them as wall-clock 24h on the frontmatter date, so every message lands on the same day at HH:MM. The fact extractor only cares about speaker + content, so this is honest enough; precise per-line wall-clock would need an elapsed_time flag on PatternEntry (v0.42+ scope). Declaration position: after imessage-slack and telegram-bracket so on the rare tie those more-specific patterns win. The regex requires `\)` immediately after the time group, so imessage-slack's `(2024-03-15 9:00 AM)` shape falls through correctly. EMPIRICAL RESULT against all Circleback meetings in ~/git/brain/meetings: - 367 total files with `source: circleback` - Pre-fix: 0 parsed (no pattern matched the shape) - Post-fix: 113 parsed (112 via bold-paren-time, 1 via telegram-bracket) - 20,167 messages flow through to the fact extractor (was 0) - 254 remain no_match (notes-only meetings without inline transcripts — transcripts in those cases live in separate files referenced via blockquote, not in the meeting body) Smoke-tested manually against 3 representative files: - 2026-03-19-yc-partner-strategy-ai-leverage-review.md → 225 messages - 2026-01-15-ro-khanna-c4.md → 294 messages - 2026-04-01-narrative-arc-equity-regrets.md → 9 messages (HH:MM:SS variant) Tests: 54 unit cases pass (88 across full parser suite). 3 new cases in parse.test.ts pin the contract: (HH:MM) shape matches, (HH:MM:SS) shape matches, imessage-slack still wins on full-datetime overlap, and meeting page with preamble + bold-paren-time transcript hits the threshold fallback correctly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.41.21.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: bump conversation-parser entry to v0.41.21.0 (13 patterns + threshold gates) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(privacy): scrub OpenClaw fork name from new pattern's source_doc + CLAUDE.md CI verify failed on check:privacy — the new bold-paren-time pattern added in80208f21referenced the private OpenClaw fork name in builtins.ts:125 (comment) and builtins.ts:178 (source_doc), and the CLAUDE.md doc-sync commit1710020aleaked it once more. CLAUDE.md privacy rule (line 550): the private fork name is banned in CHANGELOG.md, README.md, docs/, skills/, PR titles + bodies, commit messages, and comments in checked-in code. Canonical replacement: "your OpenClaw" or "OpenClaw reference deployment". This commit rewrites all three sites. Source pipeline attribution stays accurate ("OpenClaw meeting-ingestion pipeline reformat of Circleback transcripts") without naming the specific private fork. bun run verify: 28/28 green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version 0.41.21.0 → 0.41.24.0 Queue reservation by user — 0.41.22.0 / 0.41.23.0 slots left for sibling worktrees. Bumps VERSION, package.json, CHANGELOG header, and the CLAUDE.md entry's version tag in lockstep. llms-full.txt regenerated. bun run verify: 28/28 green. Parser tests: 92/92. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
145 lines
7.3 KiB
JSON
145 lines
7.3 KiB
JSON
{
|
|
"name": "gbrain",
|
|
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
|
"type": "module",
|
|
"main": "src/core/index.ts",
|
|
"bin": {
|
|
"gbrain": "src/cli.ts"
|
|
},
|
|
"exports": {
|
|
".": "./src/core/index.ts",
|
|
"./engine": "./src/core/engine.ts",
|
|
"./types": "./src/core/types.ts",
|
|
"./operations": "./src/core/operations.ts",
|
|
"./minions": "./src/core/minions/index.ts",
|
|
"./engine-factory": "./src/core/engine-factory.ts",
|
|
"./pglite-engine": "./src/core/pglite-engine.ts",
|
|
"./link-extraction": "./src/core/link-extraction.ts",
|
|
"./import-file": "./src/core/import-file.ts",
|
|
"./transcription": "./src/core/transcription.ts",
|
|
"./embedding": "./src/core/embedding.ts",
|
|
"./config": "./src/core/config.ts",
|
|
"./markdown": "./src/core/markdown.ts",
|
|
"./backoff": "./src/core/backoff.ts",
|
|
"./search/hybrid": "./src/core/search/hybrid.ts",
|
|
"./search/expansion": "./src/core/search/expansion.ts",
|
|
"./ai/gateway": "./src/core/ai/gateway.ts",
|
|
"./extract": "./src/commands/extract.ts",
|
|
"./ingestion": "./src/core/ingestion/index.ts",
|
|
"./ingestion/test-harness": "./src/core/ingestion/test-harness.ts"
|
|
},
|
|
"scripts": {
|
|
"dev": "bun run src/cli.ts",
|
|
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
|
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
|
"build:admin": "cd admin && bun run build && cd .. && bun run scripts/build-admin-embedded.ts",
|
|
"build:admin-embedded": "bun run scripts/build-admin-embedded.ts",
|
|
"build:schema": "bash scripts/build-schema.sh",
|
|
"build:llms": "bun run scripts/build-llms.ts",
|
|
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
|
"test": "bash scripts/run-unit-parallel.sh",
|
|
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
|
"verify": "bash scripts/run-verify-parallel.sh",
|
|
"check:source-config-leak": "scripts/check-source-config-leak.sh",
|
|
"check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh",
|
|
"check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh",
|
|
"check:system-of-record": "scripts/check-system-of-record.sh",
|
|
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
|
"check:cli-exec": "scripts/check-cli-executable.sh",
|
|
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-source-config-leak.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh && scripts/check-skill-brain-first.sh && scripts/check-operations-filter-bypass.sh && scripts/check-gateway-routed-no-direct-anthropic.sh && scripts/check-worker-pool-atomicity.sh && scripts/check-no-double-retry.sh && scripts/check-batch-audit-site.sh",
|
|
"check:gateway-routed": "scripts/check-gateway-routed-no-direct-anthropic.sh",
|
|
"check:worker-pool-atomicity": "scripts/check-worker-pool-atomicity.sh",
|
|
"check:resolver": "bun src/cli.ts check-resolvable --strict --skills-dir skills/",
|
|
"check:skill-brain-first": "scripts/check-skill-brain-first.sh",
|
|
"check:wasm": "scripts/check-wasm-embedded.sh",
|
|
"check:newlines": "scripts/check-trailing-newline.sh",
|
|
"test:e2e": "bash scripts/run-e2e.sh",
|
|
"test:slow": "bash scripts/run-slow-tests.sh",
|
|
"test:heavy": "bash scripts/run-heavy.sh",
|
|
"test:profile": "bash scripts/profile-tests.sh",
|
|
"test:serial": "bash scripts/run-serial-tests.sh",
|
|
"ci:local": "bash scripts/ci-local.sh",
|
|
"ci:local:diff": "bash scripts/ci-local.sh --diff",
|
|
"ci:select-e2e": "bun run scripts/select-e2e.ts",
|
|
"typecheck": "tsc --noEmit",
|
|
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
|
"check:no-double-retry": "scripts/check-no-double-retry.sh",
|
|
"check:batch-audit-site": "scripts/check-batch-audit-site.sh",
|
|
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
|
"check:privacy": "scripts/check-privacy.sh",
|
|
"check:proposal-pii": "scripts/check-proposal-pii.sh",
|
|
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
|
"check:test-names": "scripts/check-test-real-names.sh",
|
|
"check:progress": "scripts/check-progress-to-stdout.sh",
|
|
"check:exports-count": "scripts/check-exports-count.sh",
|
|
"check:admin-build": "scripts/check-admin-build.sh",
|
|
"check:admin-embedded": "scripts/check-admin-embedded.sh",
|
|
"check:test-isolation": "scripts/check-test-isolation.sh",
|
|
"check:fuzz-purity": "scripts/check-fuzz-purity.sh",
|
|
"check:operations-filter-bypass": "scripts/check-operations-filter-bypass.sh",
|
|
"check:fixture-privacy": "scripts/check-fixture-privacy.sh",
|
|
"check:conversation-parser": "bun src/cli.ts eval conversation-parser test/fixtures/conversation-formats/all.jsonl --no-llm",
|
|
"check:source-scope-onboard": "scripts/check-source-scope-onboard.sh",
|
|
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
|
"prepublish:clawhub": "bun run build:all",
|
|
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
|
},
|
|
"openclaw": {
|
|
"compat": {
|
|
"pluginApi": ">=2026.4.0"
|
|
},
|
|
"extensions": [
|
|
"./src/openclaw-context-engine.ts"
|
|
]
|
|
},
|
|
"dependencies": {
|
|
"@ai-sdk/anthropic": "^3.0.71",
|
|
"@ai-sdk/google": "^3.0.64",
|
|
"@ai-sdk/openai": "^3.0.53",
|
|
"@ai-sdk/openai-compatible": "^2.0.41",
|
|
"@anthropic-ai/sdk": "^0.30.0",
|
|
"@aws-sdk/client-s3": "^3.1028.0",
|
|
"@dqbd/tiktoken": "^1.0.22",
|
|
"@electric-sql/pglite": "0.4.3",
|
|
"@jsquash/avif": "^2.1.1",
|
|
"@jsquash/png": "^3.1.1",
|
|
"@modelcontextprotocol/sdk": "1.29.0",
|
|
"ai": "^6.0.168",
|
|
"chokidar": "^4.0.3",
|
|
"cookie-parser": "^1.4.7",
|
|
"cors": "^2.8.5",
|
|
"eventsource-parser": "^3.0.8",
|
|
"exifr": "^7.1.3",
|
|
"express": "^5.1.0",
|
|
"express-rate-limit": "^7.5.0",
|
|
"gray-matter": "^4.0.3",
|
|
"heic-decode": "^2.1.0",
|
|
"js-yaml": "^3.14.2",
|
|
"marked": "^18.0.0",
|
|
"openai": "^4.0.0",
|
|
"pgvector": "^0.2.0",
|
|
"postgres": "^3.4.0",
|
|
"tree-sitter-wasms": "0.1.13",
|
|
"web-tree-sitter": "0.22.6",
|
|
"zod": "^4.3.6"
|
|
},
|
|
"devDependencies": {
|
|
"@types/bun": "latest",
|
|
"@types/cookie-parser": "^1.4.7",
|
|
"@types/cors": "^2.8.19",
|
|
"@types/express": "^5.0.6",
|
|
"@types/js-yaml": "^3.12.10",
|
|
"bun-types": "^1.3.13",
|
|
"fast-check": "^4.8.0",
|
|
"typescript": "^5.6.0"
|
|
},
|
|
"trustedDependencies": [
|
|
"@electric-sql/pglite"
|
|
],
|
|
"engines": {
|
|
"bun": ">=1.3.10"
|
|
},
|
|
"license": "MIT",
|
|
"version": "0.41.24.0"
|
|
}
|