websearch_to_tsquery('english') can't tokenize CJK, so the keyword arm
of hybrid search degrades badly for Korean/Chinese/Japanese queries on
the Postgres engine — particle-suffixed Korean often matches nothing.
PGLite has had the hasCJK() -> ILIKE + occurrence-count-ranking fallback
since v0.32.7; the Postgres engine never got it.
Port _searchKeywordCJK to postgres-engine.ts at both FTS sites
(searchKeyword dedup path + searchKeywordChunks), keeping the host
engine's conventions (sql.begin + SET LOCAL statement_timeout, named
limit params, its extra-filter set including types/exclude_slugs).
scoreExpr, ILIKE ESCAPE discipline, empty-query guard, and the
dedup/chunk-grain split are copied from PGLite exactly.
Measured on a private production Korean corpus (74k chunks):
particle-suffixed queries returned 7 rows via english FTS vs 152 via
the fallback; a common two-word query 21 vs 847 — a 20-40x recall gap
this closes.
Tests: 6 source-guardrail tests in test/postgres-engine.test.ts +
3 Postgres-vs-PGLite parity e2e tests with synthetic Korean fixtures.
bun run verify 31/31.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(migrate): v0_13_0 shells out to `gbrain` shim, not `process.execPath`
On bun-installed trees, process.execPath is the bun runtime itself.
`bun extract links ...` got reinterpreted as `bun run extract` and
crashed the upgrade mid-Phase B. The canonical shim on PATH already
wraps the right runtime+entrypoint; trust it.
Regression-guarded by test/migrations-v0_13_0.test.ts which greps
the source for `process.execPath` and `bun` invocations. This was
Bug 1 of tonight's v0.13 → v0.14 upgrade-night postmortem.
* fix(autopilot): resolveGbrainCliPath prefers shim, never returns .ts
argv[1] check used to short-circuit on /cli.ts, so bun-source installs
got a .ts path back. spawn() then failed EACCES because TypeScript
source isn't executable, and autopilot silently lost its worker.
Reordered probes: which gbrain (shim) first, then compiled execPath,
then argv[1] only if it ends in /gbrain. Deleted the .ts branch
entirely — no valid case exists.
Rewrote the existing test that enshrined the buggy .ts return.
Critical regression guard: resolver MUST NEVER return a .ts path
across any combination of argv[1] + execPath + shim availability.
This was Bug 4 of tonight's v0.13 → v0.14 upgrade-night postmortem.
* chore: bump version and changelog (v0.15.3)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(db): resolvePrepare() helper for PgBouncer transaction-mode pools
Adds port-6543 auto-detect with a 4-level precedence chain:
GBRAIN_PREPARE env var → ?prepare= URL param → port auto-detect → default.
Wires into the module-singleton connect() so the main CLI path no longer
hits "prepared statement does not exist" against Supabase transaction
pooler. Returns boolean | undefined; undefined means omit the option and
let postgres.js default (true) stand.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(postgres-engine): honor resolvePrepare in worker-instance pool
Without this, \`gbrain jobs work\` against a Supabase pooler URL hits
"prepared statement does not exist" under load even after the module
singleton was fixed in db.ts. Community PR #270 (@notjbg) caught this
second path that #284 had missed. Reuses the shared helper, no regex
duplication.
Co-Authored-By: Jonah Berg <jonah.berg.g@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): pgbouncer_prepare check
URL-only check (no DB roundtrip) that reads the configured URL via
loadConfig() and flags the footgun: port 6543 with prepared statements
still enabled. Warns with the exact env override (GBRAIN_PREPARE=false)
and URL-query alternative (?prepare=false). Works for both the module
singleton and worker-instance engines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: resolvePrepare precedence matrix + postgres-engine wiring guard
- test/resolve-prepare.test.ts: 11 cases covering env override, URL
query param, port auto-detect, malformed URLs, postgres:// scheme,
URL-encoded credentials. Uses bun:test — #284's original vitest file
would never have run in this project.
- test/postgres-engine.test.ts: new source-level grep case asserting
the worker-pool connect() branch calls db.resolvePrepare(url) and
includes a typeof prepare === 'boolean' check. Mirrors the existing
SET LOCAL regression guard. If anyone rips out the wiring, the build
fails before shipping starts dropping rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.15.4)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Jonah Berg <jonah.berg.g@gmail.com>
* fix(sync): remove nested transaction that deadlocks > 10 file syncs
sync.ts wraps the add/modify loop in engine.transaction(), and each
importFromContent inside opens another one. PGLite's
_runExclusiveTransaction is a non-reentrant mutex — the second call
queues on the mutex the first is holding, and the process hangs forever
in ep_poll. Reproduced with a 15-file commit: unpatched hangs, patched
runs in 3.4s. Fix drops the outer wrap; per-file atomicity is correct
anyway (one file's failure should not roll back the others).
(cherry picked from commit 4a1ac00105)
* test(sync): regression guard for #132 top-level engine.transaction wrap
Reads src/commands/sync.ts verbatim and asserts no uncommented
engine.transaction() call appears above the add/modify loop. Protects
against silent reintroduction of the nested-mutex deadlock that hung
> 10-file syncs forever in ep_poll.
* feat(utils): tryParseEmbedding() skip+warn sibling for availability path
parseEmbedding() throws on structural corruption — right call for ingest/
migrate paths where silent skips would be data loss. Wrong call for
search/rescore paths where one corrupt row in 10K would kill every
query that touches it.
tryParseEmbedding() wraps parseEmbedding in try/catch: returns null on
any shape that would throw, warns once per session so the bad row is
visible in logs. Use it anywhere we'd rather degrade ranking than blow
up the whole query.
Retrofit postgres-engine.getEmbeddingsByChunkIds (the #175 slice call
site) — the 5-line rescore loop was the direct motivator. Keep the
throwing parseEmbedding() for everything else (pglite-engine rowToChunk,
migrate-engine round-trips, ingest).
* postgres-engine: scope search statement_timeout to the transaction
searchKeyword and searchVector run on a pooled postgres.js client
(max: 10 by default). The original code bounded each search with
await sql`SET statement_timeout = '8s'`
try { await sql`<query>` }
finally { await sql`SET statement_timeout = '0'` }
but every tagged template is an independent round-trip that picks an
arbitrary connection from the pool. The SET, the query, and the reset
could all land on DIFFERENT connections. In practice the GUC sticks
to whichever connection ran the SET and then gets returned to the
pool — the next unrelated caller on that connection inherits the 8s
timeout (clipping legitimate long queries) or the reset-to-0 (disabling
the guard for whoever expected it). A crash in the middle leaves the
state set permanently.
Wrap each search in sql.begin(async sql => …). postgres.js reserves
a single connection for the transaction body, so the SET LOCAL, the
query, and the implicit COMMIT all run on the same connection. SET
LOCAL scopes the GUC to the transaction — COMMIT or ROLLBACK restores
the previous value automatically, regardless of the code path out.
Error paths can no longer leak the GUC.
No API change. Timeout value and semantics are identical (8s cap on
search queries, no effect on embed --all / bulk import which runs
outside these methods). Only one transaction per search — BEGIN +
COMMIT round-trips are negligible next to a ranked FTS or pgvector
query.
Also closes the earlier audit finding R4-F002 which reported the same
pattern on searchKeyword. This PR covers both searchKeyword and
searchVector so the pool-leak class is fully closed.
Tests (test/postgres-engine.test.ts, new file):
- No bare SET statement_timeout remains after stripping comments.
- searchKeyword and searchVector each wrap their query in sql.begin.
- Both use SET LOCAL.
- Neither explicitly clears the timeout with SET statement_timeout=0.
Source-level guardrails keep the fast unit suite DB-free. Live
Postgres coverage of the search path is in test/e2e/search-quality.test.ts,
which continues to exercise these methods end-to-end against
pgvector when DATABASE_URL is set.
(cherry picked from commit 6146c3b470)
* feat(orphans): add gbrain orphans command for finding under-connected pages
Surfaces pages with zero inbound wikilinks. Essential for content
enrichment cycles in KBs with 1000+ pages. By default filters out
auto-generated pages, raw sources, and pseudo-pages where no inbound
links is expected; --include-pseudo to disable.
Supports text (grouped by domain), --json, --count outputs.
Also exposed as find_orphans MCP operation.
Tests cover basic detection, filtering, all output modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit f50954f8e0)
* feat(extract): support Obsidian wikilinks + wiki-style domain slugs in canonical extractor
extractEntityRefs now recognizes both syntaxes equally:
[Name](people/slug) -- upstream original
[[people/slug|Name]] -- Obsidian wikilink (new)
Extends DIR_PATTERN to include domain-organized wiki slugs used by
Karpathy-style knowledge bases:
- entities (legacy prefix some brains keep during migration)
- projects (gbrain canonical, was missing from regex)
- tech, finance, personal, openclaw (domain-organized wiki roots)
Before this change, a 2,100-page brain with wikilinks throughout extracted
zero auto-links on put_page because the regex only matched markdown-style
[name](path). After: 1,377 new typed edges on a single extract --source db
pass over the same corpus.
Matches the behavior of the extract.ts filesystem walker (which already
handled wikilinks as of the wiki-markdown-compat fix wave), so the db and
fs sources now produce the same link graph from the same content.
Both patterns share the DIR_PATTERN constant so adding a new entity dir
only requires updating one string.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1cfb15679a)
* feat(doctor): jsonb_integrity + markdown_body_completeness detection
Add two v0.12.1-era reliability checks to `gbrain doctor`:
- `jsonb_integrity` scans the 4 known write sites from the v0.12.0
double-encode bug (pages.frontmatter, raw_data.data,
ingest_log.pages_updated, files.metadata) and reports rows where
jsonb_typeof(col) = 'string'. The fix hint points at
`gbrain repair-jsonb` (the standalone repair command shipped in
v0.12.1).
- `markdown_body_completeness` flags pages whose compiled_truth is
<30% of the raw source content length when raw has multiple H2/H3
boundaries. Heuristic only; suggests `gbrain sync --force` or
`gbrain import --force <slug>`.
Also adds test/e2e/jsonb-roundtrip.test.ts — the regression coverage
that should have caught the original double-encode bug. Hits all four
write sites against real Postgres and asserts jsonb_typeof='object'
plus `->>'key'` returns the expected scalar.
Detection only: doctor diagnoses, `gbrain repair-jsonb` treats.
No overlap with the standalone repair path.
* chore: bump to v0.12.3 + changelog (reliability wave)
Master shipped v0.12.1 (extract N+1 + migration timeout) and v0.12.2
(JSONB double-encode + splitBody + wiki types + parseEmbedding) while
this wave was mid-flight. Ships the remaining pieces as v0.12.3:
- sync deadlock (#132, @sunnnybala)
- statement_timeout scoping (#158, @garagon)
- Obsidian wikilinks + domain patterns (#187 slice, @knee5)
- gbrain orphans command (#187 slice, @knee5)
- tryParseEmbedding() availability helper
- doctor detection for jsonb_integrity + markdown_body_completeness
No schema, no migration, no data touch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs: update project documentation for v0.12.3
CLAUDE.md:
- Add src/commands/orphans.ts entry
- Expand src/commands/doctor.ts with v0.12.3 jsonb_integrity +
markdown_body_completeness check descriptions
- Update src/core/link-extraction.ts to mention Obsidian wikilinks +
extended DIR_PATTERN (entities/projects/tech/finance/personal/openclaw)
- Update src/core/utils.ts to mention tryParseEmbedding sibling
- Update src/core/postgres-engine.ts to note statement_timeout scoping +
tryParseEmbedding usage in getEmbeddingsByChunkIds
- Add Key commands added in v0.12.3 section (orphans, doctor checks)
- Add test/orphans.test.ts, test/postgres-engine.test.ts, updated
descriptions for test/sync.test.ts, test/doctor.test.ts,
test/utils.test.ts
- Add test/e2e/jsonb-roundtrip.test.ts with note on intentional overlap
- Bump operation count from ~36 to ~41 (find_orphans shipped in v0.12.3)
README.md:
- Add gbrain orphans to ADMIN commands block
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: sunnnybala <dhruvagarwal5018@gmail.com>
Co-authored-by: Gustavo Aragon <gustavoraularagon@gmail.com>
Co-authored-by: Clevin Canales <clevin@Clevins-MacBook-Pro.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Clevin Canales <clev.canales@gmail.com>
