mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
814258dda67945ffec9457a1e73980e947b7e462
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
f09f9177a9 |
v0.42.10.0 feat(extract): opt-in global-basename wikilink resolution (closes #972) (#1388)
* v0.40.8.2 fix(extract): opt-in global-basename wikilink resolution (#972) Bare wikilinks like [[struktura]] that point at pages in another folder were silently dropped from the graph. The issue reporter saw 71 wikilinks in Obsidian render to 12 in gbrain (~83% lost). Symptoms downstream: `gbrain graph` returns thin neighborhoods, `gbrain backlinks` undercounts. This release adds an opt-in mode that resolves bare wikilinks by basename match, covers all three resolver surfaces (FS-source extract, DB-source extract, put_page auto-link), and emits one edge per match — no silent winner on ambiguity. `gbrain doctor` surfaces a paste-ready enable hint when ≥5 bare wikilinks would resolve under the new mode. Enable with: gbrain config set link_resolution.global_basename true gbrain extract links Default stays off. Existing brains see zero behavior change on upgrade. Closes #972. Adapts PR #1233 from @rayers (regex shape + slug-tail index) into a multi-match, opt-in form with FS-source coverage that the original PR explicitly skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: document opt-in global-basename wikilink resolution (#972) The #972 feature shipped with no user-facing docs — only CHANGELOG + CLAUDE.md. Anyone migrating an Obsidian/Notion vault with bare [[name]] wikilinks couldn't discover the link_resolution.global_basename flag unless gbrain doctor happened to surface its hint. - README "Self-wiring knowledge graph": one sentence on the opt-in mode for Obsidian-style cross-folder bare wikilinks + the doctor pre-check, linking to the install step. - INSTALL_FOR_AGENTS Step 4.5 (Wire the Knowledge Graph): a dedicated agent- facing subsection — when bare [[name]] links need it, the enable command, re-running extract, the doctor opportunity hint, and the multi-match behavior. - Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): resolve aliased wikilinks by target slug, not display text Codex outside-voice [P1]: `[[struktura|the project]]` resolved the basename "the project" (the alias) instead of `struktura` (the target), because extractPageLinks called resolveBasenameMatches(ref.name) and the doctor check keyed basenameIndex.get(e.name). ref.name is the display alias (match[2]); ref.slug is the wikilink target (match[1]). - extractPageLinks resolves ref.slug; context excerpt locates ref.slug. - doctor link_resolution_opportunity keys e.slug so its estimate matches what extraction actually resolves. - Test: aliased wikilink calls resolveBasenameMatches with the target, never the display text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): reconcile wikilink-resolved edges in put_page auto-link Codex outside-voice [P1]: put_page's reconcilableOut filter excluded link_source='wikilink-resolved', so a basename edge written by auto-link survived after the bare wikilink was deleted from the page OR the link_resolution.global_basename flag was turned off (the stale-removal loop only iterates reconcilableOut). Add 'wikilink-resolved' to the reconcilable set; manual edges still untouched. Test: write page with [[struktura]] (flag on) → edge lands; re-put without the wikilink → edge reconciled away. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): source-scope basename resolution (no cross-source edges) Codex outside-voice [P1]: makeResolver.resolveBasenameMatches called engine.getAllSlugs() unscoped, so a bare [[name]] could resolve to a same-tail page in a DIFFERENT source and create a cross-source edge. The engine exposes getAllSlugs({sourceId}) precisely to prevent this. #972 is "global basename across folders," not "cross-source federation" — the canonical gbrain multi-source bug class. - makeResolver gains opts.sourceId; ensureBasenameIndex passes it to getAllSlugs (unscoped only when sourceId omitted — back-compat). - runAutoLink (put_page) passes opts.sourceId; extractLinksFromDB passes sourceIdFilter. FS extract is already single-source (walks one dir). - Tests: scoped index returns only the source's slugs (no cross-source); unscoped call stays brain-wide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): FS-source basename edges carry link_source='wikilink-resolved' The FS extract path is the issue's default repro (gbrain extract links with no --source db). ExtractedLink had no link_source field, so FS basename edges landed with the engine default ('markdown') instead of the 'wikilink-resolved' provenance the DB / put_page paths set and the docs promise. The e2e FS test only asserted link_type, so it was blind to this. - ExtractedLink gains link_source?; extractLinksFromFile sets it to 'wikilink-resolved' on basename edges (undefined for ordinary markdown). - Carries through the addLinksBatch snapshots automatically (LinkBatchInput already has link_source); single-row addLink fallback now passes it too. - e2e FS repro asserts link_source === 'wikilink-resolved'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(#972): one shared basename matcher across resolver/FS/doctor Codex outside-voice [P2] DRY: three surfaces each hand-rolled a basename matcher with divergent key sets — the doctor omitted the slugified key, so its link_resolution_opportunity estimate undercounted what extraction resolves, and the resolver returned matches in unsorted getAllSlugs bucket order. New shared exports in link-extraction.ts: buildBasenameIndex(slugs) + queryBasenameIndex(index, name) (keys raw/lower/slugified tail; stable sort shorter-first then lexical) + normalizeBasename. - makeResolver.resolveBasenameMatches → queryBasenameIndex (now stable-sorted). - extract.ts resolveBasenameMatchesFromSlugs → delegates to the shared pair. - doctor link_resolution_opportunity → shared builder/query (slugified key added; estimate now matches extraction). - Test: doctor counts a slugified-only match ([[Fast Weigh]] → companies/fast-weigh). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): P2 cluster — masking, code-fence, self-link, dedup decision Codex outside-voice P2 findings: - P2a markdown-label masking: a wikilink inside a markdown-link label ([see [[acme]]](companies/acme.md)) spawned a stray generic basename ref. Pass-1 can't match the nested brackets, so a new MARKDOWN_LABEL_WIKILINK_RE masks those spans out of pass 2c. Inner [[acme]] is now inert. - P2b FS code-fence: the FS path (extractMarkdownLinks on raw content) didn't strip code blocks like the DB path. extractLinksFromFile now scans stripCodeBlocks(content) so [[name]] inside a fence creates no FS edge. - P2c self-link guard: a basename [[own-tail]] on its own page resolved back to itself. Dropped in both extractPageLinks and the FS path. - P2d dedup: documented the decision to KEEP qualified + bare edges to the same target as separate rows (distinct provenance/audit trail). - P2e: skipFrontmatter unresolved-contract tests added. Tests: P2a inert-label, P2c self-link drop, P2b code-fence, P2e unresolved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(#972): bound the doctor link_resolution_opportunity scan The check did listAllPageRefs() + a getPage() per page under a 60s budget. On a large brain (the eng-review concern) it hit the budget every non-fast doctor run and returned a perpetual partial, adding ~60s. Now batch-loads the 1000 most-recent pages in ONE query (ORDER BY id DESC LIMIT SAMPLE_LIMIT) and scans in memory, with the 60s cap kept as a backstop. Mirrors the v0.40.9 sampling convention. The estimate message names the bound when the brain exceeds the sample ("scanned the 1000 most-recent of N pages"). Test: source-grep pins the bounded query + the absence of the per-page getPage walk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(#972): reconcile stale version/migration references to v112 / 0.42.6.0 Merge churn left intermediate refs: schema.sql + schema-embedded.ts said "migration v93", CLAUDE.md said "v0.41.32.0 / Migration v109", CHANGELOG said "Migration v93". Reconciled all to migration v112 / shipping 0.42.6.0. The CLAUDE.md annotation is also refreshed to describe the final behavior (shared matcher, source-scoping, alias-by-target, stale-edge reconciliation, bounded doctor scan) and credit @rayers + @ukd1. Regenerated schema-embedded + llms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#972): register doctor check category + bump llms budget to 800KB Two full-suite gate failures from the re-sync: - doctor-categories drift guard: the new `link_resolution_opportunity` check wasn't in any category set. Added to BRAIN_CHECK_NAMES (alongside graph_coverage / orphan_ratio — it's a graph-quality signal). - build-llms size budget: the #972 Key Files annotation (landing with master's #1696/#1699 waves) pushed llms-full.txt past 750KB. Bumped FULL_SIZE_BUDGET 750KB→800KB, the established "budget tracks CLAUDE.md's legitimate per-feature growth" pattern (600→700→750→800 across releases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Garry Tan <garrytan@gmail.com> |
||
|
+10
|
ff53a4c9bc |
v0.31.1.1-fixwave fix-wave: 22 community fixes (auth-code P0, upgrade-path, sync, multi-source, privacy) (#776)
* fix: bootstrap forward-references for v39-v41 schema replay
Three column-with-index forward references in the embedded schema blob were
missing from applyForwardReferenceBootstrap, so any brain at config.version
< 39 (Postgres) or < 41 (PGLite) wedges before the migration runner can
advance. Reproduced end-to-end on a PlanetScale Postgres brain stuck at
config.version=34 trying to upgrade to v0.30.0:
ERROR: column "effective_date" does not exist
ERROR: column cc.modality does not exist
(After upgrading, gbrain search and gbrain reindex-frontmatter both fail.)
The schema-blob references that crash before migrations run:
- v39 (multimodal_dual_column_v0_27_1):
CREATE INDEX idx_chunks_embedding_image
ON content_chunks USING hnsw (embedding_image vector_cosine_ops)
WHERE embedding_image IS NOT NULL;
- v41 (pages_recency_columns):
CREATE INDEX pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
PGLite already covered v39 (lines 273+, 308+, 382-392). Postgres and PGLite
both lacked v40+v41 coverage. This commit adds:
- Postgres engine probe + branch for v39 (modality, embedding_image) — was
entirely missing on Postgres, so Postgres brains < v39 hit the wedge that
PGLite already protected against.
- Both engines: probe + branch for v40+v41. Bootstraps all five additive
pages columns (emotional_weight, effective_date, effective_date_source,
import_filename, salience_touched_at) gated on `effective_date_exists`
as the proxy.
- test/schema-bootstrap-coverage.test.ts: extends REQUIRED_BOOTSTRAP_COVERAGE
with the six new columns AND the pre-test DROP block so both the per-target
assertion test and the end-to-end "bootstrap + SCHEMA_SQL replay" test
exercise the new coverage.
All 5 tests in schema-bootstrap-coverage pass. typecheck clean.
Bootstrap stays additive-columns-only. Indexes are left to schema replay /
migrations as before.
* fix(deps): declare @jsquash/png and heic-decode
Both packages are direct imports in src/core/import-file.ts (decodeIfNeeded
for HEIC/AVIF → PNG) but only @jsquash/avif was declared. bun --compile
fails on a fresh install:
error: Could not resolve: "@jsquash/png/encode.js"
error: Could not resolve: "heic-decode"
Adds the missing declarations so npm install / bun install bring them in.
Versions chosen as latest at time of fix:
@jsquash/png ^3.1.1
heic-decode ^2.1.0
* fix(backfill-effective-date): replace bare BEGIN/COMMIT with engine.transaction()
postgres.js refuses bare BEGIN/COMMIT on pooled connections with
UNSAFE_TRANSACTION. The migration runner and other call sites already
use engine.transaction() (which routes through sql.begin() with a
reserved backend) — backfill-effective-date.ts was the holdout.
Reproduces on PlanetScale Postgres (us-east-4.pg.psdb.cloud) running
the v0.29.1 orchestrator's Phase B against a brain that has any rows
needing backfill:
Reindex ok ... UNSAFE_TRANSACTION: Only use sql.begin, sql.reserved or max: 1
Switches the per-batch transaction to engine.transaction(async tx => …).
The SET LOCAL statement_timeout still scopes to the transaction; UPDATE
runs through the tx-scoped engine. ROLLBACK on error happens
automatically via sql.begin's contract.
Equivalent fix shape to existing usages in src/core/postgres-engine.ts
(lines 703, 806, 925) and the migration runner in src/core/migrate.ts
(line 2147).
* fix(v0_29_1): connect engine before use in Phase B and Phase C
phaseBBackfill() and phaseCVerify() build their own engine via
createEngine(toEngineConfig(cfg)) but never call engine.connect().
This worked accidentally before because executeRaw lazily falls back
to db.getConnection(), but engine.transaction() (added in the
companion backfill fix) requires a connected backend and surfaces
the missing-connect with:
No database connection: connect() has not been called.
Fix: Run gbrain init --supabase or gbrain init --url <connection_string>
Other orchestrators in the same directory get this right —
v0_28_0.ts:181 already does `await engine.connect(engineConfig)`
right after createEngine. Aligning v0_29_1 with that pattern.
After this + the backfill fix, v0.29.1 orchestrator runs to
'complete' on a fresh upgrade with backfill-needed rows, instead
of wedging at 'partial' status.
Note: anyone hitting the wedged state after the prior failures will
need `gbrain apply-migrations --force-retry 0.29.1` once before the
next apply-migrations --yes succeeds (the 3-consecutive-partials
guard in apply-migrations.ts is still active).
* fix: connect engine in v0.29.1 migration
* fix(upgrade): detectBunLink fails because bun resolves symlinks in argv[1]
bun resolves the entire symlink chain before setting process.argv[1],
so lstatSync(argv1).isSymbolicLink() always returns false for bun-link
installs, short-circuiting the git-config walk that would correctly
identify the repo. Remove the symlink gate — argv[1] is already the
real path inside the checkout, which is what the walk needs.
Also: return { repoRoot } so the upgrade path can auto-execute
git pull + bun install via execFileSync (no shell injection surface).
Fixes #368, supersedes incomplete v0.28.5 fix for #656.
* fix(oauth): clamp authorize() requested scopes against client.scope (RFC 6749 §3.3)
The MCP SDK's authorize handler (`@modelcontextprotocol/sdk/.../auth/handlers/authorize.js`)
splits `?scope=...` verbatim and forwards the parsed list to the provider, so the
provider has to clamp against the client's registered grant. v0.28.11
`authorize()` (src/core/oauth-provider.ts:235-259) inserted `params.scopes || []`
raw into `oauth_codes`, so a `read`-registered client requesting
`?scope=admin` had `['admin']` stored and `exchangeAuthorizationCode` issued
a fully-admin access token at /token exchange.
The asymmetry is the bug: the other two grant entry points already clamp.
`exchangeClientCredentials` (line 513-515) filters requested scopes through
`hasScope(allowedScopes, s)`, and `exchangeRefreshToken`'s F3 (line 372-380)
enforces RFC 6749 §6 subset against the original grant. authorize() lined up
with neither.
Fix mirrors the client_credentials filter shape so all three grant entry
points clamp consistently:
const allowedScopes = parseScopeString(client.scope);
const grantedScopes = (params.scopes || []).filter(s => hasScope(allowedScopes, s));
Empty/omitted requested scope keeps storing `[]` (existing shape, not a
security boundary). The clamped subset is what the client sees in the
`scope` field of the token response, which is the spec-compliant signal
that the grant was reduced.
Test coverage:
- New: authorize clamps requested scopes against client.scope (RFC 6749 §3.3)
— read-only client requests ['read','write','admin'] and the issued token
carries only ['read'].
- New: authorize subset request returns subset — 'read write' client
requesting ['read'] gets ['read'] (regression guard against over-clamping).
The existing v0.26.9 oauth.test.ts pins F3 (refresh clamp) but had no
authorize-side coverage, which is why the regression survived.
* fix(sync): handle detached HEAD by skipping pull and ingesting local working tree
* fix(sync): --skip-failed acks pre-existing unacked failures up-front
The recovery flow that doctor + printSyncResult both advertise was broken:
1. User has files with bad YAML → they hit the failure log + sync stays
blocked at last_commit.
2. User fixes the YAML.
3. User re-runs `gbrain sync` — sync succeeds, advances last_commit.
4. `gbrain doctor` still reports N unacked failures from step 1 because
sync-failures.jsonl is append-only history, never auto-cleared.
5. doctor message says: "use 'gbrain sync --skip-failed' to acknowledge".
6. User runs `gbrain sync --skip-failed` → "Already up to date." → log
unchanged.
The bug: --skip-failed only acknowledges failures from the CURRENT run.
performSync's ack path is gated on `failedFiles.length > 0` after sync —
it never fires when the diff is empty (because the user already fixed
the bad files) or when the sync is up to date. So the documented recovery
sequence is a no-op exactly when the user needs it.
The fix: at the top of runSync, when --skip-failed is set, eagerly ack
any pre-existing unacked failures before any sync work runs. Now the flag
means "acknowledge whatever is currently flagged and move on" regardless
of whether the current run produces new failures or finds nothing to do.
The inner per-run ack path stays — it still handles new failures from
the CURRENT run, which is the (a) syncing now produces failures + (b)
caller wants to ack them path. The two paths compose: `gbrain sync
--skip-failed` clears stale + advances past anything new, all in one
command, matching what the doctor message promises.
Tests: 2 added in test/sync-failures.test.ts. One source-string pin on
the new gate (the file's existing pattern for CLI-flag tests). One
behavioral test on the underlying acknowledgeSyncFailures path.
Repro:
$ gbrain doctor
[WARN] sync_failures: 27 unacknowledged sync failure(s)...
Fix the file(s) and re-run 'gbrain sync', or use
'gbrain sync --skip-failed' to acknowledge.
$ # ... fix the YAML ...
$ gbrain sync
Already up to date.
$ gbrain sync --skip-failed
Already up to date. # before this PR
$ gbrain doctor
[WARN] sync_failures: 27 unacknowledged sync failure(s)... # still!
After:
$ gbrain sync --skip-failed
Acknowledged 27 pre-existing failure(s).
Already up to date.
$ gbrain doctor
[OK] sync_failures: N historical sync failure(s), all acknowledged
* fix(extract): default --dir to configured brain dir, not cwd
`gbrain extract links` (and timeline / all) defaulted --dir to '.' when
not explicitly passed (src/commands/extract.ts:357). Combined with a
walker that skips dotfiles but NOT node_modules/dist/build/vendor, this
turned a no-arg invocation into a footgun.
Repro:
$ cd ~/Documents/some-project # has a node_modules/ tree
$ gbrain extract links
[extract.links_fs] 28989/28989 (100%) done
Links: created 0 from 28989 pages
Done: 0 links, 0 timeline entries from 28989 pages
The "28989 pages" is `walkMarkdownFiles('.')` recursively eating package
READMEs, dependency docs, fixture content. Their from_slug doesn't match
any row in the pages table, so addLinksBatch rejects every insert and
returns 0. Output looks like a healthy idempotent no-op; was actually a
wasteful junk walk that wrote nothing.
Fix: when --dir is not passed AND source is fs, resolve from
sources(local_path) via getDefaultSourcePath — same helper sync uses
(src/commands/sync.ts:1089). The default behavior now matches `sync`:
"work on the configured brain". Falls back to a clear error when no
source is configured, telling the user to either pass --dir, register
a source, or use --source db.
Behavior matrix:
--dir explicit → use that path (unchanged)
--dir absent + cfg → resolve from sources(local_path)
--dir absent + no → error with actionable hint (was: walk cwd silently)
--dir . → cwd (user opted in explicitly — unchanged)
Tests: three added in test/extract-fs.test.ts:
1. configured source → no-arg invocation extracts from that path
2. no source configured → exit 1 + actionable error message
3. explicit --dir wins over a configured (decoy) source path
* fix(extract): normalize slugs to lowercase via pathToSlug() (T-OBS-1)
The extractor was generating from_slug and the allSlugs lookup set from
`relPath.replace('.md', '')` in 5 places, producing CAPS slugs for files
named ETHOS.md, AGENTS.md, ROADMAP.md, etc.
Pages persist in the DB with lowercase slug (core/sync.ts pathToSlug()
applies .toLowerCase()). The CAPS extractor output mismatched the DB rows,
so INSERT ... JOIN pages ON pages.slug = v.from_slug silently dropped
links from CAPS-named source files. The link batch returned 'inserted'
counts that were lower than the wikilinks actually present, with no error.
Reproduction (in a brain with CAPS-named canonical docs):
1. echo 'See [agents](agents.md).' > ETHOS.md
2. gbrain put ethos < ETHOS.md # page row: slug='ethos'
3. gbrain extract links --source fs
4. gbrain backlinks agents → [] (expected: contains 'ethos')
Fix: import pathToSlug from core/sync.ts and use it in all 5 sites:
- extractLinksFromFile (line 200): from_slug derivation
- runIncrementalExtractInternal (line 456): allSlugs set
- extractLinksFromDir (line 552): allSlugs set
- timeline loop (line 643): from_slug for timeline entries
- extractLinksForSlugs (line 673): allSlugs set used by sync hook
This single-line-per-site change keeps the extractor consistent with the
sync layer's slug normalization and doesn't introduce any new behavior
for already-lowercase paths (idempotent).
Tests: added 'extractLinksFromFile — slug normalization (T-OBS-1
regression)' suite with 4 cases covering CAPS, mixed-case, idempotent
lowercase, and nested path. Full extract suite (54 → 58 tests) passes.
Reported by Claude Code (Opus 4.7) during Obsidian PKM integration on
the gstack-plan Living Repo, where ~111 wikilinks pointing to ETHOS,
AGENTS, ROADMAP, etc. failed to count toward brain_score (54/100 vs
expected 75+/100). Documented as T-OBS-1 in the consumer's blocked.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): CLI_ONLY commands should short-circuit on --help instead of executing
* fix(doctor): correct command syntax in graph_coverage warn message
graph_coverage warn directs users to run `gbrain link-extract &&
gbrain timeline-extract`, but no commands by those names are
registered in cli.ts. The actual commands are `gbrain extract links`
and `gbrain extract timeline` (registered as the 'extract'
subcommand at src/cli.ts:525, with the kind argument 'links' /
'timeline' / 'all' parsed inside src/commands/extract.ts).
A user who runs the suggested command gets:
$ gbrain link-extract
Unknown command: link-extract
This is the only place in src/ with the wrong syntax — the rest of
the docs (init.ts:221, init.ts:331, features.ts:120,
v0_13_0.ts:67, sync.ts:752 comment) all already say 'extract links'.
This patch just brings doctor.ts in line.
* fix(doctor): use autoDetectSkillsDir so OpenClaw workspaces are reachable
`gbrain doctor` was the only consumer of `findRepoRoot` from
`core/repo-root.ts`. Every other consumer (check-resolvable.ts:145,
skillify.ts, etc.) uses `autoDetectSkillsDir`, which has the full
detection chain:
1. \$OPENCLAW_WORKSPACE
2. ~/.openclaw/workspace
3. findRepoRoot() walk from cwd
4. ./skills
`findRepoRoot` only does step 3. Result: when the user runs `gbrain
doctor` from any directory outside the gbrain repo or the OpenClaw
workspace tree (e.g., a project's checkout), `resolver_health` reports
"Could not find skills directory" even though the dispatcher exists at
~/.openclaw/workspace/skills/RESOLVER.md.
Reproduces in any directory other than ~/gbrain or its descendants on
a system with ~/.openclaw/workspace/skills/RESOLVER.md present:
\$ cd ~/Documents
\$ gbrain doctor
[WARN] resolver_health: Could not find skills directory # before
[WARN] resolver_health: 5 issue(s): 0 error(s), 5 warning(s) # after
Switching doctor to `autoDetectSkillsDir` brings it inline with the rest
of the codebase. The detected dir is also passed to
`checkSkillConformance` (step 2 of the resolver_health block), which
previously rebuilt the path from `repoRoot` — now uses the same
detected path for consistency.
All 15 existing tests in test/doctor.test.ts continue to pass.
* fix(mcp): exit serve process on stdin-close/SIGTERM
MCP stdio server was keeping the bun process alive indefinitely after
the client disconnected. Over days this accumulated 20+ orphaned
gbrain serve processes, all holding the PGLite directory open.
Since PGLite is single-writer, this caused write-lock contention that
made email-sync fail its 15s per-put timeout: 114 puts x 15s = 28.5min
runs with 0 emails written.
Now listens for stdin end/close, transport close, and SIGTERM/SIGINT/
SIGHUP; calls engine.disconnect() and exits cleanly.
Root cause for the no-gbrain-run-in-50h alert.
* fix(skills): broaden RESOLVER triggers + 1 ambiguity flag (37 misses → 0, 100% top-1 accuracy)
`bun run src/cli.ts routing-eval` was reporting 37 ROUTING_MISS entries
across 10 skills whose RESOLVER.md trigger phrases didn't match any of
their own routing-eval.jsonl fixture intents. Two distinct causes:
1. Single-phrase triggers in 9 skills under '## Uncategorized' didn't
cover the paraphrased fixture variations they're supposed to route.
Broadened each trigger cell to a quoted-phrase list that covers the
fixtures (5 fixtures per skill on average).
2. The media-ingest row used unquoted prose
('Video, audio, PDF, book, YouTube, screenshot') which
extractTriggerPhrases() collapses into one impossible long phrase
('video audio pdf book youtube screenshot') under normalizeText —
no fixture intent will ever contain that exact substring. Converted
to a quoted phrase list.
3. One fixture ('web research pass on this person') legitimately
matches both `perplexity-research` and `data-research`
(data-research's trigger row contains "Research"). Marked the
fixture `ambiguous_with: ["data-research"]` since the overlap
on the keyword 'research' is inherent and expected.
Skills with broadened triggers:
- voice-note-ingest, article-enrichment, book-mirror,
archive-crawler, brain-pdf, academic-verify, concept-synthesis,
perplexity-research, strategic-reading, media-ingest
Before: 58 cases, 37 misses, ~36% top-1 accuracy
After: 58 cases, 0 misses, 100% top-1 accuracy
This also clears `gbrain doctor`'s `resolver_health: 37 issue(s)` warning.
* fix(multi-source): thread source_id through per-page tx surface
Multi-source brains crashed mid-import with Postgres 21000 ("more than one
row returned by a subquery used as an expression"). Root cause: putPage's
INSERT column list omitted source_id, so writes intended for a non-default
source (e.g. 'jarvis-memory') silently fabricated a duplicate row at
(default, slug). The schema has UNIQUE(source_id, slug) but DEFAULT 'default'
for source_id; calling putPage(slug, page) without source_id landed at
(default, slug) and ON CONFLICT updated the wrong row, leaving the intended
source row stale. Subsequent bare-slug subqueries inside the same tx —
(SELECT id FROM pages WHERE slug = $1) in getTags / removeTag / deleteChunks
/ removeLink / addLink (cross-product) — then matched 2 rows and crashed
with 21000, rolling back the entire import. Observed: 18 sync failures
against a 'jarvis-memory'-sourced brain.
Fix:
- putPage adds source_id to the INSERT column list (defaults 'default' for
back-compat).
- Every bare-slug page-id subquery becomes source-qualified
(AND source_id = $X) in both engines: createVersion, upsertChunks,
getChunks, addTag, removeTag, getTags, deleteChunks, removeLink,
addTimelineEntry, deletePage, updateSlug.
- addLink rewritten away from FROM pages f, pages t cross-product into a
VALUES + JOIN-on-(slug, source_id) shape mirroring addLinksBatch.
- engine.ts interface: 11 method signatures gain optional opts.sourceId
(or opts.{from,to,origin}SourceId for addLink/removeLink). All optional;
existing callers default to source='default' and behave identically.
- import-file.ts: importFromContent / importFromFile / importCodeFile take
opts.sourceId and thread txOpts = { sourceId } through every per-page tx
call. engine.getPage callsite source-scoped for accurate idempotency.
- commands/sync.ts: thread opts.sourceId at importFile (line 581 + 641),
un-syncable cleanup (487-498), delete phase (557), rename phase (574),
and post-sync extract phase (815-816).
- commands/reindex-code.ts: thread opts.sourceId at importCodeFile call.
- commands/extract.ts: extractLinksForSlugs / extractTimelineForSlugs accept
opts.sourceId and propagate via linkOpts / entryOpts.
- commands/reconcile-links.ts: ReconcileLinksOpts.sourceId was declared but
ignored end-to-end; now wired through getPage + addLink calls.
- commands/migrate-engine.ts: --force wipe switched to executeRaw('DELETE
FROM pages') to preserve the pre-PR all-sources semantic after deletePage
became default-source-scoped.
Regression test: test/source-id-tx-regression.test.ts (19 tests). Validates
two sources × same slug coexist; getTags/addTag/removeTag/deleteChunks/
upsertChunks/createVersion/addLink/addTimelineEntry/deletePage/updateSlug
source-scoped writes don't 21000; back-compat without opts targets
source='default'; addLink fail-fast on missing source-qualified endpoint;
importFromContent end-to-end tx thread without fabricating duplicate.
Adversarial review: Codex (gpt-5.5 reviewer) + Grok (xAI flagship reviewer)
3-round crew loop. Round 1: 2 HIGH (addTimelineEntry + extract.ts thread)
+ 2 MED. Round 2: 1 CRITICAL + 1 HIGH (deletePage + updateSlug bare-slug)
+ 2 MED. Round 3: 2 HIGH (getChunks + migrate-engine semantic regression
introduced by R2 fix). Round 4: both reviewers CLEAR.
Deferred to follow-up PRs (noted as TODO):
- src/commands/embed.ts source-aware threading (auto-embed at sync.ts:823
has a TODO; try/catch swallows the failure as best-effort).
- src/core/postgres-engine.ts:1511 / pglite-engine.ts:1446 putRawData
bare-slug (lower-impact metadata path).
- Read-surface bare-slug consistency cleanup (getLinks/getBacklinks/
getTimeline/getRawData/getVersions): non-mutating, won't 21000.
- reconcile-links.ts CLI --source flag exposure (internal opt is wired;
CLI parser is a UX feature for later).
Existing rows in production written under (default, slug) by the old
putPage when caller meant another source remain misrouted. Backfill
heuristics need install-specific knowledge of intended source and are
outside this PR's scope; surface as a deployment-side cleanup task.
bun run typecheck clean, bun run build clean, 19/19 regression tests pass,
4082 unit pass / 1 pre-existing fail (BrainRegistry test depending on
test-env ~/.gbrain/ absence — fails on untouched main, unrelated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(multi-source): plumb sourceId through performFullSync (PR #707 gap)
PR #707 fixed source_id routing for sync's incremental loop (lines 581/641)
but performFullSync (line 922) calls runImport without threading sourceId.
Result: full syncs route pages to default even with --source <id>. Verified
on v0.30.1 by direct PGLite probe after `gbrain sync --source X --full`:
all pages landed in default, not the named source.
Fix:
- runImport accepts sourceId in opts (programmatic only — no CLI flag,
preserving PR #707's design intent of `gbrain import` being default-only).
- runImport threads sourceId to importFile + importImageFile.
- performFullSync passes opts.sourceId to runImport.
- ImportImageOptions type accepts sourceId for runImport branch (importImageFile
body wiring deferred — image imports out of scope for current use case;
TS error fix only).
Verified: real sync test against /tmp/test-sync routes 1 page to "testsync"
source, 0 to default (post-fix). 19/19 source-id regression tests still pass.
Typecheck clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test: regression test for performFullSync sourceId threading
PR #707's existing 19-test suite at test/source-id-tx-regression.test.ts
covers the engine-layer transaction surface (putPage / addTag / etc.)
but does NOT exercise commands/sync.ts:performFullSync. Verified via
`grep -c 'performFullSync' test/source-id-tx-regression.test.ts → 0`.
This means the +18/-4 fix at sync.ts:892 (performFullSync passing
sourceId to runImport) had no automated coverage.
Adds 2 PGLite-only regression tests:
1. `performFullSync with --source routes pages to named source (not default)`
— fixture: temp git repo with 2 markdown files. Calls performSync with
{ full: true, sourceId: 'testsrc-pfs', noPull: true, noEmbed: true }.
Asserts pages.source_id = 'testsrc-pfs', not 'default'. Pre-fix: FAILS
(verified by checking out
|
||
|
|
96178d726e |
fix(subagent): v0.16.3 — bind Anthropic SDK correctly + enable tsc in CI (#318)
* fix(subagent): bind Anthropic SDK messages.create() correctly The makeSubagentHandler was casting `new Anthropic()` directly to MessagesClient, but MessagesClient.create() maps to sdk.messages.create(), not sdk.create(). Every subagent job immediately died with: client.create is not a function Fix: wrap the SDK instance so .create() delegates to .messages.create() with proper `this` binding via .bind(sdk.messages). Discovered on first production run of gbrain agent against Supabase. Co-Authored-By: Wintermute <wintermute@openclaw.ai> * chore(ci): add typescript typecheck to test pipeline + clean up baseline errors Root cause infra gap that let the v0.16.0 subagent bug ship: CI ran only `bun test`, which transpiles types without checking them. Type errors only surfaced at runtime, in production. Changes: - Add `typescript` devDep and a `typecheck` npm script (`tsc --noEmit`). - Chain `bun run typecheck` into `bun run test` so developers get the same pipeline locally that CI runs. - Flip `.github/workflows/test.yml` to invoke `bun run test` (the npm script, including typecheck) instead of `bun test` (runner only). - Clean up 100+ pre-existing type errors across 30+ files so the first run of `tsc --noEmit` is green. Root causes were: - `databaseUrl` → `database_url` rename drift in test fixtures (9 files) - `PageType` union missing `'meeting'` / `'note'` entries that are already used in both src and tests (link-extraction.ts comments acknowledged the gap) - `GBrainConfig.storage` field never declared despite being read in files.ts and operations.ts - `ErrorCode` union missing `'permission_denied'` - `OrchestratorOpts` shape changed; test callers not updated - Dead-code comparisons in migration orchestrators against narrowed status types - postgres.js `Row`-callback type drift on several `.map()` calls - Buffer-as-BodyInit assignment in supabase.ts (real but non-fatal runtime bug; Uint8Array slice works and is type-correct) - Various `as X` single-step casts that now need `as unknown as X` per TS's stricter structural-conversion rules - Bump `beforeAll` hook timeout to 30s on four PGLite-heavy tests that were flaky under parallel test execution: wait-for-completion, extract-fs, e2e/search-quality, e2e/graph-quality. All pass in isolation; timeouts only happened when dozens of PGLite instances init'd simultaneously. The new CI pipeline now fails on any type error across src/ or test/, giving us the compile-time regression guard the subagent fix depends on. * fix(subagent): bind Anthropic SDK messages.create() correctly Shipped bug: v0.16.0 cast `new Anthropic()` to `MessagesClient`, but `.create()` lives at `sdk.messages.create`, not on the top-level client. Every subagent job in production died on first LLM call with `client.create is not a function`. Discovered on the first `gbrain agent run` against Supabase. Fix: assign `sdk.messages` directly to the `MessagesClient` slot. `sdk.messages` IS the object with a callable `.create()`; the original bug was picking the wrong entry point on the SDK. No helper, no wrapper, no `.bind()` — JS method-call semantics preserve `this` at the call site because `subagent.ts:336` invokes `client.create(...)` with `client === sdk.messages`. The one-line assignment also typechecks cleanly against the existing `MessagesClient` interface (SDK's first `create` overload: `(MessageCreateParamsNonStreaming, Core.RequestOptions?) => APIPromise<Message>` is assignable structurally). This gives us compile-time regression protection: anyone reverting to `new Anthropic()` would fail tsc because `Anthropic` has no top-level `.create`. (The companion chore commit puts `tsc --noEmit` in CI so this guard is enforced.) Also adds a `makeAnthropic?: () => Anthropic` dep-injection seam so the factory default construction branch is testable without real API calls. Regression test drives one handler turn through a fake SDK, asserting `sdk.messages.create` is actually called. If someone later reverts to `new Anthropic()`, both guards fire: tsc fails AND the test fails. Co-Authored-By: Wintermute <wintermute@garrytan.com> * chore(tests): add bunfig.toml + 60s hook timeouts to stabilize PGLite-heavy suites After turning on tsc in CI (previous commit), running the full `bun run test` suite in one shot triggered flaky `beforeEach/afterEach hook timed out` failures on 8+ test files. Every failure traced to PGLite WASM init contention when many test files spin up fresh PGLite instances in parallel; each one alone passes in isolation. - `bunfig.toml` sets the global test hook timeout to 60s (default is 5s), covering every test file without per-file edits. - Individual `beforeAll(fn, 60_000)` / `beforeEach(fn, 15_000)` calls on the 8 tests that flaked most stay in place as explicit safety nets so a future bunfig config change doesn't silently re-introduce the flake. Result: 1997 pass, 0 fail on `bun run test` (117 tests added since the prior baseline by picking up typecheck-gated passes). No infrastructure flake tolerated in CI. * chore: bump version and changelog (v0.16.3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Wintermute <wintermute@openclaw.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
699db50a3d |
fix(extract+migrate): kill N+1 hang + v0.12.0 migration timeout (v0.12.1) (#198)
* feat(engine): add addLinksBatch + addTimelineEntriesBatch via unnest()
Multi-row INSERT...SELECT FROM unnest() JOIN pages ON CONFLICT DO NOTHING
RETURNING 1. 4 array-typed bound parameters (links) or 5 (timeline)
regardless of batch size, sidesteps Postgres's 65535-parameter cap.
Returns count of rows actually inserted (excluding ON CONFLICT no-ops
and JOIN-dropped rows whose slugs don't exist).
Per-row addLink / addTimelineEntry signatures and SQL behavior unchanged.
All 10 existing call sites compile and behave identically.
Tests: 11 PGLite cases (empty batch, missing optionals, within-batch dedup,
JOIN drops missing slug, half-existing batch, batch of 100) + 9 E2E
postgres-engine cases against real Postgres+pgvector.
* fix(migrate): pre-create btree helper in v8 + v9 dedup; bump phaseASchema timeout
Production bug: v0.12.0 schema migration timed out at Supabase Management API's
60s ceiling on brains with 80K+ duplicate timeline rows. The DELETE...USING
self-join was O(n²) without an index on the dedup columns.
Fix: pre-create idx_links_dedup_helper / idx_timeline_dedup_helper on the
dedup columns BEFORE the DELETE, drop after. Turns O(n²) into O(n log n).
On 80K+ rows the migration completes in <1s instead of timing out.
Also bumps the v0.12.0 orchestrator's phaseASchema timeout 60s -> 600s as
belt-and-suspenders for unforeseen slowness.
Exports MIGRATIONS for structural test assertions.
Tests: 2 structural assertions (helper-index DDL must appear in v8/v9 SQL
in the right order — catches regression even at 0-row scale) + 2 behavioral
regression tests (1000-row dedup completes <5s).
* perf(extract): kill N+1 dedup pre-load; switch to batched writes
Production bug: gbrain extract hung 10+ minutes producing zero output on
47K-page brains. The pre-load loop called engine.getLinks(slug) (or
getTimeline) once per page across engine.listPages({limit: 100000}) — 47K
serial round-trips over the Supabase pooler before the first file was read.
Both engines already enforced uniqueness at the SQL layer
(UNIQUE(from, to, link_type) on links, idx_timeline_dedup on timeline_entries).
The in-memory dedup Set was redundant insurance that became the bottleneck.
Fix: delete the pre-load entirely. Buffer 100 candidates per file walk,
flush via engine.addLinksBatch / engine.addTimelineEntriesBatch. ~99% fewer
DB round-trips per re-extract.
Also fixes counter accuracy: 'created' now counts rows actually inserted
(via batch RETURNING 1 row count). Re-run on a fully-extracted brain
prints 'Done: 0 links' instead of lying.
Dry-run mode keeps a per-run dedup Set so duplicate candidates from N
markdown files print exactly once, not N times.
Batch errors are visible in BOTH json and human modes — silent loss of
100 rows is worse than per-row error visibility.
Tests: extract-fs.test.ts (idempotency + truthful counter + dry-run dedup
+ perf regression guard <2s).
* chore: bump version + changelog (v0.12.1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: update CLAUDE.md for v0.12.1 (batch engine API, test counts)
Reflect what shipped in v0.12.1:
- New engine methods addLinksBatch + addTimelineEntriesBatch (PGLite via
unnest() + manual $N, postgres-engine via INSERT...SELECT FROM
unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING).
- extract.ts no longer pre-loads dedup set; candidates are buffered 100
at a time and flushed via the new batch methods.
- v0.12.0 orchestrator phaseASchema timeout bumped 60s to 600s.
- Test counts 1297 unit / 105 E2E to 1412 unit / 119 E2E.
- New test/extract-fs.test.ts covers the N+1 regression guard.
- BrainEngine method count 37/38 to 40.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|