admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
333 Commits 507 Branches 1 Tag
814258dda67945ffec9457a1e73980e947b7e462
Commit Graph
5 Commits
Author SHA1 Message Date
Garry TanGitHubvinsew313094319-sudoClaude Opus 4.7
 		c9652443cf v0.32.7 feat: CJK fix wave — 6 layers from one root cause (closes vinsew + 313094319-sudo PRs) (#898) 
* feat: shared CJK detection module (cjk.ts)

Foundation for the CJK fix wave. Single source of truth for CJK ranges
(Han, Hiragana, Katakana, Hangul Syllables), the slug-char string used
by adjacent validators, sentence + clause delimiter sets, the 30%
density threshold for word counting, and a LIKE-pattern escape helper.

Replaces the inline hasCJK regex at expansion.ts:58 so four-place
drift becomes impossible. countCJKAwareWords uses density threshold
(per codex outside-voice C13) so a long English doc with one Japanese
term stays whitespace-tokenized, not char-split.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* feat: migration v51 + pages.chunker_version/source_path columns

Schema-level support for the v0.32.7 CJK wave. Two new columns on pages:

  - chunker_version SMALLINT NOT NULL DEFAULT 1 — bumped to
    MARKDOWN_CHUNKER_VERSION (2) on every new import. The post-upgrade
    gbrain reindex --markdown sweep walks chunker_version < 2 to find
    pre-bump rows and rebuilds them.

  - source_path TEXT — captures the repo-relative path at import time
    so sync's delete/rename code can resolve frontmatter-fallback
    slugs (CJK / emoji / exotic-script files where the path itself
    doesn't derive a slug).

Both columns plumbed through PageInput, partial indexes scoped to
markdown-only / non-null. PGLite + Postgres parity via the standard
ALTER TABLE ... IF NOT EXISTS shape.

Replaces the original PR #599 plan of folding MARKDOWN_CHUNKER_VERSION
into content_hash. Codex outside-voice C2 caught that as a no-op:
performSync gates on actual file change, not hash-would-differ, so
the fold never reached existing pages. Column + sweep is the real fix.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* feat: CJK-aware slugify + SLUG_SEGMENT_PATTERN + adjacent validators

slugifySegment now preserves Han / Hiragana / Katakana / Hangul Syllables
with NFC re-normalization after the NFD-strip-accents pass so Hangul
Jamo recomposes back into precomposed syllables that fall inside the
whitelist. café still slugifies to cafe (regression preserved — iron
rule).

SLUG_SEGMENT_PATTERN (consumed by takes-holder validation) extended
with CJK_SLUG_CHARS in the same commit so CJK slugs aren't rejected by
adjacent validators downstream. Codex outside-voice C4 caught this
exact half-fix in the original plan — leaving the pattern ASCII-only
would have shipped a feature where the slugify produced 品牌圣经 but
adjacent validators flagged it.

src/core/operations.ts: validatePageSlug + validateFilename also
extended with CJK ranges. matchesSlugAllowList is unchanged (works on
string prefixes, no character class).

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* feat: recursive chunker — MARKDOWN_CHUNKER_VERSION + CJK splitting + maxChars cap

Four coordinated chunker changes for the v0.32.7 wave:

  - MARKDOWN_CHUNKER_VERSION = 2 exported. Folded into pages.chunker_version
    so the post-upgrade reindex sweep can find pre-bump pages.

  - countWords delegated to countCJKAwareWords from cjk.ts (30% density
    threshold). Below threshold: whitespace-token count (English-dominant
    docs stay tokenized). At/above: char count (Chinese paragraphs actually
    split instead of being treated as one 8192-token-overflowing word).

  - DELIMITERS extends L2 (sentences) with 。！？ and L3 (clauses) with
    ；：，、. CJK punctuation now produces real chunk boundaries.

  - maxChars hard cap (default 6000) with sliding-window splitByChars and
    500-char overlap. Catches pathological whitespace-less inputs that the
    word-level pipeline can't bound (pure-Han paragraphs, base64 blobs,
    long URLs). Applied to both single-short-chunk and merged-chunks
    paths.

  - splitOnWhitespace falls through to char-slice when ANY single "word"
    exceeds target chars (the greedy /\S+/g regex returns a whole CJK
    paragraph as one "word"; without this, the L4 fallback produces one
    huge piece). Pre-fix this was the silent-failure path.

Tests in test/chunkers/recursive.test.ts: 9 new cases — pure Chinese,
Japanese + 。, Korean Hangul, mixed CJK+English, 20KB CJK with overlap,
single-short-chunk maxChars edge, pure-English regression.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* feat: PGLite CJK keyword fallback + engine chunker_version/source_path passthrough

PGLite uses websearch_to_tsquery('english') over to_tsvector('english'),
which can't tokenize CJK. Pre-fix, CJK queries returned empty results
on PGLite brains even with proper embeddings.

searchKeyword + searchKeywordChunks now branch on hasCJK(query):

  - ASCII path: unchanged. websearch_to_tsquery('english') continues
    to drive FTS. No regression risk.

  - CJK path: switches to ILIKE '%' || $qLike || '%' ESCAPE '\\' over
    chunk_text with two distinct param bindings ($qLike escaped for
    the ILIKE clause, $qRaw raw for the ranking arithmetic). Empty
    $qRaw guard bails before binding. Bigram-frequency-count ranking
    via (LENGTH(chunk_text) - LENGTH(REPLACE(chunk_text, $qRaw, ''))) /
    LENGTH($qRaw) approximates ts_rank semantics; position-in-chunk
    tiebreaker so earlier matches outrank later ones at the same
    occurrence count.

Codex outside-voice C8 caught the original plan's one-param shortcut
(escaped chars can't be reused as ranking substrings) + missing
ESCAPE clause + asymmetric whitespace strip. C9 corrected the FTS
dialect (websearch_to_tsquery, not to_tsvector('simple')).

Source-boost CASE, hard-exclude clause, visibility clause, and the
DISTINCT ON (slug) page-dedup all survive on both branches. Postgres
engine path stays untouched (multi-tenant Postgres deployments can
install pgroonga / zhparser for CJK; out of scope for this wave).

Postgres + PGLite putPage both extended to write chunker_version
and source_path columns (with COALESCE(EXCLUDED.x, pages.x) so
auto-link / code-reindex callers that don't supply them don't blank
existing values).

Tests: 8 new cases covering Chinese / Japanese / Korean substring
search, bigram ranking (3-hit > 1-hit), LIKE-meta-char escape
(literal % does not wildcard), English query stays on FTS path.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>
Co-Authored-By: 313094319-sudo <313094319-sudo@users.noreply.github.com>

* feat: import-file frontmatter-slug fallback + audit JSONL

importFromFile gains a fallback branch: when slugifyPath returns
empty (emoji / Thai / Arabic / exotic-script filename — including
post-CJK-wave files that still don't slugify) AND the frontmatter
declares a slug, the frontmatter slug becomes authoritative.

Anti-spoof rule preserved unchanged: when slugifyPath produces a
non-empty path slug AND the frontmatter slug claims a different one,
the file is still rejected. notes/random.md cannot impersonate
people/elon via frontmatter.

D6=B error string when both path slug AND frontmatter slug are empty:
"Filename produces no usable slug. Add a 'slug:' to the frontmatter,
or rename the file to use ASCII / Chinese / Japanese / Korean
characters." Honest about the actually-supported scripts.

Every import now populates pages.chunker_version (set to
MARKDOWN_CHUNKER_VERSION) and pages.source_path (repo-relative). These
drive the post-upgrade reindex sweep + sync's delete/rename slug
resolution.

NEW src/core/audit-slug-fallback.ts — weekly ISO-week-rotated JSONL
at ~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl. Per codex C7, info
events don't belong in sync-failures.jsonl (which gates bookmark
advancement); separate audit surface keeps the failure-handling code
unchanged. logSlugFallback emits a stderr line AND appends to the
audit file (D7=D dual logging).

Tests: 5 new import-file cases (小米 with no frontmatter slug, 🚀.md
with frontmatter fallback, 🌟🚀.md friendly D6=B error, anti-spoof
regression, chunker_version + source_path populated). 6 new audit
cases covering write, weekly rotation, 7-day window, corrupt-row
tolerance.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* feat: git() helper hardening + core.quotepath=false for CJK paths

git CLI emits CJK paths as quoted octal escapes (\345\223\201 ...) by
default in diff --name-status output. Pre-fix, buildSyncManifest
silently dropped these paths because downstream filesystem lookups
saw the literal escape string. gbrain sync reported added=0 while
git had the file committed.

git() helper refactored:
  - New signature: git(repoPath, args: string[], configs?: string[])
  - Config flags emit BEFORE -C and BEFORE the subcommand (git CLI
    requires this order)
  - core.quotepath=false always prepended
  - Future callers needing extra -c config pass configs:[]; no more
    inlining -c into args (the silent-future-drift footgun codex C12
    flagged as a related concern)

New invariant test in test/sync.test.ts pins the emit order.

NEW test/e2e/sync-cjk-git.test.ts — real-git E2E in a tmpdir. Spawns
real git via execFileSync, commits a Chinese-named markdown file,
drives the helper through buildSyncManifest, asserts the manifest
contains the UTF-8 path (not the octal-escape form). Closes the
real-CLI-behavior gap that unit tests can't cover (the helper builds
the right args; only an E2E proves git actually emits UTF-8 under
the flag).

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* feat: gbrain reindex --markdown sweep command

NEW src/commands/reindex.ts — operator-facing markdown re-chunk
sweep. Walks SELECT slug, source_path FROM pages WHERE
page_kind = 'markdown' AND chunker_version < MARKDOWN_CHUNKER_VERSION
in 100-row batches, ordered by id ASC so partial-completion re-runs
pick up where they left off.

For rows with non-null source_path: re-imports via importFromFile
when the file exists on disk. For rows without (legacy pre-migration
backfill): fallback to importFromContent using the stored markdown
body.

Flags: --markdown (target selector), --limit N, --dry-run, --json,
--no-embed (offline / CI / test path that lets the chunker run
without a configured AI gateway), --repo PATH.

Wired into src/cli.ts dispatch table. Will also be invoked
automatically by gbrain upgrade's post-upgrade hook (next commit) so
chunker-version bumps reach existing markdown pages without an
explicit operator action.

Tests in test/reindex.test.ts: 5 cases covering dry-run, actual
sweep, idempotent re-run, --limit cap, skipped-already-at-current.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>
Co-Authored-By: 313094319-sudo <313094319-sudo@users.noreply.github.com>

* feat: post-upgrade chunker-bump cost prompt + auto-reindex sweep

Wires the chunker-version bump into gbrain upgrade so existing brains
heal automatically. Three new pieces:

NEW src/core/embedding-pricing.ts — EMBEDDING_PRICING map keyed
provider:model (OpenAI text-embedding-3-large + 3-small + ada-002,
Voyage 3-large + 3). lookupEmbeddingPrice returns 'known' or
'unknown' shape so the cost-estimate prompt can degrade gracefully
for unknown providers rather than fabricate numbers (codex C3).
estimateCostFromChars uses 3.5 chars/token approximation.

NEW src/core/post-upgrade-reembed.ts — pure-ish functions for the
cost-estimate prompt:
  - computeReembedEstimate: real SQL against
    COUNT(*) + COALESCE(SUM(LENGTH(compiled_truth)) + SUM(LENGTH(timeline))
    on the chunker_version-filtered query. No phantom markdown_body
    column (codex C3 caught the original plan referencing nonexistent
    schema fields).
  - formatReembedPrompt: pure string formatter for the stderr line.
  - runPostUpgradeReembedPrompt: orchestrates the prompt + 10-second
    Ctrl-C window. TTY-only wait so non-TTY upgrades (CI, cron-driven,
    headless) don't hang. GBRAIN_NO_REEMBED=1 bails out entirely
    with a doctor-warning marker; GBRAIN_REEMBED_GRACE_SECONDS=0
    skips the wait.

src/commands/upgrade.ts: after apply-migrations runs, the new prompt
fires through the gateway's configured embedding model, then invokes
gbrain reindex --markdown automatically if the user proceeds.
Wrapped in try-catch so a reindex failure is non-fatal — the user
can re-run manually.

Tests in test/upgrade-reembed-prompt.test.ts: 11 cases covering real
SQL counts, unknown-provider fallback, TTY / non-TTY paths,
GBRAIN_NO_REEMBED bail-out, GBRAIN_REEMBED_GRACE_SECONDS=0 skip-wait.

Codex outside-voice C2 caught the original plan as a no-op
(performSync doesn't re-import unchanged files just because
content_hash would differ). The migration v51 column + this sweep
+ this prompt is the real fix that actually reaches existing pages.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* feat: doctor slug_fallback_audit check + CJK roundtrip E2E

gbrain doctor learns a new slug_fallback_audit check (v0.32.7).
Reads the latest week of ~/.gbrain/audit/slug-fallback-*.jsonl,
counts info-severity entries from the last 7 days, surfaces the
total as an ok-status line. No health-score docking; no warning.

sync-failures.jsonl (which gates bookmark advancement) stays
untouched — info events live in their own surface per codex C7.

NEW test/e2e/cjk-roundtrip.test.ts — proves the wave delivers end-
to-end. PGLite-in-memory fixture with Chinese / Japanese / Korean
content. Each page: importFromContent → chunkText (CJK-aware) →
searchKeyword (LIKE-branch with bigram count). Asserts every CJK
query lands on its source page. ASCII regression: an English query
still uses the FTS path on the same brain. Vector path skips
gracefully without OPENAI_API_KEY.

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>

* chore: bump version and changelog (v0.32.7)

CJK fix wave — six layers from one root cause. Three originating PRs
from @vinsew and one extracted from @313094319-sudo's #765 land
together as a coherent collector. Codex outside-voice review on the
plan caught four critical bugs the eng review missed (no-op
re-embed, SLUG_SEGMENT_PATTERN half-fix, LIKE SQL needing two
distinct param bindings, countCJKAwareWords over-splitting on
English+1-CJK-term docs). All four addressed in the implementation.

TODOS.md: resolved the v0.32.x PGLite CJK keyword fallback entry;
filed five v0.33+ follow-ups (Postgres CJK FTS via pgroonga / wider
Unicode property escapes / -z NUL git framing / CJK overlap context /
other non-Latin scripts / embedding pricing refresh mechanism).

Co-Authored-By: vinsew <vinsew@users.noreply.github.com>
Co-Authored-By: 313094319-sudo <313094319-sudo@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix: review findings — forceRechunk + source_path lookup (codex post-merge)

Two critical issues caught by codex adversarial on the post-merge tree:

F1 — Reindex sweep was a no-op on unchanged-source pages. importFromContent
short-circuits on existing.content_hash === hash BEFORE the chunker runs,
so the v0.32.7 MARKDOWN_CHUNKER_VERSION bump (and master's v0.32.2
stripFactsFence privacy strip) never reached pages whose markdown body
hadn't been edited.

Fix: new `forceRechunk?: boolean` option on importFromContent + importFromFile.
When set, the hash short-circuit is bypassed and the page re-runs the full
chunk + write pipeline. `gbrain reindex --markdown` now passes forceRechunk:
true on every row. This means:
  - The CJK chunker bump actually reaches existing markdown pages.
  - Master's v0.32.2 stripFactsFence applies retroactively too — any
    pre-strip private fact bytes lingering in content_chunks get cleared
    when the v0.32.7 post-upgrade sweep runs.

New test in test/reindex.test.ts seeds a page, runs the sweep, mocks a
stale chunker_version=1 without changing compiled_truth, runs the sweep
again, asserts chunker_version is bumped despite hash match.

F4 — Sync delete/rename still used resolveSlugForPath(path) only, ignoring
the new pages.source_path column added in v52. Frontmatter-fallback pages
(emoji-only / Thai / Arabic filenames where slugifyPath returns empty and
the slug came from the markdown frontmatter) would orphan on delete or
rename because the path-derived slug doesn't match the stored slug.

Fix: new exported helper resolveSlugByPathOrSourcePath(engine, path,
sourceId?) queries pages.source_path first, falls back to
resolveSlugForPath when no row matches. Threaded into 3 call sites in
sync.ts (un-syncable modified cleanup at :531, deletes at :603, rename
oldSlug at :622). Best-effort: query errors fall through to the legacy
path so pre-migration brains still work.

3 new test cases in test/sync.test.ts cover: stored-slug lookup hits,
fallback when no source_path row exists, and source_id scoping when two
sources have the same source_path value.

Codex finding #3 (reindex not in CLI_ONLY) was verified as a false
positive — CLI_ONLY is the set that doesn't need an engine; reindex
correctly belongs to the engine-backed dispatch.

302 wave tests pass / 0 fail. bun run verify green.

* docs: update CLAUDE.md + llms-full.txt for v0.32.7 CJK fix wave

CLAUDE.md Key Files: added entries for the five new modules introduced by
the wave — src/core/cjk.ts (shared detection + delimiters + density
threshold), src/core/audit-slug-fallback.ts (weekly JSONL),
src/core/embedding-pricing.ts (post-upgrade cost lookup table),
src/core/post-upgrade-reembed.ts (prompt + grace window), and
src/commands/reindex.ts (chunker_version sweep with forceRechunk).

Also noted src/commands/sync.ts:resolveSlugByPathOrSourcePath — the
F4 codex post-merge fix that wires the new pages.source_path column
into sync delete/rename so frontmatter-fallback pages don't orphan.

CLAUDE.md Commands: added a v0.32.7 section covering `gbrain reindex
--markdown`, the new doctor slug_fallback_audit check, PGLite CJK
keyword fallback in `gbrain search`, and the post-upgrade
chunker-bump cost prompt with its env-var overrides.

llms-full.txt: regenerated via bun run build:llms (CI gate runs the
generator on every release; commit must include the bundle).

README.md: no changes needed — v0.32.7 is internal correctness
across the existing pipeline, not a new skill or setup story.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: vinsew <vinsew@users.noreply.github.com>
Co-authored-by: 313094319-sudo <313094319-sudo@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-11 22:54:35 -07:00
Garry TanGitHubClaude Opus 4.6
 		91ced664b6 feat: Voice v0.8.0 + feature discovery + Edge Function removal (#55) 
* chore: remove Supabase Edge Function MCP deployment

The Edge Function never worked reliably. All MCP traffic goes through
self-hosted server + ngrok tunnel. Removes deploy-remote.sh, edge-entry.ts,
supabase/functions/, .env.production.example, and CHATGPT.md (OAuth not
implemented).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite MCP docs for self-hosted + ngrok deployment

All per-client guides updated from Edge Function URLs to self-hosted
server + ngrok tunnel pattern. DEPLOY.md rewritten with local vs remote
paths. ALTERNATIVES.md now shows self-hosted as primary, with ngrok,
Tailscale, and Fly.io/Railway comparison.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: voice recipe v0.8.0 — 25 production patterns from real deployment

Identity separation, pre-computed bid system, conversation timing fix,
proactive advisor mode, radical prompt compression, OpenAI Realtime
Prompting Guide structure, auth-before-speech, brain escalation, stuck
watchdog, never-hang-up rule, thinking sounds, fallback TwiML, tool set
architecture, trusted user auth, caller routing, dynamic VAD, on-screen
debug UI, live moment capture, belt-and-suspenders post-call, mandatory
3-step post-call, WebRTC parity, dual API events, report-aware query
routing. WebRTC pseudocode updated with native FormData and 6 gotchas.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: post-upgrade feature discovery framework

upgrade.ts captures old version before upgrading, then execs
gbrain post-upgrade (new binary) to read migration files and print
feature pitches. Migration files get YAML frontmatter with feature_pitch
field (headline, description, recipe, tiers). CLI prints excited builder
tone post-upgrade. v0.8.0 migration offers voice setup with environment
detection (server vs local) and 3-tier progressive disclosure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add Voice section to README with WebRTC screenshot + tweet link

Her out of the box: voice-to-brain with 25 production patterns. WebRTC
client screenshot embedded. Remote MCP section rewritten for self-hosted
+ ngrok. Setup block genericized.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add recipe validation tests + genericize personal refs

5 new integration tests: secrets completeness, semver version, requires
resolution, all-recipes-parse, no-personal-references. Test fixture
genericized. CLAUDE.md/TODOS.md/SKILLPACK updated for v0.8.0. build:edge
script removed from package.json.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.8.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-11 10:52:30 -10:00
Garry TanGitHuborendi84Claude Opus 4.6mattbratoschangergosumeric-hthirresihowardpen9joshua-morris
 		8de04d3827 fix: community fix wave — 9 PRs, 8 contributors (v0.6.1) (#38) 
* fix: validateSlug accepts ellipsis filenames, rejects only real path traversal

Changed regex from /\.\./ to /(^|\/)\.\.($|\/)/ so filenames with "..." (like
YouTube transcripts, TED talks, podcast titles) are no longer falsely rejected.
The old regex matched ".." anywhere as a substring. The new one only matches ".."
as a complete path component (e.g., ../foo, foo/../bar, bare ..).

Fixes 1.2% silent data loss on real-world import corpora.

Co-Authored-By: orendi84 <orendi84@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: import walker skips node_modules, handles broken symlinks, supports .mdx

Three improvements to the file walker:
- Skip node_modules directories (prevents crashes importing JS/TS projects)
- try/catch around statSync for broken symlinks (warns and continues)
- Accept .mdx files alongside .md (extends to slugifyPath and isSyncable)

Co-Authored-By: mattbratos <mattbratos@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: init exits cleanly, auto-creates pgvector, updates Supabase UI hint

Three init improvements:
- process.stdin.pause() after reading URL input (prevents event loop hang)
- Auto-run CREATE EXTENSION IF NOT EXISTS vector with fallback message
- Update Supabase session pooler navigation hint to match current dashboard UI

Co-Authored-By: changergosum <changergosum@users.noreply.github.com>
Co-Authored-By: eric-hth <eric-hth@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* perf: parallelize keyword search with embedding pipeline

Run keyword search concurrently with the embed+vector pipeline instead of
sequentially. Keyword search has no embedding dependency so it can overlap
with the OpenAI API call, saving ~200-500ms per search.

Co-Authored-By: irresi <irresi@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: update Hermes Agent link to NousResearch GitHub repo

Co-Authored-By: howardpen9 <howardpen9@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add community PR wave process to CLAUDE.md

Documents the fix wave workflow: categorize, deduplicate, collector branch,
test, close with context, ship as one PR with attribution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: bump version and changelog (v0.6.1)

Community fix wave: 9 PRs re-implemented with full test coverage.
6 bug fixes, 1 perf improvement, 2 feature additions, 8 contributors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: migrate gstack from vendored to team mode

Remove vendored .claude/skills/gstack/ from git tracking. The global install
at ~/.claude/skills/gstack/ is the source of truth. Each developer runs
`cd ~/.claude/skills/gstack && ./setup` to set up symlink stubs locally.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: untrack skill symlink stubs

These are generated locally by gstack's ./setup script. Not project code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: credit community contributors in CHANGELOG

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: update OpenClaw links from .com to .ai

openclaw.com is a parked page. openclaw.ai is the real product.

Co-Authored-By: joshua-morris <joshua-morris@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: orendi84 <orendi84@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: mattbratos <mattbratos@users.noreply.github.com>
Co-authored-by: changergosum <changergosum@users.noreply.github.com>
Co-authored-by: eric-hth <eric-hth@users.noreply.github.com>
Co-authored-by: irresi <irresi@users.noreply.github.com>
Co-authored-by: howardpen9 <howardpen9@users.noreply.github.com>
Co-authored-by: joshua-morris <joshua-morris@users.noreply.github.com>
 2026-04-10 19:34:01 -10:00
Garry TanGitHubClaude Opus 4.6
 		27eb87f1f4 feat: slugify file paths with spaces and special characters (v0.5.1) (#29) 
* feat: slugify file paths with spaces and special characters

Apple Notes files (e.g., "2017-05-03 ohmygreen.md") now get clean,
URL-safe slugs instead of raw filenames with spaces. Spaces become
hyphens, special chars are stripped, accented chars normalize to ASCII.

Both import (inferSlug) and sync (pathToSlug) pipelines now use the
same slugifyPath() function, eliminating the case-preservation mismatch.

* feat: one-time migration to slugify existing page slugs

Extends Migration interface with optional TypeScript handler for
application-level data transformations. Adds version 2 migration that
renames all existing slugs to their slugified form, including link
rewriting. Collision handling via try/catch + warning.

* test: slugify unit tests, E2E tests, and updated expectations

22 new unit tests for slugifySegment and slugifyPath covering spaces,
special chars, unicode, dots, empty segments, and all 4 bug report
examples. Updated pathToSlug tests for new lowercase behavior. Updated
E2E tests for slugified Apple Notes slugs. Added 2 new E2E tests for
space-named file import and sync.

* chore: bump version and changelog (v0.5.1)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: fix changelog example to show directory with spaces

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 11:42:32 -07:00
Garry TanGitHubClaude Opus 4.6
 		912a321cfa GBrain v0.4.0 — production agent documentation + reference architecture (#10) 
* fix: widen validateSlug to accept any filename characters

Git is the system of record. Slugs are lowercased repo-relative paths.
The restrictive regex rejected spaces, parens, and special chars, blocking
5,861 Apple Notes files from importing. Now only rejects empty slugs,
path traversal (..), and leading slash.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: enable RLS on all tables with BYPASSRLS safety check

Without RLS, the Supabase anon key gives full read access to the DB.
Enable RLS on all 10 tables with no policies — the postgres role
(used by gbrain via pooler) has BYPASSRLS and is unaffected. Only
enables if the current role actually has BYPASSRLS privilege to
avoid locking ourselves out on non-Supabase setups.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: import resilience — 5MB limit, error suppression, structured progress

Raise MAX_FILE_SIZE from 1MB to 5MB for Apple Notes with attachments.
Track error patterns and suppress after 5 identical errors to prevent
5,861 identical warnings from killing the agent process. Replace \r
progress bar with structured log lines (rate, ETA) for agent parsing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: init detects IPv6-only Supabase URLs, adds pgvector check

Detect db.*.supabase.co direct URLs and warn about IPv6 failure.
On ECONNREFUSED/ETIMEDOUT to Supabase, suggest the Session pooler
connection string with exact dashboard click path. Check for pgvector
extension after connecting and fail with clear instructions if missing.
Update wizard hints to show pooler URL format.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add pre-ship requirement for E2E tests

E2E tests against real Postgres+pgvector must pass before /ship or
/review. Adds the requirement to CLAUDE.md so all agents enforce it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: parallel import with per-worker engine instances

Refactor PostgresEngine to support instance-level DB connections instead
of only the module-global singleton. Each worker gets its own connection
with poolSize:2 (vs 10 for the main engine), so 8 workers = 16 connections.

Add --workers N flag to gbrain import. Workers pull from a shared queue
and use independent engine instances — no transaction context corruption.

The bottleneck is network round-trips to Supabase (one per page upsert).
Parallel workers cut import time proportionally.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: automatic schema migration runner

Migrations are embedded as string constants in migrate.ts (survives
Bun --compile). Each migration runs in a transaction for clean rollback
on failure. Runs automatically on initSchema() — no manual step needed
when a user updates the gbrain binary against an older DB.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: pluggable storage backend (S3 + Supabase Storage + local)

Add StorageBackend interface with three implementations:
- S3Storage: works with AWS S3, Cloudflare R2, MinIO (any S3-compatible)
- SupabaseStorage: uses Supabase Storage REST API with service role key
- LocalStorage: filesystem-based, for testing

Add file-resolver.ts with fallback chain: local file → .redirect
breadcrumb → .supabase marker → storage backend. Supports the
three-stage migration (mirror → redirect → clean).

Add yaml-lite.ts for parsing marker and breadcrumb files without
adding a YAML dependency.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: gbrain doctor command — health checks with --json output

Checks: connection, pgvector extension, RLS on all tables, schema
version, embedding coverage. Outputs structured JSON with --json flag
for agent parsing. Exit code 0 if healthy, 1 if issues found.

Agents should run gbrain doctor --json when any command fails.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite setup skill + README for agent-first DX

Setup skill: add Why Supabase, step-by-step project creation, explicit
agent instructions (nohup for large imports, doctor on failure, don't
ask for anon key), available init flags, file migration offer after
first import. Remove ClawHub references.

README: simplify to single OpenClaw install path, remove ClawHub, fix
squatted npm name to github:garrytan/gbrain, add Supabase settings
note about Session pooler.

Add Apple Notes test fixtures with spaces and parens in filenames for
E2E testing of the slug fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add RLS verification, schema health, and nohup hints to maintain skill

Maintenance skill now checks RLS status and schema version as part of
periodic health checks. Adds nohup pattern for large embedding refreshes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: import resume checkpoint + Supabase smart URL parsing

Import resume: saves checkpoint every 100 files to ~/.gbrain/import-checkpoint.json.
On restart with same directory and file count, skips already-processed files.
Use --fresh to ignore checkpoint and start over. Cleared on successful completion.

Supabase admin: extractProjectRef() parses any Supabase URL format (dashboard,
direct, pooler, project URL) to extract the project ref. discoverPoolerUrl()
uses the Management API to find the correct pooler connection string (including
the exact region prefix). checkRls() verifies RLS status via the API.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add 56 unit tests for all new code

8 new test files covering every feature added in this branch:
- slug-validation.test.ts: spaces, parens, unicode, path traversal (10 tests)
- yaml-lite.test.ts: parse + stringify, marker/redirect formats (9 tests)
- supabase-admin.test.ts: extractProjectRef for 4 URL formats (7 tests)
- migrate.test.ts: version export, runMigrations callable (2 tests)
- storage.test.ts: LocalStorage CRUD + createStorage factory (14 tests)
- file-resolver.test.ts: fallback chain, redirect, marker parsing (6 tests)
- import-resume.test.ts: checkpoint save/load/resume/fresh (6 tests)
- doctor.test.ts: module export, CLI registration (3 tests)

Total: 184 pass, 0 fail (up from 128).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: bulk chunk INSERT + E2E tests for all new features

Bulk INSERT: upsertChunks now builds a multi-row VALUES query instead
of inserting chunks one-by-one. Reduces DB round-trips by ~50x per page.

E2E tests added to mechanical.test.ts:
- Slug with special chars: import Apple Notes fixtures with spaces/parens,
  verify search finds them, verify idempotency
- RLS verification: check pg_tables.rowsecurity on all tables, verify
  current user has BYPASSRLS
- Doctor command: verify exit 0 on healthy DB, --json produces valid JSON
  with check structure
- Parallel import: --workers 2 produces same page count as sequential

Unit tests added:
- setup-branching.test.ts: IPv6 detection, defaultWorkers auto-tuning,
  smart URL parsing across all Supabase URL formats

Fixtures added:
- large/big-file.md (2.1MB) for testing raised file size limit
- apple-notes/ fixtures already existed

Total: 200 pass, 0 fail (up from 184).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: --json on init/import, file migration CLI, lifecycle tests

--json flag: init and import now support --json for structured output.
Agents get parseable JSON instead of human-readable text.

File migration CLI: implement mirror, unmirror, redirect, restore,
clean, and status subcommands for the three-stage file migration
lifecycle (local → mirrored → redirected → cloud-only).

File migration tests: full lifecycle test covering every transition
in the state machine (LOCAL → MIRROR → UNMIRROR → REDIRECT → RESTORE
→ CLEAN), including edge cases and file resolver at each stage.

Bulk chunk INSERT: upsertChunks now builds multi-row parameterized
VALUES query, reducing round-trips per page from ~50 to 1.

Total: 207 pass, 0 fail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: thorough E2E tests for parallel import concurrency

Replace the weak single-comparison parallel import test with 7 tests:
- Sequential baseline: capture page count, chunk count, and all slugs
- --workers 2: verify page count matches sequential
- Chunk count matches (no duplicates from concurrent writes)
- Page slugs match exactly
- No duplicate pages (SQL GROUP BY HAVING count > 1)
- No duplicate chunks (SQL GROUP BY page_id, chunk_index)
- --workers 4: also works correctly
- Re-import with workers is idempotent

These tests catch the exact bug Codex found (db.ts singleton causing
concurrent transaction corruption) by verifying data integrity after
parallel writes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add batch embedding queue as P1 TODO

Deferred during eng review (per-worker embedding is good enough for now).
Revisit after profiling real imports to confirm embedding is the bottleneck.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: E2E test failures — fixture counts, arg parsing, doctor exit code

Fix fixture count assertions: 13 → 16 pages (added apple-notes + large file),
companies 2 → 3 (ohmygreen), concepts 3 → 5 (notes, big-file).

Fix --workers arg parsing: the worker count value (e.g. "2") was being
picked up as the directory arg. Skip flag values when finding the dir.

Fix doctor exit code: warnings (like missing embeddings) should exit 0,
only actual failures exit 1. E2E tests import with --no-embed, so
embeddings are always WARN.

Fix E2E CLI tests: add initCli() before doctor and parallel import
tests so ~/.gbrain/config.json exists for the subprocess.

All E2E tests pass: 63 pass, 0 fail.
All unit tests pass: 207 pass, 0 fail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.4.0

New CHANGELOG entry for all post-0.3.0 features (doctor, storage backends,
parallel import, resume checkpoints, RLS, schema migrations, --json output).
Version bumped 0.3.0 → 0.4.0 across all manifests.

CLAUDE.md: test count 9→19, skill count 8→7, added key files.
CONTRIBUTING.md: fixture count 13→16, added missing source files.
README.md: added gbrain doctor to commands, fixed stale welcome PRs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add GBRAIN_SKILLPACK.md reference architecture

Production agent patterns from a real deployment with 14,700+ brain files.
Covers: entity detection on every message, brain-first lookup protocol,
7-step enrichment pipeline with tiered API spend, compiled truth + timeline,
source attribution with mandatory citations, meeting ingestion with entity
propagation, cron schedule with quiet hours and travel-aware timezone,
YouTube/media ingestion via Diarize.io, integration guides for ClawVisor,
Circleback webhooks, and Quo/OpenPhone SMS. Opens with the Vannevar Bush
memex framing and the originals folder for capturing intellectual capital.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite README opener with memex pitch and production architecture

Replace code-first opener with mimetic-desire pitch: Vannevar Bush memex
tagline, production brain numbers (10K+ files, 3K+ people, 13 years of
calendar), "ask it anything" examples, compounding thesis.

New sections: The Compounding Thesis (read-write loop), Architecture
(three-column diagram), What a Production Agent Looks Like (SKILLPACK
reference), How gbrain fits with OpenClaw (three-layer complement).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update skills with brain-first lookup, entity detection, heartbeat

setup: Phase D rewritten with brain-first lookup protocol (gbrain search
→ query → get → grep fallback), sync-after-write rule, memory_search
complement table.

query: token-budget awareness (chunks not full pages), source precedence
hierarchy (user > compiled truth > timeline > external).

ingest: entity detection on every message (scan, check brain, create or
enrich, commit and sync).

maintain: heartbeat integration (doctor, embed --stale, sync verification,
stale compiled truth detection).

briefing: gbrain-native context loading (search attendees before meetings,
search sender before email, daily deal/meeting/commitment queries).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add OpenClaw positioning to README opener

Make it clear up top that GBrain is built for OpenClaw agents and
works with any OpenClaw deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: credit Karpathy's Knowledge LLM vision, add origin story

GBrain started as Karpathy's LLM wiki idea built for real. Worked great
until the brain hit thousands of files and grep fell apart. GBrain is the
search layer that had to exist once the brain outgrew grep.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-09 10:17:13 -07:00