From b7f3dc9709535be255075554f3e715ec84dbb4a3 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 11 Apr 2026 17:44:10 -1000 Subject: [PATCH] feat: skills reference actual gbrain file commands - Filing rules document upload-raw, signed-url, and .redirect.yaml format - Ingest skill uses gbrain files upload-raw for raw source preservation - Maintain skill adds file storage health checks - Setup skill adds storage configuration phase with migration guidance - Voice recipe uses upload-raw for call audio storage - Migration v0.9.0 with complete storage setup instructions --- recipes/twilio-voice-brain.md | 6 +- skills/_brain-filing-rules.md | 42 +++++++++-- skills/ingest/SKILL.md | 38 ++++++---- skills/maintain/SKILL.md | 8 ++ skills/manifest.json | 2 +- skills/migrations/v0.8.1.md | 2 +- skills/migrations/v0.9.0.md | 133 ++++++++++++++++++++++++++++++++++ skills/setup/SKILL.md | 18 +++++ 8 files changed, 226 insertions(+), 23 deletions(-) create mode 100644 skills/migrations/v0.9.0.md diff --git a/recipes/twilio-voice-brain.md b/recipes/twilio-voice-brain.md index 91e855ce4..d845459dc 100644 --- a/recipes/twilio-voice-brain.md +++ b/recipes/twilio-voice-brain.md @@ -575,8 +575,10 @@ maintains it throughout the conversation. **Fix:** Auto-upload ALL call audio immediately on call end: - Twilio calls: download the MP3 recording URL from Twilio - WebRTC calls: capture via MediaRecorder (webm/opus format) -- Upload to cloud storage (Supabase Storage, S3, etc.) -- Store `.redirect.yaml` pointer in brain repo if >= 100 MB +- Upload via `gbrain files upload-raw --page meetings/call-slug --type call-recording` +- GBrain auto-routes: small files stay in git, large files go to cloud storage + with `.redirect.yaml` pointer. Files >= 100 MB use TUS resumable upload. +- Generate signed URLs for playback: `gbrain files signed-url ` - This ensures every call has a recoverable audio source regardless of whether the transcript or brain page was created successfully diff --git a/skills/_brain-filing-rules.md b/skills/_brain-filing-rules.md index b8fbf0e82..7f3fdab88 100644 --- a/skills/_brain-filing-rules.md +++ b/skills/_brain-filing-rules.md @@ -75,10 +75,40 @@ silently pick one. ## Raw Source Preservation -Every ingested item should have its raw source preserved for provenance: -- **< 100 MB** (text, PDFs, transcripts): store in the brain repo (git-tracked) - in a `.raw/` sidecar directory alongside the brain page -- **>= 100 MB** (video, audio, large datasets): upload to cloud storage and - store a `.redirect.yaml` pointer in the brain repo +Every ingested item should have its raw source preserved for provenance. -This ensures any derived brain page can be traced back to its original source. +**Size routing (automatic via `gbrain files upload-raw`):** +- **< 100 MB text/PDF**: stays in the brain repo (git-tracked) in a `.raw/` + sidecar directory alongside the brain page +- **>= 100 MB OR media files** (video, audio, images): uploaded to cloud + storage (Supabase Storage, S3, etc.) with a `.redirect.yaml` pointer left + in the brain repo. Files >= 100 MB use TUS resumable upload (6 MB chunks + with retry) for reliability. + +**Upload command:** +```bash +gbrain files upload-raw --page --type +``` +Returns JSON: `{storage: "git"}` for small files, `{storage: "supabase", storagePath, reference}` for cloud. + +**The `.redirect.yaml` pointer format:** +```yaml +target: supabase://brain-files/page-slug/filename.mp4 +bucket: brain-files +storage_path: page-slug/filename.mp4 +size: 524288000 +size_human: 500 MB +hash: sha256:abc123... +mime: video/mp4 +uploaded: 2026-04-11T... +type: transcript +``` + +**Accessing stored files:** +```bash +gbrain files signed-url # Generate 1-hour signed URL +gbrain files restore # Download back to local +``` + +This ensures any derived brain page can be traced back to its original source, +and large files don't bloat the git repo. diff --git a/skills/ingest/SKILL.md b/skills/ingest/SKILL.md index 7c56f8cab..090671134 100644 --- a/skills/ingest/SKILL.md +++ b/skills/ingest/SKILL.md @@ -186,21 +186,33 @@ if the post is primarily about a person/company. Every ingested item must have its raw source preserved for provenance. -- **< 100 MB** (text, PDFs, transcripts): store in brain repo `.raw/` sidecar - directories alongside the brain page -- **>= 100 MB** (video, audio, large datasets): upload to cloud storage (Supabase - Storage, S3, etc.) and store a `.redirect.yaml` pointer in the brain repo - -The `.redirect.yaml` format: -```yaml -storage: supabase -bucket: brain-files -path: media/videos/raw/video-id.mp4 -uploaded: 2026-04-11T... -size_bytes: 524288000 +**Use `gbrain files upload-raw` for automatic size routing:** +```bash +gbrain files upload-raw --page --type ``` -Use `put_raw_data` in gbrain to store raw API responses and metadata. +- **< 100 MB text/PDF**: stays in git (brain repo `.raw/` sidecar directories) +- **>= 100 MB OR media** (video, audio, images): uploaded to cloud storage + via TUS resumable upload, `.redirect.yaml` pointer left in the brain repo + +The `.redirect.yaml` pointer format: +```yaml +target: supabase://brain-files/page-slug/filename.mp4 +bucket: brain-files +storage_path: page-slug/filename.mp4 +size: 524288000 +size_human: 500 MB +hash: sha256:abc123... +mime: video/mp4 +uploaded: 2026-04-11T... +type: transcript +``` + +**Accessing stored files:** +- `gbrain files signed-url ` -- generate 1-hour signed URL for viewing/sharing +- `gbrain files restore ` -- download back to local from cloud storage + +Use `put_raw_data` in gbrain to store raw API responses and metadata (JSON, not binary). ## Test Before Bulk diff --git a/skills/maintain/SKILL.md b/skills/maintain/SKILL.md index 5156fd817..b184f465f 100644 --- a/skills/maintain/SKILL.md +++ b/skills/maintain/SKILL.md @@ -67,6 +67,14 @@ Check that the schema version is up to date. `gbrain doctor --json` reports the current version vs expected. If behind, `gbrain init` runs migrations automatically. +### File storage health +Check the integrity of stored files and redirect pointers: +- Run `gbrain files verify` to check all DB records have valid data +- Run `gbrain files status` to see migration state (local, mirrored, redirected) +- Check for orphan `.redirect.yaml` pointers that reference missing storage files +- Check for large binary files (>= 100 MB) still in git that should be in cloud storage +- If storage backend is configured: verify redirect pointers resolve (download test) + ### Open threads Timeline items older than 30 days with unresolved action items. - Flag for review diff --git a/skills/manifest.json b/skills/manifest.json index 483f45faf..6999904bf 100644 --- a/skills/manifest.json +++ b/skills/manifest.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.8.1", + "version": "0.9.0", "description": "Personal knowledge brain with hybrid RAG search", "skills": [ { diff --git a/skills/migrations/v0.8.1.md b/skills/migrations/v0.8.1.md index 9f3743337..713e3446f 100644 --- a/skills/migrations/v0.8.1.md +++ b/skills/migrations/v0.8.1.md @@ -2,7 +2,7 @@ version: 0.8.1 feature_pitch: headline: "Your brain skills learned from production" - description: "Back-linking iron law, filing rules, enrichment protocol, media ingest, citation requirements, voice crash fixes -- all battle-tested from real production deployments." + description: "Back-linking iron law, filing rules, enrichment protocol, media ingest, citation requirements, voice crash fixes -- all battle-tested from real production deployments. Superseded by v0.9.0." --- # v0.8.1 Migration: Battle-Tested Skill Patterns diff --git a/skills/migrations/v0.9.0.md b/skills/migrations/v0.9.0.md new file mode 100644 index 000000000..c65b2058a --- /dev/null +++ b/skills/migrations/v0.9.0.md @@ -0,0 +1,133 @@ +--- +version: 0.9.0 +feature_pitch: + headline: "Large files, smart uploads, production-grade skills" + description: "Files over 100 MB auto-upload to cloud storage via TUS resumable upload. .redirect.yaml pointers keep your git repo lean. All skills upgraded with battle-tested patterns from a production deployment." + tiers: + - name: "Core (no storage)" + description: "Upgraded skills only. Back-linking, filing rules, enrichment protocol, media ingest, citations." + setup: "gbrain check-update && gbrain upgrade" + - name: "With Supabase Storage" + description: "Full file management. Large files in cloud, .redirect.yaml pointers in git, signed URLs, TUS resumable upload." + setup: "Configure storage backend, then gbrain files mirror + redirect for existing binaries." +--- + +# v0.9.0 Migration: Smart File Storage + Production Skills + +This migration covers both the file storage infrastructure (code changes) and +the skill patterns (latent space changes). No database schema changes required. + +## What's New + +### File Storage Infrastructure (Code) + +**Smart upload routing:** +- `gbrain files upload-raw --page --type ` auto-routes by size +- < 100 MB text/PDF: stays in git (returns `{storage: "git"}`) +- >= 100 MB or media: uploads to cloud storage, creates `.redirect.yaml` pointer +- Files >= 100 MB use TUS resumable upload (6 MB chunks with retry/backoff) + +**New commands:** +- `gbrain files upload-raw` -- smart upload with size routing and pointer creation +- `gbrain files signed-url ` -- generate 1-hour signed URL for private files +- `gbrain files status` -- show migration state of directories + +**Upgraded redirect format:** +- Old: `.redirect` (5 fields: moved_to, bucket, path, moved_at, original_hash) +- New: `.redirect.yaml` (10 fields: target, bucket, storage_path, size, size_human, + hash, mime, uploaded, source_url, type) +- File resolver supports BOTH formats for backward compatibility + +**File migration lifecycle:** +1. `gbrain files mirror ` -- upload to cloud, keep local copies +2. `gbrain files redirect ` -- replace local with `.redirect.yaml` (verifies remote first) +3. `gbrain files restore ` -- download back from cloud +4. `gbrain files clean --yes` -- remove pointers (cloud is sole source) + +### Skill Patterns (Latent Space) + +All patterns from v0.8.1 are included plus: +- Skills reference actual `gbrain files` commands (not theoretical patterns) +- Maintain skill includes file storage health checks +- Setup skill includes storage configuration phase with migration guidance +- Voice recipe uses `gbrain files upload-raw` for call audio +- Filing rules document the `.redirect.yaml` format and upload commands + +## Agent Instructions + +### Step 1: Verify upgrade + +```bash +cat VERSION # should show 0.9.0 +gbrain doctor --json +``` + +### Step 2: Configure storage backend (optional but recommended) + +If your brain repo has binary files (images, PDFs, audio, video), configure +cloud storage to keep git lean: + +```bash +# Option A: Supabase Storage (recommended if you already use Supabase) +# Get your service role key from: Supabase Dashboard > Project Settings > API +gbrain config set storage.backend supabase +gbrain config set storage.bucket brain-files +gbrain config set storage.projectUrl https://YOUR-PROJECT.supabase.co +gbrain config set storage.serviceRoleKey YOUR_SERVICE_ROLE_KEY + +# Option B: S3-compatible (AWS, Cloudflare R2, MinIO) +gbrain config set storage.backend s3 +gbrain config set storage.bucket brain-files +gbrain config set storage.region us-east-1 +gbrain config set storage.accessKeyId YOUR_KEY +gbrain config set storage.secretAccessKey YOUR_SECRET +# For R2/MinIO: also set storage.endpoint +``` + +Verify: `gbrain files upload test.txt --page test && gbrain files list test` + +### Step 3: Migrate existing binary files (if storage configured) + +```bash +# See what you have +gbrain files status + +# If local binary files exist: +gbrain files mirror --dry-run # Preview what would upload +gbrain files mirror # Upload to cloud (keeps local) +gbrain files redirect --dry-run # Preview what would redirect +gbrain files redirect # Replace local with .redirect.yaml + +# Verify +gbrain files status +gbrain files signed-url # Test URL generation +``` + +### Step 4: Upgrade legacy .redirect files (if any exist) + +If you have `.redirect` files from v0.8.x, they continue to work (backward +compatible). To upgrade them to the richer `.redirect.yaml` format: + +```bash +# Find legacy redirects +find -name "*.redirect" -not -name "*.redirect.yaml" + +# For each: restore then re-redirect (picks up new format) +gbrain files restore +gbrain files mirror +gbrain files redirect +``` + +### Step 5: Review skills (same as v0.8.1) + +- Check 5 recent pages for back-link gaps +- Check sources/ for misfiled pages +- If voice recipe is set up: apply unicode/PII fixes +- If x-to-brain is set up: add image OCR, stagger crons + +### Step 6: Done + +```bash +mkdir -p ~/.gbrain/migrations +echo '{"version":"0.9.0","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","status":"complete","storage_configured":'$(gbrain config get storage.backend 2>/dev/null && echo true || echo false)'}' >> ~/.gbrain/migrations/completed.jsonl +``` diff --git a/skills/setup/SKILL.md b/skills/setup/SKILL.md index e4f579f76..d992a47f8 100644 --- a/skills/setup/SKILL.md +++ b/skills/setup/SKILL.md @@ -124,6 +124,24 @@ echo "=== Discovery Complete ===" > "You have N binary files (X GB) in your brain repo. Want to move them to cloud > storage? Your git repo will drop from X GB to Y MB. All links keep working." + If the user agrees, configure storage and run migration: + ```bash + # Configure storage backend (Supabase Storage recommended) + gbrain config set storage.backend supabase + gbrain config set storage.bucket brain-files + gbrain config set storage.projectUrl + gbrain config set storage.serviceRoleKey + + # Migrate binary files to cloud (3-step lifecycle) + gbrain files mirror # Upload to cloud, keep local + gbrain files redirect # Replace local with .redirect.yaml pointers + # (optional) gbrain files clean --yes # Remove pointers too + ``` + + After migration, `gbrain files upload-raw` handles new files automatically: + small text/PDFs stay in git, large/media files go to cloud with `.redirect.yaml` + pointers. Files >= 100 MB use TUS resumable upload for reliability. + If no markdown repos are found, create a starter brain with a few template pages (a person page, a company page, a concept page) from docs/GBRAIN_RECOMMENDED_SCHEMA.md.