admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: skills reference actual gbrain file commands

Browse Source 
- Filing rules document upload-raw, signed-url, and .redirect.yaml format
- Ingest skill uses gbrain files upload-raw for raw source preservation
- Maintain skill adds file storage health checks
- Setup skill adds storage configuration phase with migration guidance
- Voice recipe uses upload-raw for call audio storage
- Migration v0.9.0 with complete storage setup instructions
This commit is contained in:
Garry Tan
2026-04-11 17:44:10 -10:00
parent c2a14c9a44
commit b7f3dc9709
8 changed files with 226 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
recipes/twilio-voice-brain.md
+4 -2
View File
@@ -575,8 +575,10 @@ maintains it throughout the conversation.
**Fix:** Auto-upload ALL call audio immediately on call end:
- Twilio calls: download the MP3 recording URL from Twilio
- WebRTC calls: capture via MediaRecorder (webm/opus format)
- Upload to cloud storage (Supabase Storage, S3, etc.)
- Store `.redirect.yaml` pointer in brain repo if >= 100 MB
- Upload via `gbrain files upload-raw <audio-file> --page meetings/call-slug --type call-recording`
- GBrain auto-routes: small files stay in git, large files go to cloud storage
with `.redirect.yaml` pointer. Files >= 100 MB use TUS resumable upload.
- Generate signed URLs for playback: `gbrain files signed-url <storage-path>`
- This ensures every call has a recoverable audio source regardless
of whether the transcript or brain page was created successfully
skills/_brain-filing-rules.md
+36 -6
View File
@@ -75,10 +75,40 @@ silently pick one.
## Raw Source Preservation
Every ingested item should have its raw source preserved for provenance:
- **< 100 MB** (text, PDFs, transcripts): store in the brain repo (git-tracked)
in a `.raw/` sidecar directory alongside the brain page
- **>= 100 MB** (video, audio, large datasets): upload to cloud storage and
store a `.redirect.yaml` pointer in the brain repo
Every ingested item should have its raw source preserved for provenance.
This ensures any derived brain page can be traced back to its original source.
**Size routing (automatic via `gbrain files upload-raw`):**
- **< 100 MB text/PDF**: stays in the brain repo (git-tracked) in a `.raw/`
sidecar directory alongside the brain page
- **>= 100 MB OR media files** (video, audio, images): uploaded to cloud
storage (Supabase Storage, S3, etc.) with a `.redirect.yaml` pointer left
in the brain repo. Files >= 100 MB use TUS resumable upload (6 MB chunks
with retry) for reliability.
**Upload command:**
```bash
gbrain files upload-raw <file> --page <page-slug> --type <type>
```
Returns JSON: `{storage: "git"}` for small files, `{storage: "supabase", storagePath, reference}` for cloud.
**The `.redirect.yaml` pointer format:**
```yaml
target: supabase://brain-files/page-slug/filename.mp4
bucket: brain-files
storage_path: page-slug/filename.mp4
size: 524288000
size_human: 500 MB
hash: sha256:abc123...
mime: video/mp4
uploaded: 2026-04-11T...
type: transcript
```
**Accessing stored files:**
```bash
gbrain files signed-url <storage-path> # Generate 1-hour signed URL
gbrain files restore <dir> # Download back to local
```
This ensures any derived brain page can be traced back to its original source,
and large files don't bloat the git repo.
skills/ingest/SKILL.md
+25 -13
View File
@@ -186,21 +186,33 @@ if the post is primarily about a person/company.
Every ingested item must have its raw source preserved for provenance.
- **< 100 MB** (text, PDFs, transcripts): store in brain repo `.raw/` sidecar
directories alongside the brain page
- **>= 100 MB** (video, audio, large datasets): upload to cloud storage (Supabase
Storage, S3, etc.) and store a `.redirect.yaml` pointer in the brain repo
The `.redirect.yaml` format:
```yaml
storage: supabase
bucket: brain-files
path: media/videos/raw/video-id.mp4
uploaded: 2026-04-11T...
size_bytes: 524288000
**Use `gbrain files upload-raw` for automatic size routing:**
```bash
gbrain files upload-raw <file> --page <page-slug> --type <type>
```
Use `put_raw_data` in gbrain to store raw API responses and metadata.
- **< 100 MB text/PDF**: stays in git (brain repo `.raw/` sidecar directories)
- **>= 100 MB OR media** (video, audio, images): uploaded to cloud storage
via TUS resumable upload, `.redirect.yaml` pointer left in the brain repo
The `.redirect.yaml` pointer format:
```yaml
target: supabase://brain-files/page-slug/filename.mp4
bucket: brain-files
storage_path: page-slug/filename.mp4
size: 524288000
size_human: 500 MB
hash: sha256:abc123...
mime: video/mp4
uploaded: 2026-04-11T...
type: transcript
```
**Accessing stored files:**
- `gbrain files signed-url <storage-path>` -- generate 1-hour signed URL for viewing/sharing
- `gbrain files restore <dir>` -- download back to local from cloud storage
Use `put_raw_data` in gbrain to store raw API responses and metadata (JSON, not binary).
## Test Before Bulk
skills/maintain/SKILL.md
+8
View File
@@ -67,6 +67,14 @@ Check that the schema version is up to date. `gbrain doctor --json` reports
the current version vs expected. If behind, `gbrain init` runs migrations
automatically.
### File storage health
Check the integrity of stored files and redirect pointers:
- Run `gbrain files verify` to check all DB records have valid data
- Run `gbrain files status` to see migration state (local, mirrored, redirected)
- Check for orphan `.redirect.yaml` pointers that reference missing storage files
- Check for large binary files (>= 100 MB) still in git that should be in cloud storage
- If storage backend is configured: verify redirect pointers resolve (download test)
### Open threads
Timeline items older than 30 days with unresolved action items.
- Flag for review
skills/manifest.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.8.1",
"version": "0.9.0",
"description": "Personal knowledge brain with hybrid RAG search",
"skills": [
{
skills/migrations/v0.8.1.md
+1 -1
View File
@@ -2,7 +2,7 @@
version: 0.8.1
feature_pitch:
headline: "Your brain skills learned from production"
description: "Back-linking iron law, filing rules, enrichment protocol, media ingest, citation requirements, voice crash fixes -- all battle-tested from real production deployments."
description: "Back-linking iron law, filing rules, enrichment protocol, media ingest, citation requirements, voice crash fixes -- all battle-tested from real production deployments. Superseded by v0.9.0."
---
# v0.8.1 Migration: Battle-Tested Skill Patterns
skills/migrations/v0.9.0.md
+133
View File
@@ -0,0 +1,133 @@
---
version: 0.9.0
feature_pitch:
headline: "Large files, smart uploads, production-grade skills"
description: "Files over 100 MB auto-upload to cloud storage via TUS resumable upload. .redirect.yaml pointers keep your git repo lean. All skills upgraded with battle-tested patterns from a production deployment."
tiers:
- name: "Core (no storage)"
description: "Upgraded skills only. Back-linking, filing rules, enrichment protocol, media ingest, citations."
setup: "gbrain check-update && gbrain upgrade"
- name: "With Supabase Storage"
description: "Full file management. Large files in cloud, .redirect.yaml pointers in git, signed URLs, TUS resumable upload."
setup: "Configure storage backend, then gbrain files mirror + redirect for existing binaries."
---
# v0.9.0 Migration: Smart File Storage + Production Skills
This migration covers both the file storage infrastructure (code changes) and
the skill patterns (latent space changes). No database schema changes required.
## What's New
### File Storage Infrastructure (Code)
**Smart upload routing:**
- `gbrain files upload-raw <file> --page <slug> --type <type>` auto-routes by size
- < 100 MB text/PDF: stays in git (returns `{storage: "git"}`)
- >= 100 MB or media: uploads to cloud storage, creates `.redirect.yaml` pointer
- Files >= 100 MB use TUS resumable upload (6 MB chunks with retry/backoff)
**New commands:**
- `gbrain files upload-raw` -- smart upload with size routing and pointer creation
- `gbrain files signed-url <path>` -- generate 1-hour signed URL for private files
- `gbrain files status` -- show migration state of directories
**Upgraded redirect format:**
- Old: `.redirect` (5 fields: moved_to, bucket, path, moved_at, original_hash)
- New: `.redirect.yaml` (10 fields: target, bucket, storage_path, size, size_human,
hash, mime, uploaded, source_url, type)
- File resolver supports BOTH formats for backward compatibility
**File migration lifecycle:**
1. `gbrain files mirror <dir>` -- upload to cloud, keep local copies
2. `gbrain files redirect <dir>` -- replace local with `.redirect.yaml` (verifies remote first)
3. `gbrain files restore <dir>` -- download back from cloud
4. `gbrain files clean <dir> --yes` -- remove pointers (cloud is sole source)
### Skill Patterns (Latent Space)
All patterns from v0.8.1 are included plus:
- Skills reference actual `gbrain files` commands (not theoretical patterns)
- Maintain skill includes file storage health checks
- Setup skill includes storage configuration phase with migration guidance
- Voice recipe uses `gbrain files upload-raw` for call audio
- Filing rules document the `.redirect.yaml` format and upload commands
## Agent Instructions
### Step 1: Verify upgrade
```bash
cat VERSION # should show 0.9.0
gbrain doctor --json
```
### Step 2: Configure storage backend (optional but recommended)
If your brain repo has binary files (images, PDFs, audio, video), configure
cloud storage to keep git lean:
```bash
# Option A: Supabase Storage (recommended if you already use Supabase)
# Get your service role key from: Supabase Dashboard > Project Settings > API
gbrain config set storage.backend supabase
gbrain config set storage.bucket brain-files
gbrain config set storage.projectUrl https://YOUR-PROJECT.supabase.co
gbrain config set storage.serviceRoleKey YOUR_SERVICE_ROLE_KEY
# Option B: S3-compatible (AWS, Cloudflare R2, MinIO)
gbrain config set storage.backend s3
gbrain config set storage.bucket brain-files
gbrain config set storage.region us-east-1
gbrain config set storage.accessKeyId YOUR_KEY
gbrain config set storage.secretAccessKey YOUR_SECRET
# For R2/MinIO: also set storage.endpoint
```
Verify: `gbrain files upload test.txt --page test && gbrain files list test`
### Step 3: Migrate existing binary files (if storage configured)
```bash
# See what you have
gbrain files status
# If local binary files exist:
gbrain files mirror <brain-dir> --dry-run # Preview what would upload
gbrain files mirror <brain-dir> # Upload to cloud (keeps local)
gbrain files redirect <brain-dir> --dry-run # Preview what would redirect
gbrain files redirect <brain-dir> # Replace local with .redirect.yaml
# Verify
gbrain files status
gbrain files signed-url <any-storage-path> # Test URL generation
```
### Step 4: Upgrade legacy .redirect files (if any exist)
If you have `.redirect` files from v0.8.x, they continue to work (backward
compatible). To upgrade them to the richer `.redirect.yaml` format:
```bash
# Find legacy redirects
find <brain-dir> -name "*.redirect" -not -name "*.redirect.yaml"
# For each: restore then re-redirect (picks up new format)
gbrain files restore <dir>
gbrain files mirror <dir>
gbrain files redirect <dir>
```
### Step 5: Review skills (same as v0.8.1)
- Check 5 recent pages for back-link gaps
- Check sources/ for misfiled pages
- If voice recipe is set up: apply unicode/PII fixes
- If x-to-brain is set up: add image OCR, stagger crons
### Step 6: Done
```bash
mkdir -p ~/.gbrain/migrations
echo '{"version":"0.9.0","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","status":"complete","storage_configured":'$(gbrain config get storage.backend 2>/dev/null && echo true || echo false)'}' >> ~/.gbrain/migrations/completed.jsonl
```
skills/setup/SKILL.md
+18
View File
@@ -124,6 +124,24 @@ echo "=== Discovery Complete ==="
> "You have N binary files (X GB) in your brain repo. Want to move them to cloud
> storage? Your git repo will drop from X GB to Y MB. All links keep working."
If the user agrees, configure storage and run migration:
```bash
# Configure storage backend (Supabase Storage recommended)
gbrain config set storage.backend supabase
gbrain config set storage.bucket brain-files
gbrain config set storage.projectUrl <supabase-url>
gbrain config set storage.serviceRoleKey <service-role-key>
# Migrate binary files to cloud (3-step lifecycle)
gbrain files mirror <brain-dir> # Upload to cloud, keep local
gbrain files redirect <brain-dir> # Replace local with .redirect.yaml pointers
# (optional) gbrain files clean <brain-dir> --yes # Remove pointers too
```
After migration, `gbrain files upload-raw` handles new files automatically:
small text/PDFs stay in git, large/media files go to cloud with `.redirect.yaml`
pointers. Files >= 100 MB use TUS resumable upload for reliability.
If no markdown repos are found, create a starter brain with a few template pages
(a person page, a company page, a concept page) from docs/GBRAIN_RECOMMENDED_SCHEMA.md.