gbrain/skills/migrations/v0.15.2.md

version, feature_pitch

version

feature_pitch

0.15.2

headline description recipe tiers Silent binaries are dead. Every bulk action now heartbeats. `gbrain doctor` on a 52K-page brain used to sit silent for 10+ minutes before an agent timeout killed it. Same pattern on embed, sync, import, extract, migrate, and every orchestrator. v0.15.2 routes 14 bulk commands through one shared reporter that writes to stderr. Non-TTY default is plain human lines; agents that want structured events add `--progress-json` and get one JSON object per line. Stdout stays clean for data output. Event schema is locked in docs/progress-events.md. docs/progress-events.md null

v0.15.2 Migration: Bulk-action progress streaming

Audience: host agents reading this after gbrain apply-migrations has run. v0.15.2 is purely additive to the CLI surface, there is no schema change, no data rewrite, and no orchestrator for this release. Your binaries just got observable. This file tells you how to use it.

Mechanical migration: nothing

There is no mechanical step. If gbrain upgrade completed, progress events are already flowing the next time you invoke a bulk command. Read on to know what's there and how to consume it.

What's new at the CLI

Three new global flags

These work on any gbrain subcommand:

• --progress-json — emit one JSON event per line on stderr.
• --quiet — suppress progress output entirely.
• --progress-interval=<ms> — minimum ms between progress emits (default 1000).

Parsed before command dispatch, so both work:

gbrain --progress-json doctor --json
gbrain doctor --json --progress-json

Per-TTY behavior

Without --progress-json:

• TTY: \r-rewriting single-line progress on stderr (fancy).
• Non-TTY (pipe, CI, agent): one plain-text line per event on stderr. No JSON, no noise. Human-readable.

The default was deliberately NOT JSON-on-non-TTY. Shell pipelines that just pipe gbrain ... | less should get readable logs, not a JSON blob. Agents opt in to JSON explicitly.

What's new per command

Fourteen commands now stream progress through the shared reporter:

Command	What you'll see
doctor	doctor.db_checks phase + per-check heartbeats, including a 1s heartbeat while markdown_body_completeness scans
orphans	orphans.scan heartbeat while the anti-join runs
embed	embed.pages with per-page ticks
files sync	files.sync with per-file ticks
export	export.pages with per-page ticks
import	import.files with per-file ticks (replaces per-100 stdout logs)
`extract [links	timeline
sync	sync.deletes, sync.renames, sync.imports phases
migrate --to ...	migrate.copy_pages, migrate.copy_links
repair-jsonb	repair_jsonb.run + per-column heartbeats
check-backlinks	backlinks.scan heartbeat
lint	lint.pages per-page ticks
integrity auto	integrity.auto per-page ticks
eval	eval.single / eval.ab per-query ticks
apply-migrations (v0_11/v0_12_0/v0_12_2)	Child processes inherit the parent's progress mode

JSON event schema

Documented in docs/progress-events.md (canonical reference). Stable from v0.15.2, additive changes only.

Quick agent cheat sheet:

{"event":"start","phase":"doctor.db_checks","ts":"..."}
{"event":"tick","phase":"orphans.scan","done":15000,"total":52000,"pct":28.8,"elapsed_ms":4200,"eta_ms":10300,"ts":"..."}
{"event":"heartbeat","phase":"doctor.markdown_body_completeness","note":"scanning pages for truncation...","elapsed_ms":1000,"ts":"..."}
{"event":"finish","phase":"doctor.db_checks","elapsed_ms":187000,"ts":"..."}
{"event":"abort","phase":"orphans.scan","reason":"SIGINT","elapsed_ms":5300,"ts":"..."}

Parser rules:

1. One JSON object per line on stderr.
2. Ignore unknown event types and unknown fields. Schema is additive.
3. Group by phase prefix to track one run: all doctor.* events belong to the same doctor invocation.
4. total / pct / eta_ms are absent when the scan doesn't have a total up front (e.g. heartbeat-only paths). Don't assume they exist.

Minion jobs

gbrain jobs work (the Minion worker daemon) writes progress to the DB via job.updateProgress, not to stderr. Read per-job progress via the get_job_progress MCP op or:

gbrain jobs submit embed
# while it runs:
gbrain jobs get <id>   # .progress updates live as the handler ticks

The embed Minion handler is wired as of v0.15.2. Other bulk cores (sync, extract, backlinks, import, autopilot-cycle) have the callback plumbing ready and will follow.

Backward-compatibility warnings

Five commands moved per-page progress from stdout to stderr:

• embed (was \r-on-stdout)
• files sync (was \r-on-stdout)
• export (was \r-on-stdout, newly in scope)
• migrate-engine (was per-50 console.log to stdout)
• import (was per-100 console.log to stdout)

If you have scripts that grep stdout for progress strings like Progress: 1234/52000 or \r 1234/52000 pages... — those strings now live on stderr. The final data summaries (Embedded N chunks across M pages, Import complete, etc.) remain on stdout so the "did it finish" signal is unchanged.

integrity auto still writes ~/.gbrain/integrity-progress.jsonl, but its role is now "resume marker only" — live progress goes through the reporter. If you depended on tailing that file for real-time progress, switch to the stderr stream.

Verification

# Your agent sees structured events; stdout stays JSON-parseable:
gbrain --progress-json doctor --json > doctor.json 2> doctor.progress.log
wc -l doctor.progress.log   # should be non-zero
jq . doctor.json             # should parse cleanly

# For a very large brain, watch the heartbeat:
gbrain --progress-json doctor 2>&1 >/dev/null | grep '"event"'

If you see silence for more than a second or two on a non-trivial command, file an issue with the exact command and the first 100 lines of stderr.

That's the whole migration

No mechanical step. No config change. Agents that parse stdout keep working; agents that want progress now have it on a clean stderr channel with a documented schema.