admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 22:15:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
84fed4194a0596dce05052934079c656fd305d16
gbrain/skills/migrations/v0.41.11.0.md
T
Garry TanGitHubgarrytan-agentsClaude Opus 4.7
 84fed4194a v0.41.11.0 feat: conversation retrieval upgrade — production-bar replacement for PR #1406 (#1446) 
* v0.41.11.0 feat: conversation retrieval upgrade — production-bar replacement for PR #1406

Long chat threads stop swallowing your search results. The recall miss
class on long iMessage/Slack imports (60K+ msg history; a chunk that
reads only "Locker 93 code 9494" has no topical anchor because "cabin"
was established 50K messages earlier) gets fixed by walking
conversation/meeting/slack/email pages, splitting into time-windowed
segments (30-min gap or 30-msg cap), prepending a topical/temporal
header, and running through the existing extractFactsFromTurn() so
the resulting anchor-rich facts surface in gbrain search.

This is the production-bar replacement for PR #1406 (which closes
LAST per Codex T6d, AFTER this PR is green). The bug fix survives 1:1;
the wrapping closes 14 load-bearing issues the original PR deferred
or shipped silent bugs around. The wave went through CEO scope review,
3 rounds of spec review, 2 rounds of Codex outside voice grounding
the plan against actual code, and 2 passes of eng review.

Version-slot note: originally planned as v0.41.2.0; master shipped its
own v0.41.2.0 (lens packs) plus v0.41.3-6.0 between plan-time and
ship-time. Re-bumped to v0.41.11.0 (next free slot; v0.41.7-10 claimed
by other open PRs).

Key files (new):
- src/commands/extract-conversation-facts.ts — CLI command with
  --types, --max-cost-usd, --background, --override-disabled,
  --slug, --dry-run, --limit, --since, --force, --sleep,
  --segment-limit, --source-id. Strict per-source core; two-phase
  page enumeration (paginated listPages with 10×25MB cap = 250MB
  worst case); 25MB body cap; page-global row_num accumulator
  (Codex C1 unique-index collision fix); page-level TERMINAL audit
  row after all segments commit (Codex C7 durable extraction marker);
  optional opts.budgetTracker (Codex C5 — nested withBudgetTracker
  REPLACES, so caller-managed scope passes tracker through); reads
  compiled_truth + timeline (F1 — PR silently dropped timeline half);
  honors facts.extraction_enabled kill-switch with --override-disabled
  escape (F2); --types reads cycle config as single source of truth
  (Eng-v2 A2); fingerprint on sourceId only (Eng-v2 A3 — widening
  types doesn't invalidate completion); string-encoded op-checkpoint
  entries "sourceId|slug|endIso" for resume; segment caps tuned
  6500/30 (Eng-v2 T5) to stay under extract.ts MAX_TURN_TEXT_CHARS=8000.
- src/core/cycle/conversation-facts-backfill.ts — cycle phase wrapper
  (default OFF). Iterates listSources() directly; creates ONE
  brain-wide BudgetTracker per tick + wraps the loop in
  withBudgetTracker + passes tracker through opts.budgetTracker so
  core doesn't nest-replace. Two-layer cost AND walltime protection:
  per-source caps ($1, 20min) AND brain-wide caps ($5, 30min).
- test/extract-conversation-facts.test.ts — 27 unit cases (parse,
  segment, render, checkpoint encoding, fingerprint, terminal audit
  row, row_num accumulator, F2 kill-switch, --override-disabled).
- skills/migrations/v0.41.11.0.md — agent-facing migration guide.

Key files (modified):
- src/commands/jobs.ts — register extract-conversation-facts Minion
  handler. NOT in PROTECTED_JOB_NAMES; BudgetExhausted catch + persist
  + mark completed with result.budget_exhausted (NOT a failure).
- src/commands/doctor.ts — computeConversationFactsBacklogCheck (3-state:
  SKIPPED when feature disabled per Eng-v2 C9, OK at backlog=0, WARN
  at >10 with paste-ready remediation step via makeRemediationStep).
  Doctor query is source-scoped (Codex C2 cross-source safety) and
  matches the TERMINAL audit row (Codex C7), not any-fact-for-slug.
- src/commands/sources.ts — runAudit extended with
  facts_backfill_estimate field for cost preview.
- src/cli.ts — CLI_ONLY + CLI_ONLY_SELF_HELP + THIN_CLIENT_REFUSED_COMMANDS
  + dispatch case for extract-conversation-facts.
- src/core/cycle.ts — new CyclePhase 'conversation_facts_backfill';
  PHASE_SCOPE='source' (taxonomy only per cycle.ts:131 — wrapper does
  own multi-source iteration); wired into ALL_PHASES + NEEDS_LOCK_PHASES;
  dispatch block runs between consolidate and embed.
- src/core/migrate.ts — migration v94 adds partial index
  idx_facts_extract_conversation_session ON facts(source_id, source_session)
  WHERE source LIKE 'cli:extract-conversation-facts%' so doctor query
  stays fast on million-fact brains. v14 precedent: transaction:false +
  invalid-index pre-drop on Postgres, plain CREATE INDEX on PGLite.
- src/core/schema-pack/base/gbrain-base.yaml — promote conversation
  (temporal, extractable) and atom (annotation, NOT extractable —
  atoms ARE the extracted form) into base. Flip concept.extractable:
  true semantically (cosmetic on backstop path per Codex T3; the
  original grandfather migration was solving a phantom, dropped).
  Filing rules added for both new types.
- src/core/schema-pack/base/gbrain-recommended.yaml — remove duplicate
  conversation (now inherits via extends: gbrain-base).
- src/core/types.ts — ALL_PAGE_TYPES extended with conversation, atom.
- test/extractable-pack.test.ts — updated parity gate (24 page types
  vs PR's 22; concept + conversation now extractable, atom not).
- test/schema-cli.test.ts — page-count expectation 22→24.
- VERSION + package.json bumped to 0.41.11.0.
- CHANGELOG.md release-summary in the required ELI10-first voice +
  itemized changes section.
- CLAUDE.md Key Files entry for the new modules + architecture notes.
- llms.txt + llms-full.txt regenerated.

Plan + decisions persisted at:
  ~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md
CEO plan at:
  ~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md

Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: align v0.41.11.0 phase ordering + bump hardcoded counts after master merge

Three CI failures from the master merge in this branch:

1. test/phase-scope-coverage.test.ts pinned `ALL_PHASES.length === 19`
   and `Object.keys(PHASE_SCOPE).length === 19`. After merging master's
   v0.41 lens-packs (extract_atoms + synthesize_concepts) + my new
   conversation_facts_backfill phase, the total is 20.

2. test/core/cycle.serial.test.ts had two hardcoded `19` assertions
   (`hookCalls` and `report.phases.length`) tracking the same count.
   Both bumped to 20.

3. cycle.serial's `'default: all 6 phases run in order'` test asserts
   `report.phases.map(p => p.phase) === ALL_PHASES`. My initial commit
   put `conversation_facts_backfill` in ALL_PHASES between consolidate
   and propose_takes, but the runCycle dispatch block runs it AFTER
   the calibration trio (propose_takes / grade_takes / calibration_profile)
   and BEFORE embed. List and dispatch order didn't match, so the
   equality assertion failed.

   Resolution: moved 'conversation_facts_backfill' in ALL_PHASES to
   AFTER 'calibration_profile' so list-order matches dispatch-order.
   The dispatch block placement was correct (and remains correct);
   the list-position comment originally said "AFTER consolidate" but
   the dispatch runs it after the WHOLE consolidate→calibration_profile
   block, not just after consolidate. Comment now reflects reality.

Verified: 61/61 pass across the 3 affected test files (2.9s wallclock).

The CI logs also showed a "(unnamed) [3058.07ms]" failure in shard 1;
unable to reproduce locally (test/scripts/run-unit-parallel.test.ts
passes 6/6 in 1s). Suspected CI-load flakiness under bun's parallel
scheduler. If it persists on the next CI run, will dig in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): orphan sleep cleanup in run-unit-parallel.sh heartbeat

Two CI runs in a row reported `(fail) (unnamed) [~2400ms]` in shard 1
on this PR. Investigation:

- CI's end-of-job cleanup logged: "Terminate orphan process: pid (3344)
  (sleep)" × 6 sleep processes.
- The 6 matches exactly the 6 `runWrapper()` calls in
  test/scripts/run-unit-parallel.test.ts (1 orphan sleep per invocation).
- Each `runWrapper()` spawns scripts/run-unit-parallel.sh, which spawns
  a heartbeat function that runs `while true; do sleep 10; ...; done`
  in the background.
- The wrapper's EXIT trap was `kill "$HB_PID" 2>/dev/null` — kills the
  heartbeat shell, but its currently-running `sleep 10` child gets
  reparented to init/launchd because SIGTERM to a bash shell sleeping
  inside `sleep` doesn't propagate to the sleep child before wait
  returns. Known bash quirk on Linux.
- bun's test runner treats the orphan sleeps as a `(unnamed)` failure
  attributed to the test file that spawned the wrapper.

Fix: pkill children FIRST, then kill heartbeat. If we kill heartbeat
first, its child sleep orphans and pkill -P can no longer find it
(ppid changes to 1). Reorder applied to both the trap AND the normal
shutdown path.

Verified locally: before fix, 6 orphan sleeps after the test ran;
after fix, 0 orphan sleeps. Test still passes 6/6 in ~1s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 15:21:59 -07:00

3.5 KiB
Raw Blame History

v0.41.11.0 migration — conversation retrieval upgrade

The v0.41.11.0 upgrade ships:

  1. Schema-pack additions: conversation (temporal, extractable) and atom (annotation, NOT extractable) promoted into gbrain-base.
  2. New CLI command gbrain extract-conversation-facts for bulk fact extraction over long-form conversation pages.
  3. Opt-in cycle phase conversation_facts_backfill (default OFF) for autopilot-driven backlog drain.
  4. New doctor check conversation_facts_backlog (3-state; SKIPPED when feature disabled so no opt-out noise).
  5. Schema migration v94: partial index on facts(source_id, source_session) WHERE source LIKE 'cli:extract-conversation-facts%' so the doctor query stays fast on big brains.

gbrain upgrade + gbrain apply-migrations --yes handles the binary + schema. The cycle phase and doctor check require no agent action.

Agent action — when to run the backfill

When the user mentions:

  • "search isn't finding things in my chat history"
  • "my iMessage / Slack import isn't searchable"
  • "the page exists but search misses the answer"
  • "long conversation pages aren't returning what I want"

…then this command is the right tool. Don't run it speculatively — it costs real money against the user's Anthropic budget.

Recommended flow

  1. Estimate cost first. For each source the user wants to backfill:

    gbrain sources audit <source-id> --json | jq '.facts_backfill_estimate'

    Returns {pages, est_segments, est_cost_usd, types}. Show the user the cost estimate before recommending the run.

  2. Run a dry-run to preview segmentation:

    gbrain extract-conversation-facts --dry-run --slug <slug-of-one-conversation-page>

    Confirms the segmenter parses the page correctly without spending any tokens.

  3. Run with explicit cost cap. Default cap is $5. Recommend matching the audit estimate (rounded up):

    gbrain extract-conversation-facts --background --max-cost-usd <cap>
gbrain jobs follow <printed-job-id>

    Use --background so the user can keep working. The Minion job is resumable — if it hits the budget cap mid-run, re-running with a higher cap continues from where it left off.

  4. Verify with doctor:

    gbrain doctor --json | jq '.checks[] | select(.name == "conversation_facts_backlog")'

    Should show OK with backlog: 0 after a successful complete run.

  5. (Optional) Enable autopilot drain if the user has steady conversation ingest:

    gbrain config set cycle.conversation_facts_backfill.enabled true

    The cycle phase will then drain new conversation pages each tick under bounded per-source ($1/cycle) and brain-wide ($5/cycle) budgets.

Caveats

  • Pages over 25MB body are skipped (memory cap). Surface in doctor details; streaming for huge pages is a v0.42+ TODO.
  • If facts.extraction_enabled is false, the command refuses. Pass --override-disabled only when the user explicitly opted out and now wants this one-time run.
  • Extracted facts use source = 'cli:extract-conversation-facts'. To remove them in bulk (rare), the only path today is raw SQL: DELETE FROM facts WHERE source LIKE 'cli:extract-conversation-facts%'. A gbrain forget --where bulk flag is a v0.42+ TODO.
  • The recall-quality eval (under test/eval/conversation-extraction-quality.eval.ts — added in this wave) is env-gated on ANTHROPIC_API_KEY. Run nightly or on-demand for quality verification; the hermetic wiring tests run in CI by default.
Reference in New Issue View Git Blame Copy Permalink