Files
gbrain/test/utils.test.ts
T
c0b621923b fix: JSONB double-encode + splitBody wiki + parseEmbedding (v0.12.1) (#196)
* fix: splitBody and inferType for wiki-style markdown content

- splitBody now requires explicit timeline sentinel (<!-- timeline -->,
  --- timeline ---, or --- directly before ## Timeline / ## History).
  A bare --- in body text is a markdown horizontal rule, not a separator.
  This fixes the 83% content truncation @knee5 reported on a 1,991-article
  wiki where 4,856 of 6,680 wikilinks were lost.

- serializeMarkdown emits <!-- timeline --> sentinel for round-trip stability.

- inferType extended with /writing/, /wiki/analysis/, /wiki/guides/,
  /wiki/hardware/, /wiki/architecture/, /wiki/concepts/. Path order is
  most-specific-first so projects/blog/writing/essay.md → writing,
  not project.

- PageType union extended: writing, analysis, guide, hardware, architecture.

Updates test/import-file.test.ts to use the new sentinel.

Co-Authored-By: @knee5 (PR #187)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: JSONB double-encode bug on Postgres + parseEmbedding NaN scores

Two related Postgres-string-typed-data bugs that PGLite hid:

1. JSONB double-encode (postgres-engine.ts:107,668,846 + files.ts:254):
   ${JSON.stringify(value)}::jsonb in postgres.js v3 stringified again
   on the wire, storing JSONB columns as quoted string literals. Every
   frontmatter->>'key' returned NULL on Postgres-backed brains; GIN
   indexes were inert. Switched to sql.json(value), which is the
   postgres.js-native JSONB encoder (Parameter with OID 3802).
   Affected columns: pages.frontmatter, raw_data.data,
   ingest_log.pages_updated, files.metadata. page_versions.frontmatter
   is downstream via INSERT...SELECT and propagates the fix.

2. pgvector embeddings returning as strings (utils.ts):
   getEmbeddingsByChunkIds returned "[0.1,0.2,...]" instead of
   Float32Array on Supabase, producing [NaN] cosine scores.
   Adds parseEmbedding() helper handling Float32Array, numeric arrays,
   and pgvector string format. Throws loud on malformed vectors
   (per Codex's no-silent-NaN requirement); returns null for
   non-vector strings (treated as "no embedding here"). rowToChunk
   delegates to parseEmbedding.

E2E regression test at test/e2e/postgres-jsonb.test.ts asserts
jsonb_typeof = 'object' AND col->>'k' returns expected scalar across
all 5 affected columns — the test that should have caught the original
bug. Runs in CI via the existing pgvector service.

Co-Authored-By: @knee5 (PR #187 — JSONB triple-fix)
Co-Authored-By: @leonardsellem (PR #175 — parseEmbedding)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: extract wikilink syntax with ancestor-search slug resolution

extractMarkdownLinks now handles [[page]] and [[page|Display Text]]
alongside standard [text](page.md). For wiki KBs where authors omit
leading ../ (thinking in wiki-root-relative terms), resolveSlug
walks ancestor directories until it finds a matching slug.

Without this, wikilinks under tech/wiki/analysis/ targeting
[[../../finance/wiki/concepts/foo]] silently dangled when the
correct relative depth was 3 × ../ instead of 2.

Co-Authored-By: @knee5 (PR #187)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: gbrain repair-jsonb + v0.12.1 migration + CI grep guard

- New gbrain repair-jsonb command. Detects rows where
  jsonb_typeof(col) = 'string' and rewrites them via
  (col #>> '{}')::jsonb across 5 affected columns:
  pages.frontmatter, raw_data.data, ingest_log.pages_updated,
  files.metadata, page_versions.frontmatter. Idempotent — re-running
  is a no-op. PGLite engines short-circuit cleanly (the bug never
  affected the parameterized encode path PGLite uses). --dry-run
  shows what would be repaired; --json for scripting.

- New v0_12_1.ts migration orchestrator. Phases: schema → repair → verify.
  Modeled on v0_12_0 pattern, registered in migrations/index.ts.
  Runs automatically via gbrain upgrade / apply-migrations.

- CI grep guard at scripts/check-jsonb-pattern.sh fails the build if
  anyone reintroduces the ${JSON.stringify(x)}::jsonb interpolation
  pattern. Wired into bun test via package.json. Best-effort static
  analysis (multi-line and helper-wrapped variants are caught by the
  E2E round-trip test instead).

- Updates apply-migrations.test.ts expectations to account for the new
  v0.12.1 entry in the registry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.12.1)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update project documentation for v0.12.1

- CLAUDE.md: document repair-jsonb command, v0_12_1 migration,
  splitBody sentinel contract, inferType wiki subtypes, CI grep
  guard, new test files (repair-jsonb, migrations-v0_12_1, markdown)
- README.md: add gbrain repair-jsonb to ADMIN command reference
- INSTALL_FOR_AGENTS.md: fix verification count (6 -> 7), add
  v0.12.1 upgrade guidance for Postgres brains
- docs/GBRAIN_VERIFY.md: add check #8 for JSONB integrity on
  Postgres-backed brains
- docs/UPGRADING_DOWNSTREAM_AGENTS.md: add v0.12.1 section with
  migration steps, splitBody contract, wiki subtype inference
- skills/migrate/SKILL.md: document native wikilink extraction
  via gbrain extract links (v0.12.1+)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 07:14:24 +08:00

160 lines
5.7 KiB
TypeScript

import { describe, test, expect } from 'bun:test';
import { validateSlug, contentHash, parseEmbedding, rowToPage, rowToChunk, rowToSearchResult } from '../src/core/utils.ts';
describe('validateSlug', () => {
test('accepts valid slugs', () => {
expect(validateSlug('people/sarah-chen')).toBe('people/sarah-chen');
expect(validateSlug('concepts/rag')).toBe('concepts/rag');
expect(validateSlug('simple')).toBe('simple');
});
test('normalizes to lowercase', () => {
expect(validateSlug('People/Sarah-Chen')).toBe('people/sarah-chen');
expect(validateSlug('UPPER')).toBe('upper');
});
test('rejects empty slug', () => {
expect(() => validateSlug('')).toThrow('Invalid slug');
});
test('rejects path traversal', () => {
expect(() => validateSlug('../etc/passwd')).toThrow('path traversal');
expect(() => validateSlug('test/../hack')).toThrow('path traversal');
});
test('rejects leading slash', () => {
expect(() => validateSlug('/absolute/path')).toThrow('start with /');
});
});
describe('contentHash', () => {
test('returns deterministic hash', () => {
const page = { title: 'Test', type: 'concept' as const, compiled_truth: 'hello', timeline: 'world' };
const h1 = contentHash(page);
const h2 = contentHash(page);
expect(h1).toBe(h2);
});
test('changes when content changes', () => {
const h1 = contentHash({ title: 'Test', type: 'concept' as const, compiled_truth: 'hello', timeline: 'world' });
const h2 = contentHash({ title: 'Test', type: 'concept' as const, compiled_truth: 'hello', timeline: 'changed' });
expect(h1).not.toBe(h2);
});
test('returns hex string', () => {
const h = contentHash({ title: 'Test', type: 'concept' as const, compiled_truth: 'test', timeline: '' });
expect(h).toMatch(/^[a-f0-9]{64}$/);
});
});
describe('rowToPage', () => {
test('parses string frontmatter', () => {
const page = rowToPage({
id: 1, slug: 'test', type: 'concept', title: 'Test',
compiled_truth: 'body', timeline: '',
frontmatter: '{"key":"val"}',
content_hash: 'abc', created_at: '2024-01-01', updated_at: '2024-01-01',
});
expect(page.frontmatter.key).toBe('val');
});
test('handles object frontmatter', () => {
const page = rowToPage({
id: 1, slug: 'test', type: 'concept', title: 'Test',
compiled_truth: 'body', timeline: '',
frontmatter: { key: 'val' },
content_hash: 'abc', created_at: '2024-01-01', updated_at: '2024-01-01',
});
expect(page.frontmatter.key).toBe('val');
});
test('creates Date objects', () => {
const page = rowToPage({
id: 1, slug: 'test', type: 'concept', title: 'Test',
compiled_truth: '', timeline: '', frontmatter: '{}',
content_hash: null, created_at: '2024-01-01T00:00:00Z', updated_at: '2024-01-01T00:00:00Z',
});
expect(page.created_at).toBeInstanceOf(Date);
expect(page.updated_at).toBeInstanceOf(Date);
});
});
describe('rowToChunk', () => {
test('nulls embedding by default', () => {
const chunk = rowToChunk({
id: 1, page_id: 1, chunk_index: 0, chunk_text: 'text',
chunk_source: 'compiled_truth', embedding: new Float32Array(10),
model: 'test', token_count: 5, embedded_at: '2024-01-01',
});
expect(chunk.embedding).toBeNull();
});
test('includes embedding when requested', () => {
const emb = new Float32Array(10).fill(0.5);
const chunk = rowToChunk({
id: 1, page_id: 1, chunk_index: 0, chunk_text: 'text',
chunk_source: 'compiled_truth', embedding: emb,
model: 'test', token_count: 5, embedded_at: '2024-01-01',
}, true);
expect(chunk.embedding).not.toBeNull();
});
test('parses pgvector string embeddings when requested', () => {
const chunk = rowToChunk({
id: 1, page_id: 1, chunk_index: 0, chunk_text: 'text',
chunk_source: 'compiled_truth', embedding: '[0.1, 0.2, 0.3]',
model: 'test', token_count: 5, embedded_at: '2024-01-01',
}, true);
expect(chunk.embedding).toBeInstanceOf(Float32Array);
expect(Array.from(chunk.embedding || [])).toHaveLength(3);
expect(chunk.embedding?.[0]).toBeCloseTo(0.1, 6);
expect(chunk.embedding?.[1]).toBeCloseTo(0.2, 6);
expect(chunk.embedding?.[2]).toBeCloseTo(0.3, 6);
});
});
describe('parseEmbedding', () => {
test('returns Float32Array unchanged', () => {
const emb = new Float32Array([0.1, 0.2]);
expect(parseEmbedding(emb)).toBe(emb);
});
test('parses pgvector text into Float32Array', () => {
const parsed = parseEmbedding('[0.1, 0.2, 0.3]');
expect(parsed).toBeInstanceOf(Float32Array);
expect(Array.from(parsed || [])).toHaveLength(3);
expect(parsed?.[0]).toBeCloseTo(0.1, 6);
expect(parsed?.[1]).toBeCloseTo(0.2, 6);
expect(parsed?.[2]).toBeCloseTo(0.3, 6);
});
test('returns null for unsupported embedding values', () => {
expect(parseEmbedding(null)).toBeNull();
expect(parseEmbedding(undefined)).toBeNull();
expect(parseEmbedding('not-a-vector')).toBeNull();
});
test('parses numeric array into Float32Array', () => {
const parsed = parseEmbedding([0.5, 0.25, 0.125]);
expect(parsed).toBeInstanceOf(Float32Array);
expect(parsed?.[0]).toBeCloseTo(0.5, 6);
});
test('throws on vector-like string with non-numeric content (no silent NaN)', () => {
expect(() => parseEmbedding('[abc, def]')).toThrow();
expect(() => parseEmbedding('[1, NaN, 3]')).toThrow();
});
});
describe('rowToSearchResult', () => {
test('coerces score to number', () => {
const r = rowToSearchResult({
slug: 'test', page_id: 1, title: 'Test', type: 'concept',
chunk_text: 'text', chunk_source: 'compiled_truth',
score: '0.95', stale: false,
});
expect(typeof r.score).toBe('number');
expect(r.score).toBe(0.95);
});
});