mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 11:22:34 +00:00
* feat: search quality boost — compiled truth ranking, detail parameter, cosine re-scoring Compiled truth chunks now rank 2x higher in hybrid search via RRF normalization + source boost. New --detail flag (low/medium/high) controls timeline inclusion. Cosine re-scoring blends query-chunk similarity before dedup for query-specific ranking. Also: remove DISTINCT ON from keyword search (dedup handles per-page capping), add chunk_id + chunk_index to SearchResult, add getEmbeddingsByChunkIds to BrainEngine interface. Inspired by Ramp Labs' "Latent Briefing" paper (April 2026). * feat: RRF normalization, source-aware dedup, detail param in operations RRF scores normalized to 0-1 before 2.0x compiled truth boost. Source-aware dedup guarantees compiled truth chunk per page. Detail parameter added to query operation, dedupResults added to bare search operation. Debug logging via GBRAIN_SEARCH_DEBUG=1. * chore: bump version and changelog (v0.8.1) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: CJK word count in query expansion CJK text is not space-delimited. A query like "向量搜索优化" was counted as 1 word and silently skipped expansion. Now counts characters for CJK queries instead of space-separated tokens. Co-Authored-By: YIING99 <yiing99@users.noreply.github.com> * feat: retrieval evaluation harness — P@k, R@k, MRR, nDCG@k + gbrain eval Full IR evaluation framework: precisionAtK, recallAtK, mrr, ndcgAtK metrics with runEval() orchestrator. gbrain eval CLI with single-run table and A/B comparison mode (--config-a / --config-b) for parameter tuning. HybridSearchOpts now accepts rrfK and dedupOpts overrides. Co-Authored-By: 4shut0sh <4shut0sh@users.noreply.github.com> * test: search quality tests — RRF boost, dedup guarantee, cosine similarity, E2E benchmark 42 new tests across 3 files: - test/search.test.ts: RRF normalization, compiled truth 2x boost, dedup key collision prevention, cosine similarity edge cases, CJK word count detection - test/dedup.test.ts: source-aware compiled truth guarantee, layer interactions, custom maxPerPage, empty/single result edge cases - test/e2e/search-quality.test.ts: full pipeline against PGLite with basis vector embeddings — chunk_id/chunk_index fields, detail parameter filtering, getEmbeddingsByChunkIds, keyword multi-chunk, vector ordering Also: export rrfFusion + cosineSimilarity for unit testing, fix PGLite getEmbeddingsByChunkIds to parse string vectors from pgvector. * test: search quality benchmark with A/B comparison (baseline vs PR#64) Benchmark measures P@1, MRR, nDCG@5, and source accuracy across 8 queries against 5 seeded pages. Key finding: boost helps entity lookups but over-corrects temporal queries. Validates the --detail parameter as the right control mechanism. Output at docs/benchmarks/2026-04-13.md. * feat: query intent classifier — auto-selects detail level, 100% source accuracy Zero-latency heuristic classifier detects query intent from text patterns: - "Who is Pedro?" → entity → detail=low (compiled truth only) - "When did we last meet?" → temporal → detail=high (no boost, natural ranking) - "Variant fund announcement" → event → detail=high - General queries → detail=medium (default with boost) The key insight: skip the 2.0x compiled truth boost for detail=high queries. Temporal/event queries want natural ranking where timeline entries can win. Benchmark results (source accuracy = does the top chunk match expected type): - Baseline: 100% (already good, no boost needed) - Boost only: 71.4% (boost over-corrects temporal queries) - Boost + intent classifier: 100% (best of both worlds) 35 unit tests for the classifier. 590 total tests pass. * feat: query intent classifier — auto-selects detail level, 100% source accuracy Heuristic classifier detects query intent from text patterns (zero latency, no LLM call). Maps temporal queries ("when did we last meet") to detail=high, entity queries ("who is X") to detail=low, events to detail=high. Benchmark results (29 pages, 20 queries, graded relevance): - Baseline: P@1=0.947, MRR=0.974, source accuracy=89.5% - Boost only: P@1=0.895, MRR=0.939, source accuracy=63.2% (over-correction) - Boost + intent: P@1=0.947, MRR=0.974, source accuracy=89.5% (fully recovered) The intent classifier eliminates the boost's over-correction on temporal queries while preserving its benefits for entity lookups. 35 unit tests for the classifier. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: search quality benchmark with A/B comparison (baseline vs PR#64) Rich benchmark: 29 pages, 58 chunks, 20 queries with graded relevance. Now measures CHUNK-LEVEL quality, not just page-level retrieval. Key findings (C. Boost+Intent vs A. Baseline): - Unique pages in top-10: 7.2 → 8.7 (+21% broader coverage) - Compiled truth ratio: 51.6% → 66.8% (+15pp more signal) - CT-first rate: 100% (compiled truth leads for entity queries) - Timeline accessible: 100% (temporal queries still find dates) - Source accuracy: 89.5% maintained (intent classifier prevents regression) The boost alone (B) causes -26pp source accuracy regression. Intent classifier (C) recovers it fully. * docs: clean benchmark report — ELI10 search quality analysis for PR#64 Replaces two drafts with one clean report. Explains what changed, why it matters, and what the numbers mean. All fictional data, no private info. Key findings: 21% more page coverage per query, 29% more compiled truth in results. Intent classifier prevents boost from burying timeline for temporal queries. Full per-query breakdown with before/after comparison. * chore: remove auto-generated benchmark file (clean version is 2026-04-14-search-quality.md) * docs: update project documentation for search quality boost CLAUDE.md: added search/intent.ts, search/eval.ts, commands/eval.ts to key files. Added 5 new test files (search, dedup, intent, eval, e2e/search-quality). Updated test count from 23+4 to 28+5. Added docs/benchmarks/ to key files. README.md: updated search pipeline diagram with intent classifier, RRF normalization, compiled truth boost, cosine re-scoring, and 5-layer dedup. Added --detail flag explanation and benchmark instructions. CHANGELOG.md: added search quality entries to v0.9.3 (intent classifier, --detail flag, gbrain eval, CJK fix). Credited @4shut0sh and @YIING99. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: headline benchmark gains in changelog * docs: add community attribution rule to CHANGELOG voice section --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: YIING99 <yiing99@users.noreply.github.com> Co-authored-by: 4shut0sh <4shut0sh@users.noreply.github.com>
667 lines
23 KiB
TypeScript
667 lines
23 KiB
TypeScript
import { PGlite } from '@electric-sql/pglite';
|
|
import { vector } from '@electric-sql/pglite/vector';
|
|
import { pg_trgm } from '@electric-sql/pglite/contrib/pg_trgm';
|
|
import type { Transaction } from '@electric-sql/pglite';
|
|
import type { BrainEngine } from './engine.ts';
|
|
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
|
import { runMigrations } from './migrate.ts';
|
|
import { PGLITE_SCHEMA_SQL } from './pglite-schema.ts';
|
|
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
|
|
import type {
|
|
Page, PageInput, PageFilters, PageType,
|
|
Chunk, ChunkInput,
|
|
SearchResult, SearchOpts,
|
|
Link, GraphNode,
|
|
TimelineEntry, TimelineInput, TimelineOpts,
|
|
RawData,
|
|
PageVersion,
|
|
BrainStats, BrainHealth,
|
|
IngestLogEntry, IngestLogInput,
|
|
EngineConfig,
|
|
} from './types.ts';
|
|
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
|
|
|
|
type PGLiteDB = PGlite;
|
|
|
|
export class PGLiteEngine implements BrainEngine {
|
|
private _db: PGLiteDB | null = null;
|
|
private _lock: LockHandle | null = null;
|
|
|
|
get db(): PGLiteDB {
|
|
if (!this._db) throw new Error('PGLite not connected. Call connect() first.');
|
|
return this._db;
|
|
}
|
|
|
|
// Lifecycle
|
|
async connect(config: EngineConfig): Promise<void> {
|
|
const dataDir = config.database_path || undefined; // undefined = in-memory
|
|
|
|
// Acquire file lock to prevent concurrent PGLite access (crashes with Aborted())
|
|
this._lock = await acquireLock(dataDir);
|
|
|
|
if (!this._lock.acquired) {
|
|
throw new Error('Could not acquire PGLite lock. Another gbrain process is using the database.');
|
|
}
|
|
|
|
this._db = await PGlite.create({
|
|
dataDir,
|
|
extensions: { vector, pg_trgm },
|
|
});
|
|
}
|
|
|
|
async disconnect(): Promise<void> {
|
|
if (this._db) {
|
|
await this._db.close();
|
|
this._db = null;
|
|
}
|
|
if (this._lock?.acquired) {
|
|
await releaseLock(this._lock);
|
|
this._lock = null;
|
|
}
|
|
}
|
|
|
|
async initSchema(): Promise<void> {
|
|
await this.db.exec(PGLITE_SCHEMA_SQL);
|
|
|
|
const { applied } = await runMigrations(this);
|
|
if (applied > 0) {
|
|
console.log(` ${applied} migration(s) applied`);
|
|
}
|
|
}
|
|
|
|
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
|
return this.db.transaction(async (tx) => {
|
|
const txEngine = Object.create(this) as PGLiteEngine;
|
|
Object.defineProperty(txEngine, 'db', { get: () => tx });
|
|
return fn(txEngine);
|
|
});
|
|
}
|
|
|
|
// Pages CRUD
|
|
async getPage(slug: string): Promise<Page | null> {
|
|
const { rows } = await this.db.query(
|
|
`SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
|
|
FROM pages WHERE slug = $1`,
|
|
[slug]
|
|
);
|
|
if (rows.length === 0) return null;
|
|
return rowToPage(rows[0] as Record<string, unknown>);
|
|
}
|
|
|
|
async putPage(slug: string, page: PageInput): Promise<Page> {
|
|
slug = validateSlug(slug);
|
|
const hash = page.content_hash || contentHash(page.compiled_truth, page.timeline || '');
|
|
const frontmatter = page.frontmatter || {};
|
|
|
|
const { rows } = await this.db.query(
|
|
`INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
|
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, now())
|
|
ON CONFLICT (slug) DO UPDATE SET
|
|
type = EXCLUDED.type,
|
|
title = EXCLUDED.title,
|
|
compiled_truth = EXCLUDED.compiled_truth,
|
|
timeline = EXCLUDED.timeline,
|
|
frontmatter = EXCLUDED.frontmatter,
|
|
content_hash = EXCLUDED.content_hash,
|
|
updated_at = now()
|
|
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at`,
|
|
[slug, page.type, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash]
|
|
);
|
|
return rowToPage(rows[0] as Record<string, unknown>);
|
|
}
|
|
|
|
async deletePage(slug: string): Promise<void> {
|
|
await this.db.query('DELETE FROM pages WHERE slug = $1', [slug]);
|
|
}
|
|
|
|
async listPages(filters?: PageFilters): Promise<Page[]> {
|
|
const limit = filters?.limit || 100;
|
|
const offset = filters?.offset || 0;
|
|
|
|
let result;
|
|
if (filters?.type && filters?.tag) {
|
|
result = await this.db.query(
|
|
`SELECT p.* FROM pages p
|
|
JOIN tags t ON t.page_id = p.id
|
|
WHERE p.type = $1 AND t.tag = $2
|
|
ORDER BY p.updated_at DESC LIMIT $3 OFFSET $4`,
|
|
[filters.type, filters.tag, limit, offset]
|
|
);
|
|
} else if (filters?.type) {
|
|
result = await this.db.query(
|
|
`SELECT * FROM pages WHERE type = $1
|
|
ORDER BY updated_at DESC LIMIT $2 OFFSET $3`,
|
|
[filters.type, limit, offset]
|
|
);
|
|
} else if (filters?.tag) {
|
|
result = await this.db.query(
|
|
`SELECT p.* FROM pages p
|
|
JOIN tags t ON t.page_id = p.id
|
|
WHERE t.tag = $1
|
|
ORDER BY p.updated_at DESC LIMIT $2 OFFSET $3`,
|
|
[filters.tag, limit, offset]
|
|
);
|
|
} else {
|
|
result = await this.db.query(
|
|
`SELECT * FROM pages
|
|
ORDER BY updated_at DESC LIMIT $1 OFFSET $2`,
|
|
[limit, offset]
|
|
);
|
|
}
|
|
|
|
return (result.rows as Record<string, unknown>[]).map(rowToPage);
|
|
}
|
|
|
|
async resolveSlugs(partial: string): Promise<string[]> {
|
|
// Try exact match first
|
|
const exact = await this.db.query('SELECT slug FROM pages WHERE slug = $1', [partial]);
|
|
if (exact.rows.length > 0) return [(exact.rows[0] as { slug: string }).slug];
|
|
|
|
// Fuzzy match via pg_trgm
|
|
const { rows } = await this.db.query(
|
|
`SELECT slug, similarity(title, $1) AS sim
|
|
FROM pages
|
|
WHERE title % $1 OR slug ILIKE $2
|
|
ORDER BY sim DESC
|
|
LIMIT 5`,
|
|
[partial, '%' + partial + '%']
|
|
);
|
|
return (rows as { slug: string }[]).map(r => r.slug);
|
|
}
|
|
|
|
// Search
|
|
async searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
|
const limit = clampSearchLimit(opts?.limit);
|
|
const offset = opts?.offset || 0;
|
|
const detailFilter = opts?.detail === 'low' ? `AND cc.chunk_source = 'compiled_truth'` : '';
|
|
|
|
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
|
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
|
}
|
|
|
|
const { rows } = await this.db.query(
|
|
`SELECT
|
|
p.slug, p.id as page_id, p.title, p.type,
|
|
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
|
ts_rank(p.search_vector, websearch_to_tsquery('english', $1)) AS score,
|
|
CASE WHEN p.updated_at < (
|
|
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
|
) THEN true ELSE false END AS stale
|
|
FROM pages p
|
|
JOIN content_chunks cc ON cc.page_id = p.id
|
|
WHERE p.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}
|
|
ORDER BY score DESC
|
|
LIMIT $2
|
|
OFFSET $3`,
|
|
[query, limit, offset]
|
|
);
|
|
|
|
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
|
}
|
|
|
|
async searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]> {
|
|
const limit = clampSearchLimit(opts?.limit);
|
|
const offset = opts?.offset || 0;
|
|
const vecStr = '[' + Array.from(embedding).join(',') + ']';
|
|
const detailFilter = opts?.detail === 'low' ? `AND cc.chunk_source = 'compiled_truth'` : '';
|
|
|
|
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
|
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
|
}
|
|
|
|
const { rows } = await this.db.query(
|
|
`SELECT
|
|
p.slug, p.id as page_id, p.title, p.type,
|
|
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
|
1 - (cc.embedding <=> $1::vector) AS score,
|
|
CASE WHEN p.updated_at < (
|
|
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
|
) THEN true ELSE false END AS stale
|
|
FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE cc.embedding IS NOT NULL ${detailFilter}
|
|
ORDER BY cc.embedding <=> $1::vector
|
|
LIMIT $2
|
|
OFFSET $3`,
|
|
[vecStr, limit, offset]
|
|
);
|
|
|
|
return (rows as Record<string, unknown>[]).map(rowToSearchResult);
|
|
}
|
|
|
|
async getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>> {
|
|
if (ids.length === 0) return new Map();
|
|
const { rows } = await this.db.query(
|
|
`SELECT id, embedding FROM content_chunks WHERE id = ANY($1::int[]) AND embedding IS NOT NULL`,
|
|
[ids]
|
|
);
|
|
const result = new Map<number, Float32Array>();
|
|
for (const row of rows as Record<string, unknown>[]) {
|
|
if (row.embedding) {
|
|
const emb = typeof row.embedding === 'string'
|
|
? new Float32Array(JSON.parse(row.embedding))
|
|
: row.embedding as Float32Array;
|
|
result.set(row.id as number, emb);
|
|
}
|
|
}
|
|
return result;
|
|
}
|
|
|
|
// Chunks
|
|
async upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void> {
|
|
// Get page_id
|
|
const pageResult = await this.db.query('SELECT id FROM pages WHERE slug = $1', [slug]);
|
|
if (pageResult.rows.length === 0) throw new Error(`Page not found: ${slug}`);
|
|
const pageId = (pageResult.rows[0] as { id: number }).id;
|
|
|
|
// Remove chunks that no longer exist
|
|
const newIndices = chunks.map(c => c.chunk_index);
|
|
if (newIndices.length > 0) {
|
|
// PGLite doesn't auto-serialize arrays, so use ANY with explicit array cast
|
|
await this.db.query(
|
|
`DELETE FROM content_chunks WHERE page_id = $1 AND chunk_index != ALL($2::int[])`,
|
|
[pageId, newIndices]
|
|
);
|
|
} else {
|
|
await this.db.query('DELETE FROM content_chunks WHERE page_id = $1', [pageId]);
|
|
return;
|
|
}
|
|
|
|
// Batch upsert: build dynamic multi-row INSERT
|
|
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at)';
|
|
const rowParts: string[] = [];
|
|
const params: unknown[] = [];
|
|
let paramIdx = 1;
|
|
|
|
for (const chunk of chunks) {
|
|
const embeddingStr = chunk.embedding
|
|
? '[' + Array.from(chunk.embedding).join(',') + ']'
|
|
: null;
|
|
|
|
if (embeddingStr) {
|
|
rowParts.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::vector, $${paramIdx++}, $${paramIdx++}, now())`);
|
|
params.push(pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source, embeddingStr, chunk.model || 'text-embedding-3-large', chunk.token_count || null);
|
|
} else {
|
|
rowParts.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, NULL)`);
|
|
params.push(pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source, chunk.model || 'text-embedding-3-large', chunk.token_count || null);
|
|
}
|
|
}
|
|
|
|
await this.db.query(
|
|
`INSERT INTO content_chunks ${cols} VALUES ${rowParts.join(', ')}
|
|
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
|
|
chunk_text = EXCLUDED.chunk_text,
|
|
chunk_source = EXCLUDED.chunk_source,
|
|
embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding ELSE COALESCE(EXCLUDED.embedding, content_chunks.embedding) END,
|
|
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
|
token_count = EXCLUDED.token_count,
|
|
embedded_at = COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)`,
|
|
params
|
|
);
|
|
}
|
|
|
|
async getChunks(slug: string): Promise<Chunk[]> {
|
|
const { rows } = await this.db.query(
|
|
`SELECT cc.* FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE p.slug = $1
|
|
ORDER BY cc.chunk_index`,
|
|
[slug]
|
|
);
|
|
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r));
|
|
}
|
|
|
|
async deleteChunks(slug: string): Promise<void> {
|
|
await this.db.query(
|
|
`DELETE FROM content_chunks
|
|
WHERE page_id = (SELECT id FROM pages WHERE slug = $1)`,
|
|
[slug]
|
|
);
|
|
}
|
|
|
|
// Links
|
|
async addLink(from: string, to: string, context?: string, linkType?: string): Promise<void> {
|
|
await this.db.query(
|
|
`INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
|
SELECT f.id, t.id, $3, $4
|
|
FROM pages f, pages t
|
|
WHERE f.slug = $1 AND t.slug = $2
|
|
ON CONFLICT (from_page_id, to_page_id) DO UPDATE SET
|
|
link_type = EXCLUDED.link_type,
|
|
context = EXCLUDED.context`,
|
|
[from, to, linkType || '', context || '']
|
|
);
|
|
}
|
|
|
|
async removeLink(from: string, to: string): Promise<void> {
|
|
await this.db.query(
|
|
`DELETE FROM links
|
|
WHERE from_page_id = (SELECT id FROM pages WHERE slug = $1)
|
|
AND to_page_id = (SELECT id FROM pages WHERE slug = $2)`,
|
|
[from, to]
|
|
);
|
|
}
|
|
|
|
async getLinks(slug: string): Promise<Link[]> {
|
|
const { rows } = await this.db.query(
|
|
`SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context
|
|
FROM links l
|
|
JOIN pages f ON f.id = l.from_page_id
|
|
JOIN pages t ON t.id = l.to_page_id
|
|
WHERE f.slug = $1`,
|
|
[slug]
|
|
);
|
|
return rows as unknown as Link[];
|
|
}
|
|
|
|
async getBacklinks(slug: string): Promise<Link[]> {
|
|
const { rows } = await this.db.query(
|
|
`SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context
|
|
FROM links l
|
|
JOIN pages f ON f.id = l.from_page_id
|
|
JOIN pages t ON t.id = l.to_page_id
|
|
WHERE t.slug = $1`,
|
|
[slug]
|
|
);
|
|
return rows as unknown as Link[];
|
|
}
|
|
|
|
async traverseGraph(slug: string, depth: number = 5): Promise<GraphNode[]> {
|
|
const { rows } = await this.db.query(
|
|
`WITH RECURSIVE graph AS (
|
|
SELECT p.id, p.slug, p.title, p.type, 0 as depth
|
|
FROM pages p WHERE p.slug = $1
|
|
|
|
UNION
|
|
|
|
SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1
|
|
FROM graph g
|
|
JOIN links l ON l.from_page_id = g.id
|
|
JOIN pages p2 ON p2.id = l.to_page_id
|
|
WHERE g.depth < $2
|
|
)
|
|
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
|
|
coalesce(
|
|
(SELECT jsonb_agg(jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
|
|
FROM links l2
|
|
JOIN pages p3 ON p3.id = l2.to_page_id
|
|
WHERE l2.from_page_id = g.id),
|
|
'[]'::jsonb
|
|
) as links
|
|
FROM graph g
|
|
ORDER BY g.depth, g.slug`,
|
|
[slug, depth]
|
|
);
|
|
|
|
return (rows as Record<string, unknown>[]).map(r => ({
|
|
slug: r.slug as string,
|
|
title: r.title as string,
|
|
type: r.type as PageType,
|
|
depth: r.depth as number,
|
|
links: (typeof r.links === 'string' ? JSON.parse(r.links) : r.links) as { to_slug: string; link_type: string }[],
|
|
}));
|
|
}
|
|
|
|
// Tags
|
|
async addTag(slug: string, tag: string): Promise<void> {
|
|
await this.db.query(
|
|
`INSERT INTO tags (page_id, tag)
|
|
SELECT id, $2 FROM pages WHERE slug = $1
|
|
ON CONFLICT (page_id, tag) DO NOTHING`,
|
|
[slug, tag]
|
|
);
|
|
}
|
|
|
|
async removeTag(slug: string, tag: string): Promise<void> {
|
|
await this.db.query(
|
|
`DELETE FROM tags
|
|
WHERE page_id = (SELECT id FROM pages WHERE slug = $1)
|
|
AND tag = $2`,
|
|
[slug, tag]
|
|
);
|
|
}
|
|
|
|
async getTags(slug: string): Promise<string[]> {
|
|
const { rows } = await this.db.query(
|
|
`SELECT tag FROM tags
|
|
WHERE page_id = (SELECT id FROM pages WHERE slug = $1)
|
|
ORDER BY tag`,
|
|
[slug]
|
|
);
|
|
return (rows as { tag: string }[]).map(r => r.tag);
|
|
}
|
|
|
|
// Timeline
|
|
async addTimelineEntry(slug: string, entry: TimelineInput): Promise<void> {
|
|
await this.db.query(
|
|
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
|
SELECT id, $2::date, $3, $4, $5
|
|
FROM pages WHERE slug = $1`,
|
|
[slug, entry.date, entry.source || '', entry.summary, entry.detail || '']
|
|
);
|
|
}
|
|
|
|
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
|
const limit = opts?.limit || 100;
|
|
|
|
let result;
|
|
if (opts?.after && opts?.before) {
|
|
result = await this.db.query(
|
|
`SELECT te.* FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id
|
|
WHERE p.slug = $1 AND te.date >= $2::date AND te.date <= $3::date
|
|
ORDER BY te.date DESC LIMIT $4`,
|
|
[slug, opts.after, opts.before, limit]
|
|
);
|
|
} else if (opts?.after) {
|
|
result = await this.db.query(
|
|
`SELECT te.* FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id
|
|
WHERE p.slug = $1 AND te.date >= $2::date
|
|
ORDER BY te.date DESC LIMIT $3`,
|
|
[slug, opts.after, limit]
|
|
);
|
|
} else {
|
|
result = await this.db.query(
|
|
`SELECT te.* FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id
|
|
WHERE p.slug = $1
|
|
ORDER BY te.date DESC LIMIT $2`,
|
|
[slug, limit]
|
|
);
|
|
}
|
|
|
|
return result.rows as unknown as TimelineEntry[];
|
|
}
|
|
|
|
// Raw data
|
|
async putRawData(slug: string, source: string, data: object): Promise<void> {
|
|
await this.db.query(
|
|
`INSERT INTO raw_data (page_id, source, data)
|
|
SELECT id, $2, $3::jsonb
|
|
FROM pages WHERE slug = $1
|
|
ON CONFLICT (page_id, source) DO UPDATE SET
|
|
data = EXCLUDED.data,
|
|
fetched_at = now()`,
|
|
[slug, source, JSON.stringify(data)]
|
|
);
|
|
}
|
|
|
|
async getRawData(slug: string, source?: string): Promise<RawData[]> {
|
|
let result;
|
|
if (source) {
|
|
result = await this.db.query(
|
|
`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
|
JOIN pages p ON p.id = rd.page_id
|
|
WHERE p.slug = $1 AND rd.source = $2`,
|
|
[slug, source]
|
|
);
|
|
} else {
|
|
result = await this.db.query(
|
|
`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
|
JOIN pages p ON p.id = rd.page_id
|
|
WHERE p.slug = $1`,
|
|
[slug]
|
|
);
|
|
}
|
|
return result.rows as unknown as RawData[];
|
|
}
|
|
|
|
// Versions
|
|
async createVersion(slug: string): Promise<PageVersion> {
|
|
const { rows } = await this.db.query(
|
|
`INSERT INTO page_versions (page_id, compiled_truth, frontmatter)
|
|
SELECT id, compiled_truth, frontmatter
|
|
FROM pages WHERE slug = $1
|
|
RETURNING *`,
|
|
[slug]
|
|
);
|
|
return rows[0] as unknown as PageVersion;
|
|
}
|
|
|
|
async getVersions(slug: string): Promise<PageVersion[]> {
|
|
const { rows } = await this.db.query(
|
|
`SELECT pv.* FROM page_versions pv
|
|
JOIN pages p ON p.id = pv.page_id
|
|
WHERE p.slug = $1
|
|
ORDER BY pv.snapshot_at DESC`,
|
|
[slug]
|
|
);
|
|
return rows as unknown as PageVersion[];
|
|
}
|
|
|
|
async revertToVersion(slug: string, versionId: number): Promise<void> {
|
|
await this.db.query(
|
|
`UPDATE pages SET
|
|
compiled_truth = pv.compiled_truth,
|
|
frontmatter = pv.frontmatter,
|
|
updated_at = now()
|
|
FROM page_versions pv
|
|
WHERE pages.slug = $1 AND pv.id = $2 AND pv.page_id = pages.id`,
|
|
[slug, versionId]
|
|
);
|
|
}
|
|
|
|
// Stats + health
|
|
async getStats(): Promise<BrainStats> {
|
|
const { rows: [stats] } = await this.db.query(`
|
|
SELECT
|
|
(SELECT count(*) FROM pages) as page_count,
|
|
(SELECT count(*) FROM content_chunks) as chunk_count,
|
|
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count,
|
|
(SELECT count(*) FROM links) as link_count,
|
|
(SELECT count(DISTINCT tag) FROM tags) as tag_count,
|
|
(SELECT count(*) FROM timeline_entries) as timeline_entry_count
|
|
`);
|
|
|
|
const { rows: types } = await this.db.query(
|
|
`SELECT type, count(*)::int as count FROM pages GROUP BY type ORDER BY count DESC`
|
|
);
|
|
const pages_by_type: Record<string, number> = {};
|
|
for (const t of types as { type: string; count: number }[]) {
|
|
pages_by_type[t.type] = t.count;
|
|
}
|
|
|
|
const s = stats as Record<string, unknown>;
|
|
return {
|
|
page_count: Number(s.page_count),
|
|
chunk_count: Number(s.chunk_count),
|
|
embedded_count: Number(s.embedded_count),
|
|
link_count: Number(s.link_count),
|
|
tag_count: Number(s.tag_count),
|
|
timeline_entry_count: Number(s.timeline_entry_count),
|
|
pages_by_type,
|
|
};
|
|
}
|
|
|
|
async getHealth(): Promise<BrainHealth> {
|
|
const { rows: [h] } = await this.db.query(`
|
|
SELECT
|
|
(SELECT count(*) FROM pages) as page_count,
|
|
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
|
|
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
|
|
(SELECT count(*) FROM pages p
|
|
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
|
|
) as stale_pages,
|
|
(SELECT count(*) FROM pages p
|
|
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
|
) as orphan_pages,
|
|
(SELECT count(*) FROM links l
|
|
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
|
|
) as dead_links,
|
|
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings
|
|
`);
|
|
|
|
const r = h as Record<string, unknown>;
|
|
return {
|
|
page_count: Number(r.page_count),
|
|
embed_coverage: Number(r.embed_coverage),
|
|
stale_pages: Number(r.stale_pages),
|
|
orphan_pages: Number(r.orphan_pages),
|
|
dead_links: Number(r.dead_links),
|
|
missing_embeddings: Number(r.missing_embeddings),
|
|
};
|
|
}
|
|
|
|
// Ingest log
|
|
async logIngest(entry: IngestLogInput): Promise<void> {
|
|
await this.db.query(
|
|
`INSERT INTO ingest_log (source_type, source_ref, pages_updated, summary)
|
|
VALUES ($1, $2, $3::jsonb, $4)`,
|
|
[entry.source_type, entry.source_ref, JSON.stringify(entry.pages_updated), entry.summary]
|
|
);
|
|
}
|
|
|
|
async getIngestLog(opts?: { limit?: number }): Promise<IngestLogEntry[]> {
|
|
const limit = opts?.limit || 50;
|
|
const { rows } = await this.db.query(
|
|
`SELECT * FROM ingest_log ORDER BY created_at DESC LIMIT $1`,
|
|
[limit]
|
|
);
|
|
return rows as unknown as IngestLogEntry[];
|
|
}
|
|
|
|
// Sync
|
|
async updateSlug(oldSlug: string, newSlug: string): Promise<void> {
|
|
newSlug = validateSlug(newSlug);
|
|
await this.db.query(
|
|
`UPDATE pages SET slug = $1, updated_at = now() WHERE slug = $2`,
|
|
[newSlug, oldSlug]
|
|
);
|
|
}
|
|
|
|
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
|
|
// Stub: links use integer page_id FKs, already correct after updateSlug.
|
|
}
|
|
|
|
// Config
|
|
async getConfig(key: string): Promise<string | null> {
|
|
const { rows } = await this.db.query('SELECT value FROM config WHERE key = $1', [key]);
|
|
return rows.length > 0 ? (rows[0] as { value: string }).value : null;
|
|
}
|
|
|
|
async setConfig(key: string, value: string): Promise<void> {
|
|
await this.db.query(
|
|
`INSERT INTO config (key, value) VALUES ($1, $2)
|
|
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value`,
|
|
[key, value]
|
|
);
|
|
}
|
|
|
|
// Migration support
|
|
async runMigration(_version: number, sql: string): Promise<void> {
|
|
await this.db.exec(sql);
|
|
}
|
|
|
|
async getChunksWithEmbeddings(slug: string): Promise<Chunk[]> {
|
|
const { rows } = await this.db.query(
|
|
`SELECT cc.* FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE p.slug = $1
|
|
ORDER BY cc.chunk_index`,
|
|
[slug]
|
|
);
|
|
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r, true));
|
|
}
|
|
}
|