Building a Semantic Memory System with pgvector

One of the most powerful features of Huddix is the ability to search across all your meetings using natural language. "What did Sarah say about the Q4 roadmap?" — and you get instant, accurate answers with source citations. This post explains how we built this semantic memory system.

Our Architecture

1. Embedding Generation

For each meeting, we generate embeddings for the full transcript, the AI summary, extracted key points and decisions, and action items. We use OpenAI's text-embedding-3-large model, which produces 3072-dimensional vectors with state-of-the-art performance on semantic similarity benchmarks.

2. Vector Storage with pgvector

We store embeddings in PostgreSQL using the pgvector extension. Each meeting's embeddings are stored alongside metadata (date, participants, topics) in a single table with a vector index for efficient similarity search. We use Neon Serverless PostgreSQL.

3. Hybrid Search

When you search "What did Sarah say about Q4?", we generate an embedding for your query, find similar embeddings using vector similarity (cosine distance), filter by participants if you mention a name, apply keyword boosting for exact matches, and re-rank results using our relevance model.

4. Memory Graph

We go beyond simple vector search by building a memory graph. Entities (people, projects, companies) are extracted and linked across meetings. Related meetings are connected with similarity scores — meetings above 0.72 similarity are linked, and above 0.82 are marked as strong links.

5. Ask AI: Conversational Search

On top of the semantic search infrastructure, we built Ask AI — a conversational search interface powered by GPT. Unlike simple search, Ask AI supports multi-turn conversations with session persistence. Every response includes source citations that link back to the original meetings.

Performance

Our semantic search system handles 500K+ meetings with ~50ms average query latency (p95), 94% relevance score (human eval), and 99.9% uptime.