Retrieval

We have an embedding for every chunk, neatly stored in a pgvector table. The central move now: when a question comes in, embed the question itself, then look for the K chunks whose vector is closest to the question vector.

Cosine distance, briefly

Two vectors are "similar" if the angle between them is small. We measure that with cosine similarity (1 = very similar, 0 = orthogonal, -1 = opposite). pgvector exposes the operator <=> as a distance, i.e. 1 - cosine_similarity.

A typical top-K query:

SELECT chunk_id, content,
       1 - (vector_768 <=> :query_vec) AS score
FROM rag_embeddings JOIN rag_chunks USING (chunk_id)
ORDER BY vector_768 <=> :query_vec
LIMIT 5;

The HNSW index keeps that under 50 ms even at tens of thousands of vectors.

Vector alone isn't enough - hybrid with BM25

Vector search shines on paraphrase. It's weak on rare, exact terms - proper names, codes, technical IDs. That's classic BM25 territory.

Hybrid retrieval combines both. We fetch top-K from the vector index and top-K from the BM25 index and merge with Reciprocal Rank Fusion (RRF):

RRF_score(chunk) = Σ over all lists (1 / (60 + rank_in_that_list))

RRF is embarrassingly simple and surprisingly robust in practice - no fine-tuned weights required.

Rerank - the cross-encoder gets the last word

Vector embeddings are "bi-encoded": query and chunk are encoded independently, then their distance measured. Fast but coarse. A cross-encoder like the BGE reranker sees query and chunk together and judges far more finely - just slower.

So the trick: pull ~20 candidates from hybrid retrieval, and let the reranker pick the top 5. Best-of-both: vector speed, cross-encoder precision.

HyDE - when the question is too short

Some questions are just a handful of words. The embedding vector turns out "thin" and doesn't hit well. HyDE (Hypothetical Document Embeddings) flips it: have the LLM invent a plausible-looking answer, embed that, then search. Even if the invented answer is hallucinated - it's semantically closer to the real hits than the bare question.

Lab: retrieval compare

Lab 4 lets you fire the same question into four modes side by side: vector, BM25, hybrid, hybrid+rerank. You see:

• How the top-5 ordering changes.
• Which mode favours which kind of question.
• What HyDE does to the hit list.
• With the "Cross-language" toggle: how multilingual embeddings answer EN questions on DE documents - and where BM25 collapses.

And at the bottom, you can send the top-K straight to the LLM and watch the answer stream in.