Injecting Context

The lesson from chapter 1: stuffing everything into the prompt doesn't scale. But the model can answer questions just fine - if the right information is in the context. That's the observation RAG is built on.

Snippets, not the whole corpus

Instead of shipping the entire knowledge base, we hand the model just the few paragraphs needed for the question. Two immediate wins:

• Token use and latency drop dramatically.
• Hallucinations drop too - the model invents less when the answer is right in front of it.

The prompt looks roughly like this:

[context]
<snippet A>
<snippet B>

[question]
Where do penguins live?

If we find the right snippets, the answer becomes almost trivial. Which raises the real question: how do we find them?

First attempt: full-text search

The obvious move: Postgres has full-text. Build a tsvector index, send the question as a query, take top-K hits as context.

That works for anything literal. Ask "Where do penguins live?" and the entry containing "penguin" and "Antarctica" comes back.

What doesn't work: paraphrases.

Try: "Which bird cannot fly but swims well?". Full-text search misses, because none of those keywords show up verbatim in the penguin profile - and the search has no notion of meaning.

Lab: BM25 in action

Lab 1 Naive RAG, tab "Full-text search", lets you pose the same question three different ways - verbatim, paraphrased, with unfamiliar vocabulary. Full-text hits some, misses others, and you watch the wall the moment you walk into it.

That wall is what forces the next concept on us: search by meaning, not by words.