The Limit

A language model is, at its core, a very large function: text in, text out. What happens in between is a massive statistical process that predicts, token by token, what comes next.

For that to work, the model has to be able to read your text. And that's exactly where the trouble starts that ultimately forces us into RAG.

Tokens - how language models count

A language model doesn't see letters or words - it sees tokens. Tokens are small fragments of language: sometimes a whole word, sometimes a syllable, sometimes a single character. A separate piece of code called the tokenizer decides how text gets cut up.

Rule of thumb: one token corresponds to roughly four characters of English or German prose. Long words and special characters skew that downwards.

Type into the field above - watch how the token count and character count move together, but never at a clean 1:1 ratio. Long words and emoji break into several tokens.

Why does this matter? Because the price and the speed of a model call are almost always computed per token - and because the next concept defines a hard limit in tokens.

The context window - the hard wall

Every model has a context window: the maximum number of tokens it can process in a single call. Anything that doesn't fit, the model simply doesn't see.

Sounds like a lot. It is - until you put it next to a real-world use case.

• An average page of plain prose runs about 500 tokens.
• A novel like The Lord of the Rings is roughly 600,000 tokens.
• A medium-sized Confluence knowledge base hits several million tokens in no time.

Even the largest commercial models fall over on a medium-sized knowledge base. And the bigger the window, the slower and more expensive each call gets. A 1M-token call means seconds of latency and several cents per request - even for short answers.

There's a second effect on top of that: even when a model nominally accepts a million tokens, it uses information from the middle of a very long context noticeably worse than from the beginning or end. This is called Lost in the Middle, and it's typically measured with Needle-in-a-Haystack benchmarks. We won't go deeper here - but keep this in mind: a large context window doesn't mean the model sees equally well throughout it.

Lab: what happens if we just stuff everything in?

Lab 1 Naive RAG shows what happens when you pick the naive solution: dump all documents into the prompt and see how far you get.

In there you can:

• add animal profiles to the prompt one by one,
• watch the token count climb in real time,
• measure answer quality and per-request latency,
• see exactly where it falls over - either with an error, a hallucinated answer, or just too much latency to be useful in practice.

That's the pain threshold we start from. In the next chapter we make the first clever move: don't ship everything, just the right snippets.