Chunking

We've now combined two truths:

1. Language models understand meaning.
2. Embedding models - the ones turning that meaning into vectors - have their own small context window of a few hundred to a few thousand tokens.

So long documents have to be cut up before embedding. That cutting is chunking.

Chunks are retrieval candidates

Each chunk later becomes its own vector and therefore its own candidate hit during retrieval. Pick chunks that are too big and you end up with a handful of very broad candidates - the model has to wade through a lot of material to find the actual answer.

Pick them too small and the chunk loses the context that gives a fact meaning. "1500 kg" on its own tells you nothing; "an adult bull elephant weighs roughly 1500 kg" is an answer.

Chunking is therefore the single biggest knob separating good and bad RAG systems.

Four strategies - from dumb to clever

• Fixed - split every N characters. Stupid, but a serviceable baseline.
• Fixed with overlap - same as above, but each chunk shares M characters with its predecessor. Statements straddling a cut survive.
• Semantic - split at paragraph or sentence boundaries, then merge to a target size. Keeps natural units of meaning intact.
• Structure-aware (e.g. docling) - understands headings, lists, tables, code. Chunks along those structures, not across them. Mandatory once your documents aren't pure prose.

A numeric table, naively cut by "fixed", can be sliced through the middle of a row. And that's exactly what kills retrieval later: the answer is in the corpus, but it never existed as a contiguous chunk.

Lab: chunking workbench

Lab 3 lets you pick an animal profile, drag the size and overlap sliders, and see the same text split under each strategy. The view shows:

• how many chunks each strategy produces,
• how big they are on average,
• where they overlap,
• and what the original text looks like inside each chunk.

Play with the sliders. You'll feel that there's no "right" chunking - only one that fits the corpus you happen to have.