Reality, Failures, Eval

Up to here, RAG looks tidy. In the wild it isn't. Here are the obstacles you almost certainly hit - and how to climb out.

Tables, headings, lists, code

Pure prose is the easy mode. Real knowledge bases contain:

• Tables. Naive chunking slices rows in half - header and value lose their connection.
• Headings. With "Adaptations for Antarctica" as an H2 at the top and three paragraphs of content below, the heading anchor disappears unless your chunker preserves it.
• Lists. Step 7 of a numbered list without steps 1-6 isn't useful.
• Code. Char-based splitting cuts code blocks at arbitrary points.
• Captions. "Fig. 3: distribution in central Europe" is in the document, but the figure itself isn't - caption text alone tells you nothing.

The fix is structure-aware chunking. Tools like doclingExternal - Opens in a new tab parse the document structure and chunk along it: tables stay intact, headings live in the chunk metadata, lists remain lists.

Lab 5 shows the direct comparison: "naive fixed chunks" vs. "docling".

How do we measure if the system is good?

You need a ground-truth set: a list of questions where you've decided up front which documents contain the right answer. Now you can compare retrieval configurations objectively.

Two simple metrics get you started:

• Precision @ k - fraction of questions where at least one expected document is among the top-K hits. Practical, because the LLM only sees the top-K anyway.
• Recall @ k - fraction of expected documents that land in the top-K. Matters when a question has multiple correct sources.

The interesting questions are usually the paraphrased ones, the ones requiring multiple documents, or the ones asking for table values. That's exactly what your eval-set categories should target.

Ballpark numbers from this tutorial setup:

(Exact numbers depend on your concrete corpus and eval set.)

Cost and privacy - the inconvenient axes

You pay for RAG in two currencies:

• Money - with cloud providers mainly embedding and LLM token costs. A 50k-token corpus with 200 chunks costs a few cents per embedding pass. Per query, the LLM call is on top - anywhere from cents to dollars depending on model and answer length.
• Privacy - if your corpus is sensitive, every line shipped to a cloud embedder is a leak. Local embeddings (Ollama) and a local LLM mean: 0 € in tokens, but disk footprint and cold-start latency.

Lab 5, tab "Cost & Privacy", lets you play with corpus sizes and watch each side of the trade-off.

Lab: failures & evaluation

Lab 5 has three tabs:

1. Failure showcase - four concrete examples where naive chunks fail and docling rescues.
2. Eval run - three configurations (naive, docling, docling+rerank) walked across the eval set with Precision@1/3/5 as bars.
3. Cost & privacy - sliders for corpus shape, side-by-side: cloud stack vs. this local stack.