Tool calling in production
Tool calls work in the prototype and break in production - not because of the model but because of six translations in between. What goes wrong, why it is so hard to see, and what you build against it.
Basics
Free for everyone: the concept, the analogy, the why.
Tools & Loop covers how a tool call comes about: descriptions into the prompt, read the call out of the answer, execute, result back into the history, keep turning. That is the mechanism, and it takes an afternoon to build.
This page starts where that afternoon ends - on the day your agent no longer drives five tools on your laptop but forty in a system that belongs to other people.
Two numbers that do not fit together
Models are advertised with their tool-calling hit rate, and the numbers are impressive. They describe one call: a request, a tool list, one call, right or wrong.
An agent does not make one call. It makes ten or twenty in a row, and each one works on the result of the previous one. The probabilities multiply - and that is not a metaphor, it is a multiplication:
Work the chain through
On the left, the hit rate of a single tool call - the number model announcements quote. On the right, how many calls the agent makes in a row, each building on the previous result. What you get below is the product, not the average.
fill height = remaining probability
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
From step 14 onwards, fewer than half the runs are still intact.
35.8%
95.0% to the power of 20
2
That is how short the chain would have to be for this step quality to reach 90 %.
99.47%
That is how good every single step would have to be across 20 steps.
35.8% across 20 steps: two out of three tasks do not make it through. From step 14 onwards most runs are already damaged, and everything after that keeps working on a wrong intermediate result.
Guided, in five steps
Each step sets both sliders for you - if you would rather move them yourself, you can still reveal the finding.
What people usually get wrong here
- A better model solves this.
- →A better model raises the base. The exponent stays - and it is the stronger lever. Shorter chains and checks in between buy more for the same effort.
- 95 % sounds fine.
- →Across twenty steps that is a third. Production readiness starts around 99.5 % per step, and you do not get there through the model but through the harness around it.
- An error means the run is over.
- →Only an *unhandled* one. A failure the agent learns from and avoids on the second attempt raises the base - which is exactly where the toolkit in part two of this page comes in.
The arithmetic assumes the steps build on one another, which is the normal case for agents. Calls that are independent of each other do not multiply; they only add up in cost.
Anyone who has pulled those sliders once already knows the conclusion of this page: the exponent is the stronger lever, not the base. A better model raises the hit rate per step. A shorter chain, a failure that is caught instead of passed on, a call that cannot be guessed wrong in the first place - all of that acts on the exponent, and that is why it acts harder.
The call passes through six hands
Between “the model understood what to do” and “the target system did it” sit six translations. Each can break on its own, and five of them look exactly like a model that cannot do it:
| # | Layer | What gets translated here |
|---|---|---|
| 1 | Model | Intent → decision: call, what, with what |
| 2 | Template | Tool list + history → prompt text |
| 3 | Decoding | Distribution → token sequence (temperature, cap, stop markers) |
| 4 | Parser | Raw output → call object |
| 5 | Context | Result or error → what the model reads next |
| 6 | Environment | Call → effect in the target system |
Only layer 1 can be fixed with a better model. Fine-tuning does nothing for layer 4, prompt craft nothing for layer 2, and nothing AI-related at all for layer 6.
Why you notice so late
The real problem is not that tool calls fail. It is that they fail quietly.
- A call that was not read raises no exception - it produces an empty list. The agent answers politely and does nothing.
- A model without a stop marker writes the tool response itself. The invented numbers are indistinguishable from real ones because they arrive in the same format.
- A tool returning the same error four times looks like four attempts in the log and is in truth one attempt repeated four times.
All three cases share a cause: the harness logs what it understood, not what the model wrote. Log the raw output unchanged - before any processing - and half the questions on this page are answered in ten seconds instead of two days. It is the cheapest intervention in this whole chapter and the only one that is genuinely a precondition for all the others.
What is below
Seven chapters, in the order you need them: first the failure catalogue as a reference, then five components against what is in it, and at the end the specialist - FunctionGemma, a 270-million-parameter model that does exactly one thing and therefore stands next to the large model, not in its place.
Every component comes with its price. There is no intervention on this page that only has upsides - constrained syntax costs reasoning, retrieval costs a preliminary stage, receipts cost effort at every interface. The question is never “good or bad” but “from when on is it worth it”.
Continue
The solutions start here - 7 chapters for Pro members.
- The failure catalogue - Twenty-two failure modes, sorted by layer, each with the marker that identifies it. The reference the following chapters point back to.
- Constrained syntax and the format tax - A grammar constraint makes every output well-formed. It guarantees the form and not the choice - and it costs reasoning exactly where you need it most.
- The toolbox and its size - Selection failures grow with the success of the project: the more the agent can do, the worse it chooses. Three interventions that keep the catalogue large and the context small.
- Repair instead of abort - Arithmetically the most effective group of components: turning a failure into an intermediate step raises the base of the chain calculation - without touching the model.
- Receipts instead of claims - Group E is the only one you cannot see in the text. Three levels with which a harness can check whether what the agent reports actually happened.
- FunctionGemma - the specialist - A 270-million-parameter model that does exactly one thing: turn a request and a tool list into a call. Its format, its wiring, its harness - and the three places where it usually breaks.
- Working together: the specialist next to the large model - The specialist does not replace the large model - it replaces one layer of its work. Three architectures, the question of who is responsible for what, and how you measure whether it paid off.