Inference engines

Download an open-weights model and you get … a file. A few gigabytes of learned numbers that do nothing on their own. Between that file and the familiar "JSON in, JSON out" there's still a program missing: the inference engine.

What the engine does

It loads the weights into memory, accepts your request - and produces the answer token by token: from the text so far, the model computes the most likely next piece, that piece gets appended, and the whole thing starts over until stop_reason is reached. That's why LLM answers trickle in word by word: it is a loop.

To keep that fast, engines rely on tricks, two of which come up again and again. The KV cache remembers intermediate results for text already read, so each new token doesn't recompute everything. And batching serves many requests on one GPU at once instead of politely queueing them.

The usual suspects

• Ollama - the friendliest entry: one command, model runs locally, API included.
• llama.cpp - the endurance runner: written in C++, runs on almost anything from servers to MacBooks; the core of many other tools (Ollama builds on it too).
• vLLM - the serving professional: built for throughput when one model has to serve many users at once.

And the big APIs? There, too, an inference engine answers you - just one owned by the provider, running behind load balancers in their data centre. The difference is the operator, not the principle.

One API, many engines

Conveniently, almost all engines have settled on the same dialect: OpenAI-compatible endpoints. Whether it's Ollama on localhost, vLLM on your server or a cloud provider - the request looks (almost) the same:

POST http://localhost:11434/v1/chat/completions

{
  "model": "hermes3",
  "messages": [
    { "role": "user", "content": "Who are you?" }
  ]
}

For your harness that means: switching engines = switching the base URL. Nothing more.

Keep this picture: the model is the record, the engine the record player. The record holds the music, but only the player makes it audible - and the same player plays any record in the right format.

To the harness, both are invisible: it only sees a URL that answers calls. That's exactly why you can develop locally against Ollama today and point at a cloud provider tomorrow without changing a single line of your agent.