Hardware
Why does everyone talk about graphics cards when LLMs come up? Because memory is the hard limit: it decides which models you can even choose - and what every answer costs.
Basics
Free for everyone: the concept, the analogy, the why.
The distant brain from the LLM call lives somewhere in a data centre - or, since open weights, on your desk. Either way it runs on very real hardware. A basic feel for it pays off even if you never buy a GPU: it explains the prices, the limits and the model choices.
Why GPUs, of all things
At its core, an LLM call is the same calculation millions of times over: multiplying rows of numbers. Graphics cards were built for exactly that - thousands of small compute units performing the same operation simultaneously. A CPU can do it too, just with a handful of cores instead of thousands: the model then answers in minutes instead of seconds.
VRAM is the hard limit
More important than the GPU's speed is its memory (VRAM) - because the entire model has to fit inside. The rule of thumb: parameters × bytes per parameter. A 7B model at half precision (2 bytes) needs about 14 GB; a 70B model accordingly 140 GB - more than fits into any single consumer GPU.
On top of that comes the working memory for the conversation itself: the KV cache grows with every token in context. A long conversation with a big context window costs additional gigabytes. If a local model has ever died on you with "out of memory" during a long chat - that was it.
Quantisation: the trade-off
The rescue for small budgets is called quantisation: the weights are stored more coarsely - 8 bits instead of 16, or even 4. That halves (or quarters) the memory needed, at a small cost in quality. The surprising lesson from practice: a big model at 4 bits usually beats a small one at full precision. That's why quantised variants are the norm for local work, not the exception.
Does it have to be your own GPU?
No. The road to a running model has many lanes: Apple Silicon shares memory between CPU and GPU - a MacBook with 32 GB runs surprisingly large models. GPUs can be rented by the hour instead of bought. And the API is, in the end, nothing but "someone else operates the hardware" - paid per token instead of per graphics card.
Hardware doesn't decide whether your agent is smart - it decides which models are even on the menu and what every answer costs. VRAM is the currency: model size plus context has to fit.
For the agent itself, the block's pattern holds: the harness notices none of this. Whether a data centre or your MacBook sits behind the URL is a question of budget and privacy - not of architecture.
Going deeper
With a free account: experiments, quizzes and the deeper material.
Deep dive
For pro members: the depth for everyone who wants to actually build it.