Install Ollama and load your first model

Ollama is the simplest way to run large language models locally. It handles downloads, GPU detection, quantization, and exposes an OpenAI-compatible HTTP API on localhost:11434. Exactly what you need for the Playground and Tutorial phases of this project if you don't want to use an Anthropic key.

By the end of this tutorial you'll have:

• Installed Ollama and have it running as a background service
• Downloaded a model (Llama 3.2 or Mistral)
• Successfully chatted with it from the terminal
• Tested the HTTP endpoint with curl

Step 1 - Install Ollama

The installation differs per OS but is always either a one-liner or a regular installer.

macOS and Windows: Download the native installer from ollama.com/downloadExternal - Opens in a new tab. On macOS it's a .dmg, on Windows an .exe. Double-click, click through, done - Ollama starts automatically as a background service.

Linux: Installer script straight from the official source:

The script detects your distro, installs to /usr/local/bin/ollama, and sets up a systemd service.

Step 2 - Check the background service

Ollama runs as a daemon on port 11434. That's where your code will send requests later.

If you see nothing or a connection-refused error:

• macOS: Launch the Ollama app from the dock (the llama icon in the menu bar)
• Windows: Search "Ollama" in the Start menu and launch it
• Linux: systemctl status ollama - if inactive: systemctl start ollama

Step 3 - Pull your first model

Ollama has a model catalogue at ollama.com/libraryExternal - Opens in a new tab. To get started we recommend Llama 3.2 in the 3B variant - fast, small (~2 GB), and good enough for tool-use demos.

You'll see a progress bar pulling several layers in parallel. On a normal home connection this takes 1-3 minutes.

Step 4 - Chat in the terminal

Quickest way to test the model works:

Use /bye or Ctrl+D to leave the chat. The model stays loaded - the next ollama run starts instantly.

Step 5 - Test the HTTP endpoint

This is exactly the endpoint the Playground will hit. Make sure it answers:

The response is a JSON object with response, total_duration (nanoseconds), prompt_eval_count, and eval_count (tokens). Exactly the fields we need later to surface token usage.

Troubleshooting

"Error: model requires more system memory" - the model is too large for your RAM. Try a smaller variant (:3b or :1b) or stronger quantization (:q4_0).

Very slow even though I have a GPU - ollama ps shows whether the model actually runs on the GPU (PROCESSOR should be 100% GPU). If not: check CUDA drivers / ROCm setup.

Port 11434 in use - OLLAMA_HOST=0.0.0.0:11500 ollama serve starts on a different port. You'll then need to set the same port as LLM_BASE_URL in the Playground/code.