What Is Ollama? How It Works, Models and Use Cases

Ollama has become the standard way to run open-weight language models on your own hardware, but the essentials are spread across an install script, a Modelfile syntax and an HTTP API. This tutorial explains what the runtime actually does when you type ollama run, which models fit which hardware, and how to expose the server to other services.

What is Ollama?

Ollama is an open source runtime that packages open-weight language models such as Llama 3.1, Gemma 3 and Qwen 2.5 into single pullable images and serves them through a local HTTP API on port 11434.

The project wraps a llama.cpp-based inference engine, a model registry client and a small REST server into one binary. Models are stored as content-addressed layers, similar to container images, so two tags that share the same weights do not duplicate the download. Everything runs on your host: no request leaves the machine unless you route it somewhere yourself.

Use Ollama when you want local inference with minimal operational overhead. Use vLLM when you need to saturate a GPU with many concurrent requests.

How does Ollama work?

Ollama runs as a background server that accepts requests, loads the requested model into GPU VRAM or system RAM, generates tokens with a llama.cpp-based runtime and streams the result back to the client. The CLI (ollama run) is only one of several clients.

graph TD
  A["CLI or REST client"] --> B["Ollama server on :11434"]
  B --> C{"Model already loaded?"}
  C -->|"No"| D["Read layers from the model store"]
  D --> E["Load weights into VRAM or RAM"]
  C -->|"Yes"| F["Reuse the loaded model"]
  E --> F
  F --> G["Generate tokens"]
  G --> H["Stream response to the client"]
  F --> I["Unload after OLLAMA_KEEP_ALIVE"]

Two details matter in production. First, the initial request after a cold start includes the load time, which for an 8B model is a few seconds on NVMe and considerably longer on network storage. Second, a loaded model stays resident for five minutes by default and is then evicted. Set OLLAMA_KEEP_ALIVE if your traffic pattern is bursty.

Models live under ~/.ollama/models for a user-level install and under /usr/share/ollama/.ollama/models when Ollama runs as a systemd service under the ollama user.

Prerequisites

• A Linux host with a 64-bit kernel, or macOS 12+ / Windows 10+
• At least 8 GB RAM for 7B/8B models in 4-bit quantization
• Roughly 20 GB free disk space for the first few models
• curl and a user account with sudo rights
• For GPU acceleration: an NVIDIA driver with CUDA support (compute capability 5.0 or higher) or a supported AMD ROCm setup

CPU inference works but delivers single-digit tokens per second on most server CPUs. If you plan to serve interactive workloads, run Ollama on a GPU instance for LLM inference and verify with nvidia-smi that the driver is loaded before you install.

Install Ollama and run a model

Install the binary and the systemd unit with the official script:

$ curl -fsSL https://ollama.com/install.sh | sh
$ ollama --version

Start an interactive session. The first invocation pulls the model layers, subsequent ones start from the local store:

$ ollama run llama3.1:8b
>>> Summarise what a TCP listen backlog does in two sentences.

Exit the session with /bye. To download without starting a chat, use ollama pull:

$ ollama pull qwen2.5-coder:7b
$ ollama list
$ ollama rm mistral:7b

ollama show llama3.1:8b prints the architecture, parameter count, quantization and the default context length of a model. That is the fastest way to check what you actually downloaded.

Which models can Ollama run?

Ollama runs any model published in GGUF format, including the curated library tags and arbitrary GGUF files imported from Hugging Face. Tags follow the pattern name:parameters, for example gemma3:12b.

Sizes are approximate and change when upstream tags are rebuilt. Check the current value with ollama show <model> after pulling. To import a GGUF file that is not in the library, reference it in a Modelfile with FROM /path/to/model.gguf.

How much VRAM does a model need?

A 4-bit quantized model needs roughly 0.6 GB of memory per billion parameters plus context overhead, so an 8B model fits in about 6 GB of VRAM and a 70B model needs about 42 GB. Longer context windows add several hundred megabytes for the KV cache.

If the model does not fit entirely in VRAM, Ollama offloads the remaining layers to system RAM and throughput drops sharply. Prefer a smaller model or a lower quantization over a partial offload.

Use the REST API

Ollama exposes /api/generate, /api/chat, /api/embed and /api/tags on port 11434, plus an OpenAI-compatible endpoint at /v1/chat/completions for existing SDK code.

$ curl http://localhost:11434/api/chat -d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"List three causes of high iowait."}],"stream":false}'

Set "stream": true (the default) to receive newline-delimited JSON chunks instead of one response object. Embeddings for a retrieval pipeline use a dedicated endpoint:

$ curl http://localhost:11434/api/embed -d '{"model":"nomic-embed-text","input":"backup rotation policy"}'

The OpenAI-compatible route lets you point an existing client at Ollama by changing the base URL. The API key is ignored but most SDKs require a non-empty value:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Explain the difference between RSS and PSS."}],
)
print(resp.choices[0].message.content)

Customize a model with a Modelfile

A Modelfile declares a base model with FROM, sets sampling and context parameters with PARAMETER and pins a system prompt with SYSTEM. ollama create then builds a reusable named model that behaves identically for every caller.

FROM llama3.1:8b
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
PARAMETER top_p 0.9
SYSTEM """You answer as a Linux systems engineer. Give the command first, then one sentence of explanation. Never invent flags."""

Build and use it:

$ ollama create ops-assistant -f /opt/ollama/Modelfile
$ ollama run ops-assistant "How do I list open sockets with their PIDs?"

• num_ctx sets the context window in tokens. Larger values increase KV cache memory linearly.
• temperature controls randomness. Use 0.0 to 0.3 for extraction and code tasks.
• SYSTEM replaces the base model's default system prompt for every request that does not send its own.

Run Ollama as a service

The install script creates a systemd unit called ollama.service. Configure it with a drop-in rather than editing the unit file directly:

$ sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_MODELS=/data/ollama/models"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_NUM_PARALLEL=4"

Apply the change with sudo systemctl daemon-reload && sudo systemctl restart ollama. Ollama has no built-in authentication, so binding it to 0.0.0.0 publishes an unauthenticated inference endpoint. Put a reverse proxy with TLS and an auth layer in front of it, or keep the listener on loopback and reach it through an SSH tunnel or a private network.

Small models and embedding workloads run acceptably without a GPU, which makes a standard scalable Linux Cloud-VPS a reasonable host for a retrieval index or a batch summarisation job. Move to GPU instances once interactive latency matters. Point OLLAMA_MODELS at a separate volume so the model store survives a rebuild of the root filesystem.

Use cases

• Local development against an LLM API. Swap the base URL of an OpenAI-compatible client to test prompt changes without per-token cost.
• Retrieval-augmented generation. Generate embeddings with nomic-embed-text, store them in a vector database and pass retrieved chunks to a chat model.
• Code assistance in the editor. Editor plugins that speak the Ollama API can use qwen2.5-coder against a workstation or a shared GPU host.
• Data that must not leave the network. Log excerpts, ticket contents and internal documentation stay on your infrastructure.
• Batch classification. Loop over records with /api/generate and a low temperature for deterministic output.

Verify the installation

Check the service state and the model placement:

$ systemctl is-active ollama
$ ollama ps
NAME             ID           SIZE     PROCESSOR    UNTIL
llama3.1:8b      42182419e950 6.2 GB   100% GPU     4 minutes from now

A PROCESSOR value of 100% GPU confirms full offload. A split such as 62% GPU / 38% CPU means the model does not fit in VRAM. Confirm the API answers:

$ curl -s http://localhost:11434/api/tags | head -c 200

Troubleshooting

curl: (7) Failed to connect to localhost port 11434. The server is not running or is bound elsewhere. Check systemctl status ollama and ss -tlnp | grep 11434. If you set OLLAMA_HOST to a specific address, clients must use that address, not localhost.

The model runs on CPU although a GPU is present. Run nvidia-smi to confirm the driver is loaded, then check journalctl -u ollama -n 50 for the GPU detection lines. The service needs a restart after a driver update, and a container-based install needs the NVIDIA container toolkit.

The process is killed during loading. The kernel OOM killer terminated the model because system RAM was insufficient. Check dmesg -T | tail, then choose a smaller parameter count or a lower quantization tag.

Wrap-up

Ollama gives you a single binary that pulls quantized models, keeps them resident and serves them over an HTTP API compatible with common client SDKs. Size the hardware from the quantization table before you pull a model, pin behaviour in a Modelfile instead of repeating system prompts in every call, and never expose port 11434 without an authenticating proxy in front of it.

Read next

• Calling the Ollama API from Your Own Applications
• Connecting Ollama to MCP Servers: Tools for Local Models
• Expose Ollama on the Network: Port, Bind Address and Hardening
• Install Ollama on Ubuntu 24.04 and Run It as a Service
• Use a Local Coding Model with Ollama in Your IDE
• Managing Ollama Models: Updates, Cleanup and Disk Space
• Ollama vs LM Studio vs llama.cpp: Which Tool for What?
• Run Ollama in a Docker Container
• Set Up Open WebUI as a Front End for Ollama