Run Ollama in a Docker Container

Ollama runs large language models locally and exposes them over an HTTP API on port 11434. Running it in a container keeps the inference engine, its CUDA libraries and the model store separate from the host, so upgrades and rollbacks come down to changing an image tag.

This tutorial covers the full container setup: persistent model storage, GPU passthrough, the API surface, and a Compose stack that puts a web UI in front of the server.

What is Ollama in Docker?

Ollama in Docker is the official ollama/ollama image, which bundles the Ollama server, its GGUF inference engine and the GPU runtime libraries, and serves the model API on port 11434 inside the container. Models live under /root/.ollama in the container filesystem.

The image ships in two hardware variants. Pick the one that matches your accelerator:

Without a GPU the same image still works. It falls back to CPU inference, which is usable for models in the 1B to 8B range and slow beyond that.

Prerequisites

• A Linux host with Docker Engine 24.0 or newer. The commands below were tested on Ubuntu 24.04 with Docker 27.
• A user in the docker group, or sudo for every Docker command.
• At least 8 GB RAM for 7B/8B models in 4-bit quantisation, 16 GB for comfortable headroom.
• Roughly 5 GB of free disk space per model. llama3.2:3b needs about 2 GB, llama3.1:8b about 4.7 GB.
• Optional: an NVIDIA GPU with driver 535 or newer plus the NVIDIA Container Toolkit.

If you are building this on a scalable Cloud-VPS, size the instance by model rather than by CPU count. Memory and disk are the limiting factors for local inference, not core count.

Start the Ollama container

Start the server with a named volume for the model store and publish the API port on the loopback interface only:

$ docker run -d --name ollama -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --restart unless-stopped ollama/ollama

What each flag does:

• -v ollama:/root/.ollama keeps downloaded models in a named Docker volume. Without it, every docker rm throws away gigabytes of downloads.
• -p 127.0.0.1:11434:11434 binds the API to localhost. Ollama has no authentication layer, so a bare -p 11434:11434 on a public host exposes an open inference endpoint to the internet.
• --restart unless-stopped brings the container back after a reboot or a Docker daemon restart.

Check that the server is up:

$ docker logs ollama | tail -n 5

The log ends with a line containing Listening on [::]:11434.

Pull and run a model

The Ollama CLI is part of the same image, so run it inside the container with docker exec:

$ docker exec -it ollama ollama pull llama3.2:3b
$ docker exec -it ollama ollama list

ollama list prints the local catalogue with name, digest, size and modification time. To start an interactive prompt:

$ docker exec -it ollama ollama run llama3.2:3b

Exit the session with /bye. For a one-shot query, append the prompt directly:

$ docker exec ollama ollama run llama3.2:3b "List three systemd journal filter flags."

Remove a model

Models accumulate quickly. Delete one and free the space in the volume:

$ docker exec ollama ollama rm llama3.2:3b

How do you give the container GPU access?

You give the Ollama container GPU access by installing the NVIDIA Container Toolkit on the host and starting the container with --gpus=all; Ollama then loads its CUDA runners and offloads model layers into VRAM instead of running on the CPU.

Install the toolkit and register the runtime with Docker:

$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
$ curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
$ sudo apt update && sudo apt install -y nvidia-container-toolkit
$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker

Recreate the container with GPU access. The volume survives, so no model is downloaded twice:

$ docker rm -f ollama
$ docker run -d --gpus=all --name ollama -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --restart unless-stopped ollama/ollama

Confirm that the GPU is visible inside the container and actually used:

$ docker exec ollama nvidia-smi --query-gpu=name,memory.total --format=csv
$ docker exec ollama ollama ps

In the ollama ps output, the PROCESSOR column reads 100% GPU when all layers fit in VRAM. A value such as 43%/57% CPU/GPU means the model is partially offloaded, which is correct behaviour but noticeably slower. For AMD hardware, use the ollama/ollama:rocm tag and pass --device /dev/kfd --device /dev/dri instead of --gpus=all.

Query the API

The API listens on port 11434 and offers both the native Ollama endpoints under /api and an OpenAI-compatible endpoint under /v1. A non-streaming completion:

$ curl -s http://127.0.0.1:11434/api/generate -d '{"model":"llama3.2:3b","prompt":"Name the default Ollama port.","stream":false}' | jq -r .response

The OpenAI-compatible route accepts the same payload shape as the Chat Completions API, which lets existing SDK clients point at the container without code changes:

$ curl -s http://127.0.0.1:11434/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"llama3.2:3b","messages":[{"role":"user","content":"ping"}]}' | jq -r '.choices[0].message.content'

The request path through the stack looks like this:

graph TD
  A["Client or SDK"] --> B["Host 127.0.0.1:11434"]
  B --> C["Container ollama"]
  C --> D["Volume ollama:/root/.ollama"]
  C --> E{"GPU passed in?"}
  E -->|yes| F["CUDA runner, layers in VRAM"]
  E -->|no| G["CPU runner, layers in RAM"]

Run Ollama with Docker Compose

A Compose file makes the setup reproducible and lets you attach a web UI on the same network. Save it as /opt/ollama/docker-compose.yml:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      OLLAMA_KEEP_ALIVE: "10m"
      OLLAMA_NUM_PARALLEL: "2"
      OLLAMA_MAX_LOADED_MODELS: "1"
  webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      OLLAMA_BASE_URL: "http://ollama:11434"
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - ollama
volumes:
  ollama:
  open-webui:

The web UI reaches the server at http://ollama:11434 over the Compose network, so the published port is only needed for access from the host. Start the stack:

$ cd /opt/ollama && docker compose up -d

The environment variables control resource behaviour:

On a host with limited VRAM, set OLLAMA_MAX_LOADED_MODELS to 1. Otherwise a second model request can evict the first mid-conversation and trigger a reload on every turn.

To add a GPU under Compose, declare it in the ollama service:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Verification

Run both checks after any change to the container definition:

$ docker exec ollama ollama ps
$ curl -s http://127.0.0.1:11434/api/tags | jq '.models[].name'

The first command lists models currently held in memory with their processor split. The second returns the installed models as JSON, for example:

"llama3.2:3b"

An empty models array means the volume is mounted but no model has been pulled yet.

Troubleshooting

curl: (7) Failed to connect to 127.0.0.1 port 11434 — the container is not running or the port mapping is missing. Check docker ps --filter name=ollama and inspect the mapping with docker port ollama.

Inference runs on CPU despite a GPU in the host — the container was started without --gpus=all, or the NVIDIA Container Toolkit is not registered. Run docker logs ollama | grep -i "library="; a line with library=cpu confirms the fallback. Recreate the container after sudo nvidia-ctk runtime configure --runtime=docker.

no space left on device during a pull — the Docker data root is full. Check where the model store actually lives with docker volume inspect ollama and free space by removing unused models and dangling images with docker image prune.

Model reloads on every request — OLLAMA_KEEP_ALIVE is too short or memory pressure evicts the model. Raise the value to 30m and keep OLLAMA_MAX_LOADED_MODELS at 1.

Wrap-up

The container now serves models on port 11434 with a persistent volume behind it, and the Compose file makes that state reproducible on any other host. Keep the API bound to 127.0.0.1 and put a reverse proxy with authentication in front of it before exposing it beyond the machine, since Ollama itself performs no access control.

As a next step, pin the image to a specific tag instead of latest so a docker compose pull cannot change the inference engine underneath a running workload.

More on Docker

• Encrypted Docker Volume Backups with Restic
• Run Nextcloud with Docker Compose
• Shrink Docker Images with Multi-Stage Builds