Guides by Editorial Team

How to Run Ollama in Docker So the GPU Still Works

Containerize Ollama without losing GPU acceleration: the NVIDIA toolkit setup, the AMD device flags, the volume that keeps your models, and how to check.

Put Ollama in a container and the same docker run line can either use your graphics card or ignore it completely. The difference is a single flag, but the flag only works if the host was prepared first. On NVIDIA that means installing and wiring up the NVIDIA container runtime. On AMD there is no --gpus equivalent, so the card comes in as two device nodes instead. The rest is housekeeping: a volume so the models survive, a published port so you can reach the API, and a short list of things to check when the container starts but nothing gets faster.

TL;DR

  • One flag separates CPU from GPU. Ollama’s CPU-only command is docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama. The GPU command is that same line with --gpus=all (Ollama Docker docs).
  • --gpus is not a Docker feature on its own. It depends on the NVIDIA container runtime being configured for Docker with sudo nvidia-ctk runtime configure --runtime=docker, followed by a Docker restart (NVIDIA install guide, docker run reference).
  • AMD uses device nodes, not --gpus. Ollama’s docs use a separate rocm image tag with --device /dev/kfd --device /dev/dri (Ollama Docker docs), and AMD’s own container guide adds --security-opt seccomp=unconfined so the GPU can be memory mapped (ROCm Docker).
  • The volume is the point. -v ollama:/root/.ollama is what keeps every model you pull. Without it the container starts empty each time (Docker volumes).
  • Check the card before the container. Ollama documents the floor as “Ollama supports Nvidia GPUs with compute capability 5.0+ and driver version 550 and newer,” with capability 5.0 through 6.2 needing driver 570 or newer (Ollama GPU docs).
  • Docker is the wrong box on a Mac. Ollama’s FAQ states that GPU acceleration is not available for macOS Docker Desktop, because GPU passthrough and emulation are not there (Ollama FAQ).

The three commands that matter

The shortest version is three lines, one per hardware situation. On a machine with no usable GPU, or when you are deliberately testing CPU behaviour:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

On NVIDIA, add --gpus=all:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

On AMD, switch to the ROCm image and pass the two device nodes:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

All three are from Ollama’s own Docker documentation (Ollama Docker docs). Note what they share: the -v ollama:/root/.ollama volume and the -p 11434:11434 port mapping. That port mapping is what makes localhost:11434 on your machine reach the Ollama server running inside the container rather than nothing at all.

Once it is up, Ollama’s docs show driving a model through the running container directly:

docker exec -it ollama ollama run llama3.2

That step is worth keeping even on a working setup, because it separates “the container is broken” from “the GPU is not being used”.

NVIDIA: the toolkit is the part people skip

--gpus is documented in the docker run reference as requiring the NVIDIA container runtime, and with no value Docker exposes every GPU it can see (docker run reference). A narrower form selects specific devices, including a comma-separated list such as device=0,2, and individual GPUs can also be named by UUID. If the runtime is not installed and configured, --gpus=all simply fails, and that failure is the reason this step matters more than the flag.

NVIDIA’s own install guide is the primary source for this. On Debian and Ubuntu it configures the production repository, then installs the toolkit packages; the guide pins version 1.20.1-1 in its examples (NVIDIA install guide):

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

RHEL, CentOS, Fedora and Amazon Linux use dnf against the same repository setup. Both paths end the same way, and this is the step people miss:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

That command updates /etc/docker/daemon.json so Docker can invoke the NVIDIA runtime, and the guide has you restart Docker immediately afterwards (NVIDIA install guide). Do the restart before you create the container, since the container starts under the daemon that is running at that moment.

Docker’s reference also documents a newer path: the Container Device Interface, where GPUs are exposed as --device devices rather than through --gpus. It is Linux-only and enabled by default beginning with Docker Engine 28.3.0 (docker run reference). Ollama’s documented commands predate that route and use --gpus, which is the safer thing to copy.

AMD: device nodes, not a flag

There is no AMD equivalent to --gpus, so Ollama’s documentation takes a different route: a rocm image tag plus two explicit devices (Ollama Docker docs). AMD’s ROCm documentation explains what those two are for, and its minimal GPU-access command is:

docker run --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined <image>

/dev/kfd supplies the compute interface and /dev/dri exposes the GPUs’ Direct Rendering Interface; AMD notes that /dev/kfd is always required for compute, and that the seccomp option is optional but supports memory mapping (ROCm Docker). The same page gives a Compose equivalent, which is the easier form to keep in version control:

services:
  my-service:
    image: <image>
    devices:
      - /dev/kfd
      - /dev/dri
    security_opt:
      - seccomp=unconfined

One prerequisite applies to every AMD container, containerized or not: “Docker containers share the kernel with the host OS. Therefore, the ROCm kernel-mode driver (amdgpu-dkms) must be installed on the host” (ROCm Docker). Ollama’s GPU documentation adds a host-side SELinux gotcha for RHEL-family systems: “SELinux can prevent containers from accessing the AMD GPU devices,” with sudo setsebool container_use_devices=1 as the documented fix (Ollama GPU docs).

One case that does not need any of this: the standard Ollama image already includes Vulkan, and it activates when GPU devices are accessible. It can be switched off with OLLAMA_VULKAN=0, and specific devices selected with GGML_VK_VISIBLE_DEVICES=<ids> (Ollama Docker docs).

The volume is the reason to containerize

Models are large, and pulling them is slow. That is the whole argument for a named volume, and the reason Ollama’s commands all carry -v ollama:/root/.ollama. Docker creates a named volume automatically the first time a docker run references it, and on Linux the data lands under /var/lib/docker/volumes, visible in the output of docker volume inspect (Docker volumes). The container can be deleted and recreated indefinitely and the models stay put.

The commands that manage it are the standard set - docker volume create, docker volume ls, docker volume inspect, docker volume rm - plus docker volume prune, which deletes volumes not currently attached to a container. That last one is worth reading twice, because for an Ollama setup it is the difference between reclaiming a few hundred megabytes and re-downloading tens of gigabytes of weights.

There is a second reason to keep the volume: it pins the model path inside the container, whatever the host’s own layout is. A native Linux install puts models under /usr/share/ollama/.ollama/models by default, with OLLAMA_MODELS pointing elsewhere if you want a different path; macOS and Windows default to a per-user path instead (Ollama FAQ). In a container, the volume mount supersedes all of that. If you would rather point at a directory you can see from the host, swap the named volume for a bind mount of your own.

Environment variables cross the same boundary with -e. Two matter most on a server: OLLAMA_KEEP_ALIVE, which the FAQ describes as keeping models in memory because “By default models are kept in memory for 5 minutes before being unloaded,” and OLLAMA_NUM_PARALLEL, which defaults to 1 and caps how many simultaneous requests each loaded model handles (Ollama FAQ). A container that serves a team will want both raised, and setting them on the host will not reach the process inside the container.

Check whether the GPU is actually being used

When a containerized Ollama feels slow, work down this list in order. It runs from cheapest check to most expensive.

First, confirm the host driver sees the card at all, outside Docker. Ollama’s GPU documentation points to nvidia-smi -L for discovering GPU UUIDs, and notes that CUDA_VISIBLE_DEVICES restricts which devices are used, with -1 forcing CPU execution (Ollama GPU docs). On AMD hosts, the equivalent is rocminfo, and AMD’s container documentation states that rocminfo and amd-smi list can be run inside the container to check which exposed GPUs are visible (ROCm Docker).

Second, check the card clears Ollama’s floor. The documented minimum is compute capability 5.0 with driver 550 or newer, and the GPU page lists capability 5.0 through 6.2 as needing driver 570 or newer (Ollama GPU docs). A container cannot fix a card the runtime does not support. On AMD, Linux needs the AMD ROCm v7 driver (Ollama GPU docs).

Third, check the container was started with the right flags and the right image tag. This is where most failures live: the standard ollama/ollama image on an AMD card, or --gpus=all on a host where the container toolkit was never configured. Ollama’s Docker docs call out one specific variant: on NVIDIA Jetson hardware, automatic version discovery can fail, and either JETSON_JETPACK=5 or JETSON_JETPACK=6 has to be set as appropriate (Ollama Docker docs).

Finally, remember that docker run flags only apply at creation time. Adding --gpus=all to a command line does nothing to a container that is already running; the container has to be removed and recreated.

When Docker is the wrong tool

Docker earns its keep when Ollama runs on a server, a homelab box, or a machine you manage remotely: one process to start, one port to publish, one volume to back up. It is the wrong shape in two common cases.

The first is a Mac. Ollama’s FAQ states that GPU acceleration is supported in Linux containers and in Windows with WSL2, where it requires the NVIDIA container toolkit, and that it is unavailable for macOS Docker Desktop because GPU passthrough and emulation are not available there (Ollama FAQ). On that hardware a container adds a layer between you and the card and takes the card away.

The second is a single-user laptop where you want the native runtime’s GPU path and nothing else. A VM is the better isolation boundary there.

Two network details are worth knowing before you point anything at it. If the host reaches the internet through a proxy, the FAQ shows passing the proxy address into the container with an HTTPS_PROXY environment variable, and notes that proxying can instead be configured in Docker itself. And for a self-signed HTTPS certificate, the FAQ describes deriving an image that contains the certificate and runs update-ca-certificates (Ollama FAQ). Both are configuration changes that are painful to discover after the fact.

What this means

Containerizing Ollama is mostly a packaging exercise with one hardware dependency, and the dependency is the NVIDIA container runtime. On AMD the equivalent dependency is the kernel driver on the host, which a container cannot supply. Treat those two as the setup step and everything after it - the volume, the port, the environment variables - as ordinary configuration you would write down once.

The habit worth building is checking the card before the container. When a containerized model feels slow, two causes are worth ruling out first because both are visible from host-side commands alone: a card below Ollama’s documented minimum, and a toolkit installed but never wired into Docker’s daemon config.

The bottom line

Run docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama on NVIDIA after configuring the container runtime with nvidia-ctk runtime configure --runtime=docker and restarting Docker. Run the ollama/ollama:rocm image with --device /dev/kfd --device /dev/dri on AMD, and fix SELinux with setsebool container_use_devices=1 if your host uses it. Keep the ollama volume in every command or you will re-download your models every time. And verify with a real model through docker exec -it ollama ollama run llama3.2 rather than assuming the card is being used because the container started cleanly.