Seongwoo Choi / Journal

Run vLLM in Docker on an NVIDIA GPU, step by step

At a glanceInstall the NVIDIA Container Toolkit, prove the GPU works inside a container, start vLLM with docker run, then make it a Compose service that survives reboots.

Not testedWritten from the NVIDIA, Docker and vLLM documentation. The Compose file was syntax-checked; no container was started, as there was no GPU.

You need
A Linux machine with an NVIDIA GPU and its driver, Docker, and sudo rights.
You end up with
vLLM running as a container that restarts with the machine, answering on a local port.
Series · part 2 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 10 sections
  1. 1. The four layers you are stacking
  2. 2. Check the host first
  3. 3. Install the NVIDIA Container Toolkit
  4. 4. Prove a container can see the GPU
  5. 5. Start vLLM with one command
  6. 6. Check that it works
  7. 7. Turn it into a service with Compose
  8. 8. Day-to-day operation
  9. 9. When it does not start
  10. 10. What you can do now

Part 1 started vLLM straight from a Python environment. That is fine for a first try, but a server other people depend on should come back after a reboot, keep its dependencies in one place, and be upgraded by changing one line. Docker gives you that. This part builds the setup from an empty GPU machine to a service, and explains each flag so you can change it with confidence later.

1. The four layers you are stacking

A container cannot use a GPU by default. Four things have to line up, and most failures come from one missing layer:

4. vLLM imageCUDA libraries, PyTorch and vLLM
3. NVIDIA Container Toolkithands GPUs to containers
2. Docker Engineruns containers
1. NVIDIA driveron the host, shared by every container
GPUthe hardware
Each layer depends on the one below it. Check them from the bottom up.
LayerWhat it doesHow to check it
NVIDIA driver on the hostTalks to the card. Containers share the host's driver; they do not bring their own.nvidia-smi on the host prints a table
Docker EngineRuns containers.docker run --rm hello-world
NVIDIA Container ToolkitLets Docker hand GPUs to a container.nvidia-smi inside a container prints the same table
The vLLM imageCUDA libraries, PyTorch and vLLM, already matched to each other.The server answers on its port

Work from the top down. Do not start vLLM until the third check passes, or you will be debugging two things at once.

2. Check the host first

nvidia-smi
docker --version

nvidia-smi should list every GPU with its memory. Note the "Driver Version" in the top row; you will need it if the image complains about an old driver. If the command is missing, install the driver first. NVIDIA's guide recommends installing it with your distribution's package manager.

3. Install the NVIDIA Container Toolkit

On Ubuntu and Debian, NVIDIA's guide adds its package repository and then installs four packages. First the repository:

sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

Then the packages. The guide pins a version; the one below is what it showed on the day this was written, so copy the current number from the guide:

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
  nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

Installing is not enough. Docker also has to be told about the new runtime:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The first command edits /etc/docker/daemon.json on the host. The restart makes Docker read it. The guide has separate steps for RHEL-family systems and for rootless Docker.

4. Prove a container can see the GPU

This is NVIDIA's own test, and it is the most useful command in this article:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

A plain Ubuntu image has no NVIDIA software in it. If the familiar table appears, the toolkit injected the driver into the container, and everything below vLLM works. If you see could not select device driver "" with capabilities: [[gpu]], Docker does not know the NVIDIA runtime: repeat the two commands at the end of step 3.

5. Start vLLM with one command

Use a tiny model for the first run. It downloads in a minute or two and shows whether the stack works before you wait for a large one:

docker run --rm --name vllm \
  --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 127.0.0.1:8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-0.6B

Every part of that command matters:

PartWhy it is there
--runtime nvidia --gpus allGives the container every GPU. To give it only some, use --gpus device=0 or --gpus '"device=0,1"'.
-v ~/.cache/huggingface:...Keeps downloaded weights on the host. Without it, every new container downloads the model again.
--env "HF_TOKEN=$HF_TOKEN"Your Hugging Face token, needed for gated models. Harmless when empty.
-p 127.0.0.1:8000:8000Publishes the port on the machine's loopback address only. Plain -p 8000:8000 publishes it on every network interface.
--ipc=hostPyTorch shares data between processes through shared memory, and Docker's default is small. --shm-size is the alternative.
vllm/vllm-openai:latestThe official image. Its entry point is the server, so there is no vllm serve in the command.
Everything after the imageArguments to the server, the same ones vllm serve takes.

The loopback address is deliberate. An unprotected model server on a public interface gets found. Docker also writes its own firewall rules for published ports, so a host firewall such as ufw may not block a port you published on all interfaces. Part 5 covers how to open the server to other people properly.

6. Check that it works

The first start takes a while: the image pulls, the weights download, and the model loads. The server is ready when the log shows a line like Uvicorn running on http://0.0.0.0:8000. From a second terminal:

curl -fsS http://127.0.0.1:8000/health && echo ready
curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Reply with the word ready."}],
    "max_tokens": 50
  }'

The first command prints ready once the server is healthy. The second lists the model name that clients must send. The third returns a JSON object whose choices[0].message.content holds the answer.

Now run nvidia-smi on the host. The vLLM process will hold most of the card's memory even though this model is tiny. That is by design, not a leak. On NVIDIA GPUs, the documentation says vLLM measures the memory left after loading and warmup and gives it to the KV cache, the working memory for requests. Part 6 explains how to reason about that number.

7. Turn it into a service with Compose

A long docker run line is easy to mistype and hard to review. Put the same settings in a file. Create a folder, and inside it a file named compose.yaml:

services:
  vllm:
    image: vllm/vllm-openai:latest
    command:
      - --model
      - Qwen/Qwen3-0.6B
      - --served-model-name
      - my-model
    environment:
      HF_TOKEN: ${HF_TOKEN:-}
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    ports:
      - "127.0.0.1:8000:8000"
    ipc: host
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  vllm-cache:

Three things are new compared with the command:

  • The GPU block. Compose asks for GPUs under deploy.resources.reservations.devices. Docker's documentation says capabilities is required and the service fails to deploy without it. Use count for a number of GPUs or device_ids: ["0", "1"] for specific ones; the two cannot be combined.
  • restart: unless-stopped. Docker restarts the container if it exits or if the daemon restarts, which includes a reboot, but not after you stopped it yourself.
  • The vllm-cache volume. vLLM compiles parts of the model at startup. The documentation notes that each new container starts with an empty compile cache unless you mount a volume at ~/.cache/vllm, so this saves time from the second start onward.

--served-model-name gives clients a stable name. You can swap the model behind it later without changing every client.

If you need a token, put it in a file named .env next to compose.yaml as HF_TOKEN=hf_..., and keep that file out of version control. Then:

docker compose config --quiet && echo "file is valid"
docker compose up -d
docker compose logs -f vllm

The first line checks the file without starting anything. Stop following the log with Ctrl+C; the service keeps running. Repeat the checks from step 6, with my-model as the model name.

8. Day-to-day operation

TaskCommand
See whether it is runningdocker compose ps
Read the last 100 log linesdocker compose logs --tail 100 vllm
Apply a changed compose.yamldocker compose up -d
Stop the servicedocker compose down
Which vLLM version is this?curl -s http://127.0.0.1:8000/version

Pin the version. latest moves, and flags and defaults change between releases. Once the service works, ask it for its version and replace latest with that tag, in the form v0.x.y. Upgrade on purpose: change the tag, run docker compose pull, then docker compose up -d, and repeat your checks.

Run without root. The image runs as root by default. It also ships a user named vllm with UID 2000 in group 0. To use it, add user: "2000:0" to the service, mount the model cache at /home/vllm/.cache/huggingface and the compile cache at /home/vllm/.cache/vllm, and make the host directory writable by group 0.

Give it enough CPU. The documentation counts at least one API server process, one engine process and one worker per GPU, and recommends at least 2 + N physical cores for N GPUs. With hyperthreading that is twice as many vCPUs. A starved engine process shows up as a GPU that is never fully busy.

9. When it does not start

What you seeLikely causeWhat to do
could not select device driverDocker has no NVIDIA runtimeSteps 3 and 4
The log stops while downloadingSlow link, or a gated model without a valid tokenDownload first on the host with hf download <model>, as the troubleshooting page suggests, then start the container
Loading a large model takes very longWeights on a network file system, or the machine is swappingKeep the cache on a local disk and watch RAM with free -h
An out-of-memory error at startupThe model does not fit, or another process holds the cardCheck nvidia-smi, then lower --max-model-len or use a smaller or quantized model
Errors that mention shared memoryipc: host is missingAdd it back, or set shm_size
A CUDA or driver version errorThe host driver is older than the image expectsUpdate the driver, or set VLLM_ENABLE_CUDA_COMPATIBILITY=1, which the documentation supports on selected professional and datacenter GPUs
It runs, but another machine cannot connectThe port is bound to 127.0.0.1 on purposePart 5

When none of these fit, set the environment variable VLLM_LOGGING_LEVEL=DEBUG and read the log from the top. The first error is usually the real one; the lines after it are consequences.

10. What you can do now

  • Tell which of the four layers is broken from one command.
  • Read and change every flag in the vLLM docker run line.
  • Run vLLM as a Compose service that survives a reboot, with models and compiled artifacts kept between restarts.
  • Upgrade on purpose with a pinned image tag.

The next part replaces the test model with one worth using: serving Qwen with chat, tool calling and thinking.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

The commands follow the NVIDIA, Docker and vLLM documentation as read on November 12, 2026. No GPU was available, so no container was started for this article. The Compose file was checked with docker compose config, which validates the file but does not run it. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy