Monitor vLLM with Prometheus, Grafana and DCGM in Docker
One Compose file for vLLM, node exporter, the NVIDIA DCGM exporter, Prometheus and Grafana, with the queries and alerts for CPU, RAM, GPU and the model server.
At a glanceInstall the NVIDIA Container Toolkit, prove the GPU works inside a container, start vLLM with docker run, then make it a Compose service that survives reboots.
Not testedWritten from the NVIDIA, Docker and vLLM documentation. The Compose file was syntax-checked; no container was started, as there was no GPU.
Part 1 started vLLM straight from a Python environment. That is fine for a first try, but a server other people depend on should come back after a reboot, keep its dependencies in one place, and be upgraded by changing one line. Docker gives you that. This part builds the setup from an empty GPU machine to a service, and explains each flag so you can change it with confidence later.
A container cannot use a GPU by default. Four things have to line up, and most failures come from one missing layer:
| Layer | What it does | How to check it |
|---|---|---|
| NVIDIA driver on the host | Talks to the card. Containers share the host's driver; they do not bring their own. | nvidia-smi on the host prints a table |
| Docker Engine | Runs containers. | docker run --rm hello-world |
| NVIDIA Container Toolkit | Lets Docker hand GPUs to a container. | nvidia-smi inside a container prints the same table |
| The vLLM image | CUDA libraries, PyTorch and vLLM, already matched to each other. | The server answers on its port |
Work from the top down. Do not start vLLM until the third check passes, or you will be debugging two things at once.
nvidia-smi
docker --version
nvidia-smi should list every GPU with its memory. Note the "Driver Version" in the top row; you will need it if the image complains about an old driver. If the command is missing, install the driver first. NVIDIA's guide recommends installing it with your distribution's package manager.
On Ubuntu and Debian, NVIDIA's guide adds its package repository and then installs four packages. First the repository:
sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
Then the packages. The guide pins a version; the one below is what it showed on the day this was written, so copy the current number from the guide:
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
Installing is not enough. Docker also has to be told about the new runtime:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
The first command edits /etc/docker/daemon.json on the host. The restart makes Docker read it. The guide has separate steps for RHEL-family systems and for rootless Docker.
This is NVIDIA's own test, and it is the most useful command in this article:
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
A plain Ubuntu image has no NVIDIA software in it. If the familiar table appears, the toolkit injected the driver into the container, and everything below vLLM works. If you see could not select device driver "" with capabilities: [[gpu]], Docker does not know the NVIDIA runtime: repeat the two commands at the end of step 3.
Use a tiny model for the first run. It downloads in a minute or two and shows whether the stack works before you wait for a large one:
docker run --rm --name vllm \
--runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
-p 127.0.0.1:8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-0.6B
Every part of that command matters:
| Part | Why it is there |
|---|---|
--runtime nvidia --gpus all | Gives the container every GPU. To give it only some, use --gpus device=0 or --gpus '"device=0,1"'. |
-v ~/.cache/huggingface:... | Keeps downloaded weights on the host. Without it, every new container downloads the model again. |
--env "HF_TOKEN=$HF_TOKEN" | Your Hugging Face token, needed for gated models. Harmless when empty. |
-p 127.0.0.1:8000:8000 | Publishes the port on the machine's loopback address only. Plain -p 8000:8000 publishes it on every network interface. |
--ipc=host | PyTorch shares data between processes through shared memory, and Docker's default is small. --shm-size is the alternative. |
vllm/vllm-openai:latest | The official image. Its entry point is the server, so there is no vllm serve in the command. |
| Everything after the image | Arguments to the server, the same ones vllm serve takes. |
The loopback address is deliberate. An unprotected model server on a public interface gets found. Docker also writes its own firewall rules for published ports, so a host firewall such as ufw may not block a port you published on all interfaces. Part 5 covers how to open the server to other people properly.
The first start takes a while: the image pulls, the weights download, and the model loads. The server is ready when the log shows a line like Uvicorn running on http://0.0.0.0:8000. From a second terminal:
curl -fsS http://127.0.0.1:8000/health && echo ready
curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "Reply with the word ready."}],
"max_tokens": 50
}'
The first command prints ready once the server is healthy. The second lists the model name that clients must send. The third returns a JSON object whose choices[0].message.content holds the answer.
Now run nvidia-smi on the host. The vLLM process will hold most of the card's memory even though this model is tiny. That is by design, not a leak. On NVIDIA GPUs, the documentation says vLLM measures the memory left after loading and warmup and gives it to the KV cache, the working memory for requests. Part 6 explains how to reason about that number.
A long docker run line is easy to mistype and hard to review. Put the same settings in a file. Create a folder, and inside it a file named compose.yaml:
services:
vllm:
image: vllm/vllm-openai:latest
command:
- --model
- Qwen/Qwen3-0.6B
- --served-model-name
- my-model
environment:
HF_TOKEN: ${HF_TOKEN:-}
volumes:
- ${HOME}/.cache/huggingface:/root/.cache/huggingface
- vllm-cache:/root/.cache/vllm
ports:
- "127.0.0.1:8000:8000"
ipc: host
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
vllm-cache:
Three things are new compared with the command:
deploy.resources.reservations.devices. Docker's documentation says capabilities is required and the service fails to deploy without it. Use count for a number of GPUs or device_ids: ["0", "1"] for specific ones; the two cannot be combined.restart: unless-stopped. Docker restarts the container if it exits or if the daemon restarts, which includes a reboot, but not after you stopped it yourself.vllm-cache volume. vLLM compiles parts of the model at startup. The documentation notes that each new container starts with an empty compile cache unless you mount a volume at ~/.cache/vllm, so this saves time from the second start onward.--served-model-name gives clients a stable name. You can swap the model behind it later without changing every client.
If you need a token, put it in a file named .env next to compose.yaml as HF_TOKEN=hf_..., and keep that file out of version control. Then:
docker compose config --quiet && echo "file is valid"
docker compose up -d
docker compose logs -f vllm
The first line checks the file without starting anything. Stop following the log with Ctrl+C; the service keeps running. Repeat the checks from step 6, with my-model as the model name.
| Task | Command |
|---|---|
| See whether it is running | docker compose ps |
| Read the last 100 log lines | docker compose logs --tail 100 vllm |
Apply a changed compose.yaml | docker compose up -d |
| Stop the service | docker compose down |
| Which vLLM version is this? | curl -s http://127.0.0.1:8000/version |
Pin the version. latest moves, and flags and defaults change between releases. Once the service works, ask it for its version and replace latest with that tag, in the form v0.x.y. Upgrade on purpose: change the tag, run docker compose pull, then docker compose up -d, and repeat your checks.
Run without root. The image runs as root by default. It also ships a user named vllm with UID 2000 in group 0. To use it, add user: "2000:0" to the service, mount the model cache at /home/vllm/.cache/huggingface and the compile cache at /home/vllm/.cache/vllm, and make the host directory writable by group 0.
Give it enough CPU. The documentation counts at least one API server process, one engine process and one worker per GPU, and recommends at least 2 + N physical cores for N GPUs. With hyperthreading that is twice as many vCPUs. A starved engine process shows up as a GPU that is never fully busy.
| What you see | Likely cause | What to do |
|---|---|---|
could not select device driver | Docker has no NVIDIA runtime | Steps 3 and 4 |
| The log stops while downloading | Slow link, or a gated model without a valid token | Download first on the host with hf download <model>, as the troubleshooting page suggests, then start the container |
| Loading a large model takes very long | Weights on a network file system, or the machine is swapping | Keep the cache on a local disk and watch RAM with free -h |
| An out-of-memory error at startup | The model does not fit, or another process holds the card | Check nvidia-smi, then lower --max-model-len or use a smaller or quantized model |
| Errors that mention shared memory | ipc: host is missing | Add it back, or set shm_size |
| A CUDA or driver version error | The host driver is older than the image expects | Update the driver, or set VLLM_ENABLE_CUDA_COMPATIBILITY=1, which the documentation supports on selected professional and datacenter GPUs |
| It runs, but another machine cannot connect | The port is bound to 127.0.0.1 on purpose | Part 5 |
When none of these fit, set the environment variable VLLM_LOGGING_LEVEL=DEBUG and read the log from the top. The first error is usually the real one; the lines after it are consequences.
docker run line.The next part replaces the test model with one worth using: serving Qwen with chat, tool calling and thinking.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
The commands follow the NVIDIA, Docker and vLLM documentation as read on November 12, 2026. No GPU was available, so no container was started for this article. The Compose file was checked with docker compose config, which validates the file but does not run it. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.