Run vLLM in Docker on an NVIDIA GPU, step by step
Install the NVIDIA Container Toolkit, prove the GPU works inside a container, start vLLM with docker run, then make it a Compose service that survives reboots.
At a glanceOne Compose file for vLLM, node exporter, the NVIDIA DCGM exporter, Prometheus and Grafana, with the queries and alerts for CPU, RAM, GPU and the model server.
Partly testedThe configuration, rules and host queries were run against a live node exporter. GPU and vLLM metrics were checked for syntax only: there was no GPU.
A model server fails quietly. Nothing crashes; answers just get slower until someone complains. By then you want to know whether the GPU was full, the CPU was starved, the disk ran out, or requests were queueing, and you want last Tuesday's numbers as well as today's. This part sets up that record. It is one Compose file, and most of it was tested for this article.
| Piece | Job | Port |
|---|---|---|
| vLLM | Publishes its own numbers at /metrics: requests running and waiting, cache use, latencies, tokens | 8000 |
| node exporter | Publishes the machine's numbers: CPU, memory, disk, network, load | 9100 |
| DCGM exporter | Publishes each GPU's numbers: utilization, memory, temperature, power. From NVIDIA. | 9400 |
| Prometheus | Collects all three every few seconds, stores the history, evaluates alert rules | 9090 |
| Grafana | Draws dashboards from what Prometheus stored | 3000 |
The exporters hold no history. They answer "what is the value now" each time Prometheus asks. Prometheus keeps the answers, for 15 days by default.
Make a folder with these three files. The first is compose.yaml. It contains the vLLM service from part 2 and four new ones:
services:
vllm:
image: vllm/vllm-openai:latest
command:
- --model
- Qwen/Qwen3-8B
- --served-model-name
- qwen3
- --max-model-len
- "16384"
environment:
HF_TOKEN: ${HF_TOKEN:-}
volumes:
- ${HOME}/.cache/huggingface:/root/.cache/huggingface
- vllm-cache:/root/.cache/vllm
ports:
- "127.0.0.1:8000:8000"
ipc: host
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
node-exporter:
image: quay.io/prometheus/node-exporter:latest
command:
- --path.rootfs=/host
pid: host
volumes:
- /:/host:ro,rslave
restart: unless-stopped
dcgm-exporter:
image: nvcr.io/nvidia/k8s/dcgm-exporter:latest
cap_add:
- SYS_ADMIN
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yaml:/etc/prometheus/prometheus.yml:ro
- ./alerts.yaml:/etc/prometheus/alerts.yaml:ro
- prometheus-data:/prometheus
ports:
- "127.0.0.1:9090:9090"
restart: unless-stopped
grafana:
image: grafana/grafana:latest
volumes:
- grafana-data:/var/lib/grafana
ports:
- "127.0.0.1:3000:3000"
depends_on:
- prometheus
restart: unless-stopped
volumes:
vllm-cache:
prometheus-data:
grafana-data:
Points worth understanding:
vllm:8000, node-exporter:9100 and dcgm-exporter:9400 directly. Nothing about your machine is offered to the outside.--path.rootfs, and to share the host's process namespace. The README's own example also uses the host's network. This file keeps it on the private network instead, so CPU, memory and disk figures are the host's, but network traffic figures are the container's. If you need host network counters, follow the README's variant.docker run command, which uses --gpus all --cap-add SYS_ADMIN. NVIDIA's guide asks you to review that privilege before using it in production.ssh -L 3000:127.0.0.1:3000 -L 9090:127.0.0.1:9090 you@server.The second file is prometheus.yaml, which says what to collect and how often:
global:
scrape_interval: 15s
evaluation_interval: 30s
rule_files:
- /etc/prometheus/alerts.yaml
scrape_configs:
- job_name: vllm
scrape_interval: 5s
static_configs:
- targets: ["vllm:8000"]
- job_name: node
static_configs:
- targets: ["node-exporter:9100"]
- job_name: dcgm
static_configs:
- targets: ["dcgm-exporter:9400"]
vLLM is collected every 5 seconds, as in vLLM's own example, because request load changes quickly. The machine and GPUs are collected every 15 seconds. The DCGM exporter refreshes its own readings every 30 seconds by default, so asking it more often gains nothing.
The third file is alerts.yaml, in step 7. Create it before starting, because Prometheus will not start if a file it mounts is missing.
docker compose config --quiet && echo "file is valid"
docker compose up -d
docker compose ps
Then open http://localhost:9090/targets through the tunnel. You should see three jobs, each marked UP. This page is the first place to look whenever a dashboard is empty.
| Target that is DOWN | Usual reason |
|---|---|
vllm | The model is still loading. It turns UP when the server is ready. |
dcgm | The container cannot see the GPUs. Run the test from part 2, step 4. |
node | The container did not start; read docker compose logs node-exporter. |
In the test for this article, the node target came up at once and the absent vLLM target was reported as down with a connection error, which is what the page should show while a model loads.
http://localhost:3000 and sign in with the default user admin and password admin. Grafana asks for a new password; set one.http://prometheus:9090. That is the service name on the private network, not localhost. Save and test.Now import three dashboards. In Grafana, open Dashboards, choose New, then Import. Paste a dashboard ID or upload a JSON file, click Load, select the Prometheus data source and click Import.
| Dashboard | How to import | Shows |
|---|---|---|
| Node Exporter Full | ID 1860 | CPU, memory, disk, network and load for the host |
| NVIDIA DCGM Exporter Dashboard | ID 12239, published by NVIDIA | Utilization, memory, temperature and power per GPU |
| vLLM | Upload grafana.json from the Prometheus and Grafana example in the vLLM repository | Latency, throughput, queue and cache for the model server |
The NVIDIA dashboard on grafana.com was last updated in 2021. If some of its panels stay empty with a current exporter, the metric names have moved on; the queries in the next step show how to find the current names.
Dashboards answer the questions someone else thought of. Explore, in Grafana's left menu, is where you type a query and see the result at once. It is the fastest way to learn what your metrics mean. These fifteen queries were all checked with Prometheus's own tool; the first four were also run against a live machine.
| Question | Query |
|---|---|
| How busy is the CPU, in percent? | 100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))) |
| How much memory is in use, in percent? | 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) |
| How many GiB are free on the root disk? | node_filesystem_avail_bytes{mountpoint="/"} / 2^30 |
| What is the one-minute load? | node_load1 |
| How busy is each GPU, in percent? | DCGM_FI_DEV_GPU_UTIL |
| How much memory does each GPU use, in MiB? | DCGM_FI_DEV_FB_USED |
| How many requests are being answered? | vllm:num_requests_running |
| How many are waiting for a slot? | vllm:num_requests_waiting |
| How full is the KV cache, in percent? | 100 * vllm:kv_cache_usage_perc |
| How many tokens a second are being written? | rate(vllm:generation_tokens_total[1m]) |
| How long until the first token, for 95% of requests? | histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) |
| How long between tokens, for 95%? | histogram_quantile(0.95, sum by (le) (rate(vllm:inter_token_latency_seconds_bucket[5m]))) |
| How long do requests queue, for 95%? | histogram_quantile(0.95, sum by (le) (rate(vllm:request_queue_time_seconds_bucket[5m]))) |
| Were any requests paused in the last hour? | increase(vllm:num_preemptions_total[1h]) |
| What share of prompt tokens came from the prefix cache? | rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) |
Three things to know when reading or writing these:
vllm:generation_tokens, but in queries it is vllm:generation_tokens_total, as in vLLM's own dashboard file. A counter only ever grows, so you nearly always wrap it in rate() or increase().histogram_quantile turns them into "95 percent of requests were faster than this".DCGM_FI_DEV_GPU_TEMP_CELSIUS and DCGM_FI_DEV_BOARD_POWER_WATTS for temperature and power. To see exactly what your exporter offers, type DCGM_ in Explore and read the suggestions.vLLM's usage note about memory applies here too. GPU memory will look almost full at all times, because vLLM takes the free memory for its cache at startup. That is normal. The cache gauge tells you how much of that memory is actually in use.
| What you see together | What it means | What to do |
|---|---|---|
| Requests waiting, GPU near 100% | The GPU is the limit | More GPUs or replicas, a smaller model, or shorter answers (part 6) |
| Requests waiting, GPU well below 100%, CPU high | The CPU is starving the engine | More cores; the documentation recommends at least 2 + N for N GPUs |
| KV cache near 100%, preemptions rising | Memory is the limit | Lower --max-num-seqs or --max-model-len, or add GPU memory (part 5) |
| Slow first token, nothing waiting | Prompts are long | Shorten prompts, or keep the shared part first so the prefix cache helps |
| System memory falling day by day | Something on the host is leaking | Find the process before the machine starts swapping |
| Root disk shrinking | Model downloads, images or logs | Clear unused models from the cache and old images |
| GPU temperature climbing while tokens per second fall | The card may be slowing itself to stay cool | Check airflow and fans |
Save this as alerts.yaml. The thresholds are starting points, not recommendations from any vendor. Adjust them after a week of watching your own numbers.
groups:
- name: llm-server
rules:
- alert: TargetDown
expr: up == 0
for: 2m
annotations:
summary: "{{ $labels.job }} is not answering"
- alert: RequestsQueueing
expr: vllm:num_requests_waiting > 0
for: 5m
annotations:
summary: "Requests have been waiting for a slot for 5 minutes"
- alert: KVCacheNearlyFull
expr: vllm:kv_cache_usage_perc > 0.9
for: 10m
annotations:
summary: "The KV cache has been over 90 percent full for 10 minutes"
- alert: RequestsPreempted
expr: increase(vllm:num_preemptions_total[10m]) > 0
annotations:
summary: "Requests were paused and recomputed in the last 10 minutes"
- alert: HostMemoryLow
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
for: 10m
annotations:
summary: "Less than 10 percent of system memory is available"
- alert: DiskNearlyFull
expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
for: 10m
annotations:
summary: "Less than 10 percent of the root disk is free"
for: 5m means the condition must hold for five minutes before the alert fires, which filters out brief spikes. The cache gauge runs from 0 to 1, so 0.9 is 90 percent.
After a restart of Prometheus, open http://localhost:9090/alerts to see each rule and its state. This setup shows alerts on that page only. To be sent a message when one fires, continue with part 8, alerts with Alertmanager.
/metrics endpoint needs no key, even when an API key is set. The security page lists it among the unprotected endpoints. That is convenient for Prometheus on the private network, and it is the reason the proxy in part 5 refuses the path from outside.| Symptom | Check |
|---|---|
| Every panel is empty | The data source URL must be http://prometheus:9090; use "Save and test" |
| One dashboard is empty | Is its target UP on the targets page? |
| Some panels of a dashboard are empty | Open the panel's query in Explore and compare the metric name with the suggestions |
| vLLM latency panels are empty on a new server | Histograms have no data until requests have been served; send a few |
| Disk figures look like the container's | The /:/host mount or --path.rootfs=/host is missing |
| Prometheus will not start | Run docker compose logs prometheus; a wrong indent in a YAML file is the usual cause |
Across these parts you have gone from one command on a test model to a server with a container setup you can reproduce, two real models, per-user access, a sizing method and a record of how it behaves. The parts in order:
The habit that matters most is the one this part makes possible: change one setting at a time, and look at the graphs before and after.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
The Compose file was checked with docker compose config. The Prometheus configuration, the alert rules and all fifteen queries were checked with promtool 2.45. The CPU, memory, disk and load queries were also run against a live node exporter 1.7 and returned data, and the target-down alert fired as expected. The GPU and vLLM queries were checked for syntax only: no GPU was available, so DCGM and vLLM metrics were not collected. Metric names follow the NVIDIA and vLLM documentation as read on November 12, 2026. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.