Seongwoo Choi / Journal

Monitor vLLM with Prometheus, Grafana and DCGM in Docker

At a glanceOne Compose file for vLLM, node exporter, the NVIDIA DCGM exporter, Prometheus and Grafana, with the queries and alerts for CPU, RAM, GPU and the model server.

Partly testedThe configuration, rules and host queries were run against a live node exporter. GPU and vLLM metrics were checked for syntax only: there was no GPU.

You need
The GPU host and Docker setup from part 2. About 2 GB of disk for images and data.
You end up with
Dashboards for CPU, memory, disk, every GPU and the model server, plus alert rules, all started by one command.
Series · part 7 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 10 sections
  1. 1. Five pieces, each with one job
  2. 2. Three files
  3. 3. Start it and check every target
  4. 4. Connect Grafana
  5. 5. Ask your own questions with Explore
  6. 6. Read the dashboards like a diagnosis
  7. 7. Six alerts to start with
  8. 8. Keep the monitoring itself private
  9. 9. When a panel is empty
  10. 10. What you built in this series

A model server fails quietly. Nothing crashes; answers just get slower until someone complains. By then you want to know whether the GPU was full, the CPU was starved, the disk ran out, or requests were queueing, and you want last Tuesday's numbers as well as today's. This part sets up that record. It is one Compose file, and most of it was tested for this article.

1. Five pieces, each with one job

vLLMrequests, latency, cache
node exporterCPU, memory, disk
DCGM exporterevery GPU
→
Prometheuscollects and stores
→
Grafanadashboards
Three sources of numbers, one store, one place to look.
PieceJobPort
vLLMPublishes its own numbers at /metrics: requests running and waiting, cache use, latencies, tokens8000
node exporterPublishes the machine's numbers: CPU, memory, disk, network, load9100
DCGM exporterPublishes each GPU's numbers: utilization, memory, temperature, power. From NVIDIA.9400
PrometheusCollects all three every few seconds, stores the history, evaluates alert rules9090
GrafanaDraws dashboards from what Prometheus stored3000

The exporters hold no history. They answer "what is the value now" each time Prometheus asks. Prometheus keeps the answers, for 15 days by default.

2. Three files

Make a folder with these three files. The first is compose.yaml. It contains the vLLM service from part 2 and four new ones:

services:
  vllm:
    image: vllm/vllm-openai:latest
    command:
      - --model
      - Qwen/Qwen3-8B
      - --served-model-name
      - qwen3
      - --max-model-len
      - "16384"
    environment:
      HF_TOKEN: ${HF_TOKEN:-}
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    ports:
      - "127.0.0.1:8000:8000"
    ipc: host
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  node-exporter:
    image: quay.io/prometheus/node-exporter:latest
    command:
      - --path.rootfs=/host
    pid: host
    volumes:
      - /:/host:ro,rslave
    restart: unless-stopped

  dcgm-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:latest
    cap_add:
      - SYS_ADMIN
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yaml:/etc/prometheus/alerts.yaml:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"
    depends_on:
      - prometheus
    restart: unless-stopped

volumes:
  vllm-cache:
  prometheus-data:
  grafana-data:

Points worth understanding:

  • The exporters publish no ports. All five services share a private Docker network, where each can reach the others by service name. Prometheus asks vllm:8000, node-exporter:9100 and dcgm-exporter:9400 directly. Nothing about your machine is offered to the outside.
  • node exporter needs to see the host, not its own container. Its README says to mount the host's root read-only and pass --path.rootfs, and to share the host's process namespace. The README's own example also uses the host's network. This file keeps it on the private network instead, so CPU, memory and disk figures are the host's, but network traffic figures are the container's. If you need host network counters, follow the README's variant.
  • The DCGM exporter gets the GPUs and one extra capability. This mirrors NVIDIA's docker run command, which uses --gpus all --cap-add SYS_ADMIN. NVIDIA's guide asks you to review that privilege before using it in production.
  • Prometheus and Grafana are bound to 127.0.0.1. Reach them through an SSH tunnel: ssh -L 3000:127.0.0.1:3000 -L 9090:127.0.0.1:9090 you@server.
  • Named volumes keep history and dashboards when containers are replaced.

The second file is prometheus.yaml, which says what to collect and how often:

global:
  scrape_interval: 15s
  evaluation_interval: 30s

rule_files:
  - /etc/prometheus/alerts.yaml

scrape_configs:
  - job_name: vllm
    scrape_interval: 5s
    static_configs:
      - targets: ["vllm:8000"]

  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: dcgm
    static_configs:
      - targets: ["dcgm-exporter:9400"]

vLLM is collected every 5 seconds, as in vLLM's own example, because request load changes quickly. The machine and GPUs are collected every 15 seconds. The DCGM exporter refreshes its own readings every 30 seconds by default, so asking it more often gains nothing.

The third file is alerts.yaml, in step 7. Create it before starting, because Prometheus will not start if a file it mounts is missing.

3. Start it and check every target

docker compose config --quiet && echo "file is valid"
docker compose up -d
docker compose ps

Then open http://localhost:9090/targets through the tunnel. You should see three jobs, each marked UP. This page is the first place to look whenever a dashboard is empty.

Target that is DOWNUsual reason
vllmThe model is still loading. It turns UP when the server is ready.
dcgmThe container cannot see the GPUs. Run the test from part 2, step 4.
nodeThe container did not start; read docker compose logs node-exporter.

In the test for this article, the node target came up at once and the absent vLLM target was reported as down with a connection error, which is what the page should show while a model loads.

4. Connect Grafana

  1. Open http://localhost:3000 and sign in with the default user admin and password admin. Grafana asks for a new password; set one.
  2. Go to Connections, then Data sources, add a new one and choose Prometheus.
  3. For the server URL enter http://prometheus:9090. That is the service name on the private network, not localhost. Save and test.

Now import three dashboards. In Grafana, open Dashboards, choose New, then Import. Paste a dashboard ID or upload a JSON file, click Load, select the Prometheus data source and click Import.

DashboardHow to importShows
Node Exporter FullID 1860CPU, memory, disk, network and load for the host
NVIDIA DCGM Exporter DashboardID 12239, published by NVIDIAUtilization, memory, temperature and power per GPU
vLLMUpload grafana.json from the Prometheus and Grafana example in the vLLM repositoryLatency, throughput, queue and cache for the model server

The NVIDIA dashboard on grafana.com was last updated in 2021. If some of its panels stay empty with a current exporter, the metric names have moved on; the queries in the next step show how to find the current names.

5. Ask your own questions with Explore

Dashboards answer the questions someone else thought of. Explore, in Grafana's left menu, is where you type a query and see the result at once. It is the fastest way to learn what your metrics mean. These fifteen queries were all checked with Prometheus's own tool; the first four were also run against a live machine.

QuestionQuery
How busy is the CPU, in percent?100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])))
How much memory is in use, in percent?100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
How many GiB are free on the root disk?node_filesystem_avail_bytes{mountpoint="/"} / 2^30
What is the one-minute load?node_load1
How busy is each GPU, in percent?DCGM_FI_DEV_GPU_UTIL
How much memory does each GPU use, in MiB?DCGM_FI_DEV_FB_USED
How many requests are being answered?vllm:num_requests_running
How many are waiting for a slot?vllm:num_requests_waiting
How full is the KV cache, in percent?100 * vllm:kv_cache_usage_perc
How many tokens a second are being written?rate(vllm:generation_tokens_total[1m])
How long until the first token, for 95% of requests?histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
How long between tokens, for 95%?histogram_quantile(0.95, sum by (le) (rate(vllm:inter_token_latency_seconds_bucket[5m])))
How long do requests queue, for 95%?histogram_quantile(0.95, sum by (le) (rate(vllm:request_queue_time_seconds_bucket[5m])))
Were any requests paused in the last hour?increase(vllm:num_preemptions_total[1h])
What share of prompt tokens came from the prefix cache?rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m])

Three things to know when reading or writing these:

  • Counters get a suffix. The vLLM documentation lists a counter as vllm:generation_tokens, but in queries it is vllm:generation_tokens_total, as in vLLM's own dashboard file. A counter only ever grows, so you nearly always wrap it in rate() or increase().
  • Latencies are histograms. They arrive as buckets, and histogram_quantile turns them into "95 percent of requests were faster than this".
  • GPU metric names depend on the exporter version. NVIDIA's guide names DCGM_FI_DEV_GPU_TEMP_CELSIUS and DCGM_FI_DEV_BOARD_POWER_WATTS for temperature and power. To see exactly what your exporter offers, type DCGM_ in Explore and read the suggestions.

vLLM's usage note about memory applies here too. GPU memory will look almost full at all times, because vLLM takes the free memory for its cache at startup. That is normal. The cache gauge tells you how much of that memory is actually in use.

6. Read the dashboards like a diagnosis

What you see togetherWhat it meansWhat to do
Requests waiting, GPU near 100%The GPU is the limitMore GPUs or replicas, a smaller model, or shorter answers (part 6)
Requests waiting, GPU well below 100%, CPU highThe CPU is starving the engineMore cores; the documentation recommends at least 2 + N for N GPUs
KV cache near 100%, preemptions risingMemory is the limitLower --max-num-seqs or --max-model-len, or add GPU memory (part 5)
Slow first token, nothing waitingPrompts are longShorten prompts, or keep the shared part first so the prefix cache helps
System memory falling day by daySomething on the host is leakingFind the process before the machine starts swapping
Root disk shrinkingModel downloads, images or logsClear unused models from the cache and old images
GPU temperature climbing while tokens per second fallThe card may be slowing itself to stay coolCheck airflow and fans

7. Six alerts to start with

Save this as alerts.yaml. The thresholds are starting points, not recommendations from any vendor. Adjust them after a week of watching your own numbers.

groups:
  - name: llm-server
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 2m
        annotations:
          summary: "{{ $labels.job }} is not answering"

      - alert: RequestsQueueing
        expr: vllm:num_requests_waiting > 0
        for: 5m
        annotations:
          summary: "Requests have been waiting for a slot for 5 minutes"

      - alert: KVCacheNearlyFull
        expr: vllm:kv_cache_usage_perc > 0.9
        for: 10m
        annotations:
          summary: "The KV cache has been over 90 percent full for 10 minutes"

      - alert: RequestsPreempted
        expr: increase(vllm:num_preemptions_total[10m]) > 0
        annotations:
          summary: "Requests were paused and recomputed in the last 10 minutes"

      - alert: HostMemoryLow
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        annotations:
          summary: "Less than 10 percent of system memory is available"

      - alert: DiskNearlyFull
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 10m
        annotations:
          summary: "Less than 10 percent of the root disk is free"

for: 5m means the condition must hold for five minutes before the alert fires, which filters out brief spikes. The cache gauge runs from 0 to 1, so 0.9 is 90 percent.

After a restart of Prometheus, open http://localhost:9090/alerts to see each rule and its state. This setup shows alerts on that page only. To be sent a message when one fires, continue with part 8, alerts with Alertmanager.

8. Keep the monitoring itself private

  • vLLM's /metrics endpoint needs no key, even when an API key is set. The security page lists it among the unprotected endpoints. That is convenient for Prometheus on the private network, and it is the reason the proxy in part 5 refuses the path from outside.
  • Change Grafana's default password at first sign-in, and do not publish ports 3000 or 9090 on a public interface.
  • Metrics reveal your model, your load and your hardware. Treat the dashboards as internal.

9. When a panel is empty

SymptomCheck
Every panel is emptyThe data source URL must be http://prometheus:9090; use "Save and test"
One dashboard is emptyIs its target UP on the targets page?
Some panels of a dashboard are emptyOpen the panel's query in Explore and compare the metric name with the suggestions
vLLM latency panels are empty on a new serverHistograms have no data until requests have been served; send a few
Disk figures look like the container'sThe /:/host mount or --path.rootfs=/host is missing
Prometheus will not startRun docker compose logs prometheus; a wrong indent in a YAML file is the usual cause

10. What you built in this series

Across these parts you have gone from one command on a test model to a server with a container setup you can reproduce, two real models, per-user access, a sizing method and a record of how it behaves. The parts in order:

  1. vLLM quickstart
  2. Docker on an NVIDIA GPU
  3. Serving Qwen
  4. Phind-CodeLlama-34B on A100 GPUs
  5. Serving a team
  6. GPU sizing
  7. Monitoring, this part
  8. Alerts with Alertmanager

The habit that matters most is the one this part makes possible: change one setting at a time, and look at the graphs before and after.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

The Compose file was checked with docker compose config. The Prometheus configuration, the alert rules and all fifteen queries were checked with promtool 2.45. The CPU, memory, disk and load queries were also run against a live node exporter 1.7 and returned data, and the target-down alert fired as expected. The GPU and vLLM queries were checked for syntax only: no GPU was available, so DCGM and vLLM metrics were not collected. Metric names follow the NVIDIA and vLLM documentation as read on November 12, 2026. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy