How many GPUs for N users? A vLLM sizing method
Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
At a glanceSend your vLLM server's alerts to your phone with Alertmanager and Telegram: routing by urgency, timing, silences, and a full test without a GPU.
Partly testedRules, routing, a delivered alert and a silence were tested locally with Prometheus and Alertmanager. Delivery to Telegram was not.
In part 7 you built dashboards and six alert rules. They have one weakness: an alert that fires at 3 a.m. turns a line red on a page nobody is looking at. This part sends those alerts to your phone. It also covers the three things that decide whether you keep alerting switched on after the first week: sending only what matters, pausing it during maintenance, and knowing when the alerting itself has died.
The split matters. Without the middle box, one failed server produces a message per rule, per check, every 30 seconds. Alertmanager's job is to turn that stream into a small number of messages a person can act on.
Add one label to every rule: how fast a person has to react. Two levels are enough to start.
| Severity | Meaning | Which of our alerts |
|---|---|---|
page | Users are affected now, or will be within minutes. Wake someone. | TargetDown, HostMemoryLow |
ticket | Something is degrading. Look at it during working hours. | RequestsQueueing, KVCacheNearlyFull, RequestsPreempted, DiskNearlyFull |
Here is the rule file from part 7 with those labels, and one new rule at the end that section 8 explains. Replace your alerts.yaml with it:
groups:
- name: llm-server
rules:
- alert: TargetDown
expr: up == 0
for: 2m
labels:
severity: page
annotations:
summary: "{{ $labels.job }} is not answering"
- alert: RequestsQueueing
expr: vllm:num_requests_waiting > 0
for: 5m
labels:
severity: ticket
annotations:
summary: "Requests have been waiting for a slot for 5 minutes"
- alert: KVCacheNearlyFull
expr: vllm:kv_cache_usage_perc > 0.9
for: 10m
labels:
severity: ticket
annotations:
summary: "The KV cache has been over 90 percent full for 10 minutes"
- alert: RequestsPreempted
expr: increase(vllm:num_preemptions_total[10m]) > 0
labels:
severity: ticket
annotations:
summary: "Requests were paused and recomputed in the last 10 minutes"
- alert: HostMemoryLow
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
for: 10m
labels:
severity: page
annotations:
summary: "Less than 10 percent of system memory is available"
- alert: DiskNearlyFull
expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
for: 10m
labels:
severity: ticket
annotations:
summary: "Less than 10 percent of the root disk is free"
# Always firing, on purpose. If its message stops arriving, the alerting itself is broken.
- alert: Watchdog
expr: vector(1)
labels:
severity: heartbeat
annotations:
summary: "Alerting is alive"
$ promtool check rules alerts.yaml
Checking alerts.yaml
SUCCESS: 7 rules found
Any receiver works the same way. Telegram is used here because a bot is free, takes two minutes and rings a phone.
@BotFather and send /newbot. It asks for a name and a username, then gives you a token. The token is a password: anyone who has it can send messages as your bot.https://api.telegram.org/bot<token>/getUpdates in a browser, with your token in place of <token>. In the reply, find "chat" and note the number after "id".printf '%s' 'PASTE-YOUR-TOKEN-HERE' > telegram-token
chmod 644 telegram-token
nobody, so the file must be readable by other users on the machine. Keep it in a folder that only you can enter (chmod 700 on the folder), keep it out of version control, and add telegram-token to .gitignore now.Save this as alertmanager.yaml, with your own chat ID in place of 123456789:
route:
receiver: tickets # anything not matched below
group_by: [alertname, job]
group_wait: 30s # collect related alerts before the first message
group_interval: 5m # then at most one update per group every 5 minutes
repeat_interval: 4h # remind while it is still firing
routes:
- matchers: [severity="page"]
receiver: phone
repeat_interval: 1h
- matchers: [severity="heartbeat"]
receiver: heartbeat
group_wait: 0s
repeat_interval: 5m
receivers:
- name: phone
telegram_configs:
- bot_token_file: /etc/alertmanager/telegram-token
chat_id: 123456789
parse_mode: ""
message: |-
{{ if eq .Status "firing" }}🔴 FIRING{{ else }}🟢 RESOLVED{{ end }}: {{ .CommonLabels.alertname }}
{{ range .Alerts }}{{ .Annotations.summary }}
{{ end }}
- name: tickets
telegram_configs:
- bot_token_file: /etc/alertmanager/telegram-token
chat_id: 123456789
disable_notifications: true # arrives silently
parse_mode: ""
message: |-
{{ if eq .Status "firing" }}🟠 {{ else }}🟢 resolved: {{ end }}{{ .CommonLabels.alertname }}
{{ range .Alerts }}{{ .Annotations.summary }}
{{ end }}
- name: heartbeat
webhook_configs:
- url: https://heartbeat.example.com/ping/your-check-id
send_resolved: false
Read it as a decision tree. Every alert enters at the top and takes the first matching branch:
$ amtool check-config alertmanager.yaml
Checking 'alertmanager.yaml' SUCCESS
Found:
- global config
- route
- 0 inhibit rules
- 3 receivers
- 0 templates
$ amtool config routes show --config.file=alertmanager.yaml
Routing tree:
.
└── default-route receiver: tickets
├── {severity="page"} receiver: phone
└── {severity="heartbeat"} receiver: heartbeat
$ amtool config routes test --config.file=alertmanager.yaml severity=page alertname=TargetDown
phone
$ amtool config routes test --config.file=alertmanager.yaml severity=ticket alertname=RequestsQueueing
tickets
$ amtool config routes test --config.file=alertmanager.yaml alertname=SomethingNew
tickets
amtool ships with Alertmanager. The last test shows the safety net: an alert with no severity still reaches someone.The two Telegram receivers differ in one line. disable_notifications: true makes a message arrive without a sound, so a ticket waits quietly in the chat while a page rings.
For Slack, replace telegram_configs with slack_configs and give it api_url_file, a file holding an incoming-webhook address. For email, use email_configs. The route above stays the same.
Add this service to compose.yaml, next to prometheus, and add alertmanager-data: under volumes: at the bottom:
alertmanager:
image: prom/alertmanager:latest
volumes:
- ./alertmanager.yaml:/etc/alertmanager/alertmanager.yml:ro
- ./telegram-token:/etc/alertmanager/telegram-token:ro
- alertmanager-data:/alertmanager
ports:
- "127.0.0.1:9093:9093"
restart: unless-stopped
Then tell Prometheus where to send what fires. Add this block to prometheus.yaml, at the top level:
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
docker compose config -q && docker compose up -d
The first command prints nothing when the file is valid. Alertmanager's own page is then at http://localhost:9093, reachable from the server itself, like the rest of the monitoring.
Do not wait for a real failure to find out whether alerting works. This test runs on any machine, with no GPU and no vLLM. It needs three small files.
A stand-in for vLLM. It serves two metrics and reads the number of waiting requests from a file, so you can change it while it runs. Save as fake_vllm.py:
"""fake_vllm.py: pretend to be a vLLM server's /metrics page, so alerts can be tested without a GPU.
The number of waiting requests is read from waiting.txt each time, so you can change it while this runs.
"""
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
class Metrics(BaseHTTPRequestHandler):
def do_GET(self):
waiting = Path("waiting.txt").read_text().strip() if Path("waiting.txt").exists() else "0"
body = f"vllm:num_requests_waiting {waiting}\nvllm:kv_cache_usage_perc 0.42\n".encode()
self.send_response(200)
self.send_header("Content-Type", "text/plain; version=0.0.4")
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
HTTPServer(("127.0.0.1", 8000), Metrics).serve_forever()
A stand-in for the chat app. It prints each message it is sent. Save as catch.py:
"""catch.py: stand in for a chat app. Print each alert message Alertmanager delivers to port 5001."""
import json
from datetime import datetime
from http.server import BaseHTTPRequestHandler, HTTPServer
class Catch(BaseHTTPRequestHandler):
def do_POST(self):
message = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
for alert in message["alerts"]:
print(f'{datetime.now():%H:%M:%S} to {self.path[1:]:8} {alert["status"]:9} {alert["labels"]["alertname"]}: {alert["annotations"]["summary"]}', flush=True)
self.send_response(200)
self.end_headers()
def log_message(self, *args):
pass
HTTPServer(("127.0.0.1", 5001), Catch).serve_forever()
Test configurations with short timings. Save as test-alertmanager.yaml:
route:
receiver: tickets
group_by: [alertname, job]
group_wait: 5s
group_interval: 10s
repeat_interval: 1h
routes:
- matchers: [severity="page"]
receiver: phone
- matchers: [severity="heartbeat"]
receiver: heartbeat
receivers:
- name: phone
webhook_configs: [{url: "http://127.0.0.1:5001/phone"}]
- name: tickets
webhook_configs: [{url: "http://127.0.0.1:5001/tickets"}]
- name: heartbeat
webhook_configs: [{url: "http://127.0.0.1:5001/heartbeat", send_resolved: false}]
And as test-prometheus.yaml:
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- test-alerts.yaml
alerting:
alertmanagers:
- static_configs:
- targets: ["127.0.0.1:9093"]
scrape_configs:
- job_name: vllm
static_configs:
- targets: ["127.0.0.1:8000"]
Make a copy of your real rules in which every waiting period is 15 seconds, so you are testing the real conditions without the real patience:
sed 's/for: .*/for: 15s/' alerts.yaml > test-alerts.yaml
Start the four programs, each in its own terminal. If you installed Alertmanager from Ubuntu's or Debian's packages, the command is prometheus-alertmanager:
python3 catch.py
echo 3 > waiting.txt && python3 fake_vllm.py
alertmanager --config.file=test-alertmanager.yaml --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
prometheus --config.file=test-prometheus.yaml --web.listen-address=127.0.0.1:9091
Now play out an incident. Three requests are "waiting" from the start. After the first message arrives, clear the queue with echo 0 > waiting.txt. Then stop fake_vllm.py with Ctrl+C, as if the server had crashed. This is what catch.py printed in our run:
12:18:38 to tickets firing RequestsQueueing: Requests have been waiting for a slot for 5 minutes
12:18:58 to tickets resolved RequestsQueueing: Requests have been waiting for a slot for 5 minutes
12:19:18 to heartbeat firing Watchdog: Alerting is alive
12:19:58 to phone firing TargetDown: vllm is not answering
That one run proves five things: Prometheus reads the rules, the rules fire, Alertmanager receives them, the routing sends each severity to the right place, and "resolved" messages arrive. When it works here, the only untested link in production is the receiver itself, and you test that by stopping the real vLLM container for three minutes on a quiet afternoon.
Four settings add up, and it helps to know which one you are waiting for.
| Setting | Where | What it does | Our value |
|---|---|---|---|
for | The rule | How long the condition must hold before the alert fires. Filters out brief spikes. | 2 to 10 minutes |
group_wait | Alertmanager | How long to collect related alerts before the first message. | 30 seconds |
group_interval | Alertmanager | The shortest gap between updates about the same group. | 5 minutes |
repeat_interval | Alertmanager | How often to remind you while nothing has changed. | 1 hour for pages, 4 hours for tickets |
So with the real settings, a crashed server reaches your phone about two and a half minutes after it stops answering: 2 minutes of for, then 30 seconds of group_wait, plus up to one check interval. Shorten for and you are woken by restarts that fix themselves. Lengthen it and users notice first. Two minutes is a reasonable place to begin.
When you restart the server on purpose, tell Alertmanager first. A silence mutes matching alerts for a set time and then removes itself, which is safer than switching alerting off and forgetting to switch it on again.
$ amtool --alertmanager.url=http://127.0.0.1:9093 silence add job=vllm --duration=2h --author=you --comment="Model reload"
d8ce8c10-f8ef-4a4f-8d3e-1bcd27933c13
$ amtool --alertmanager.url=http://127.0.0.1:9093 silence query
ID Matchers Ends At Created By Comment
d8ce8c10-f8ef-4a4f-8d3e-1bcd27933c13 job="vllm" 2026-11-12 14:20:23 UTC you Model reload
$ amtool --alertmanager.url=http://127.0.0.1:9093 alert query --silenced
Alertname Starts At Summary State
TargetDown 2026-11-12 12:19:53 UTC vllm is not answering suppressed
amtool silence expire and its ID.With Docker, run the same commands inside the container: docker compose exec alertmanager amtool --alertmanager.url=http://localhost:9093 silence add …. The web page on port 9093 has a button for it too.
The worst alerting failure is silent. If the monitoring machine loses power, no alert fires, and "no messages" looks the same as "all is well".
The Watchdog rule at the end of the rule file is the standard answer. Its condition is always true, so it fires all the time, on purpose. The heartbeat route sends it every 5 minutes to a service outside your server whose only job is to notice when the pings stop, and to tell you. Several such services exist, some free for one check; search for "dead man's switch monitoring" or "heartbeat monitoring". Replace the address in the heartbeat receiver with the one your service gives you.
heartbeat route and receiver, or the pings will go to an address that does not exist and fill Alertmanager's log with errors. The other two routes work without it.| Mistake | What happens | Instead |
|---|---|---|
| Everything is urgent | The phone rings for a full disk at 3 a.m. Within a week the chat is muted. | Page only when users are affected now |
No for on a rule | A message for every brief spike | Give every threshold rule a waiting period |
| The token in the configuration file | It ends up in a repository | bot_token_file, and the file in .gitignore |
| Alerts nobody can act on | "KV cache nearly full", and then what? | Put the first step in the annotation, or a link to your notes on the fix |
| Never testing | The first real incident is also the first test | Run section 6 after every change to the rules or routes |
| Publishing port 9093 | Anyone can silence your alerts | Keep it on 127.0.0.1, as in the Compose file |
You now have a server that tells you when it needs you, and stays quiet when it does not. The whole series, in order: a first server, Docker, Qwen, a coding model on A100s, many users, sizing, dashboards, and alerts.
When an alert does fire, two guides help you answer it: vLLM out of memory for the cache and memory alerts, and the multi-user guide for a queue that keeps growing.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
The rule file, both Alertmanager configurations and the Compose file were validated with promtool, amtool and docker compose config. The end-to-end test in section 6 was run for this article on Linux with Prometheus 2.45.3 and Alertmanager 0.26.0: the messages shown were really delivered to a local receiver, and the silence was really applied. Delivery to Telegram itself was not tested, because that needs a real bot and chat; the Telegram settings are from Alertmanager's documentation and passed amtool's check. The Compose service was checked for syntax, not started. Thresholds and timings are starting points. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.