Seongwoo Choi / Journal

vLLM alerts with Alertmanager: know before your users do

At a glanceSend your vLLM server's alerts to your phone with Alertmanager and Telegram: routing by urgency, timing, silences, and a full test without a GPU.

Partly testedRules, routing, a delivered alert and a silence were tested locally with Prometheus and Alertmanager. Delivery to Telegram was not.

You need
The monitoring stack from part 7, and a Telegram account. The test in section 6 needs only Prometheus, Alertmanager and Python on any machine.
You end up with
Urgent alerts that ring your phone, minor ones that arrive silently, a way to pause them during maintenance, and a check that the alerting itself is alive.
Series · part 8 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 11 sections
  1. 1. The path of an alert
  2. 2. Say how urgent each alert is
  3. 3. Create a place for the messages to go
  4. 4. Tell Alertmanager who gets what
  5. 5. Connect it to the stack from part 7
  6. 6. Test the whole chain without breaking anything
  7. 7. How long until the message arrives
  8. 8. Pause alerts during maintenance
  9. 9. Who watches the watcher
  10. 10. Mistakes that make people turn alerting off
  11. 11. Where the series stands

In part 7 you built dashboards and six alert rules. They have one weakness: an alert that fires at 3 a.m. turns a line red on a page nobody is looking at. This part sends those alerts to your phone. It also covers the three things that decide whether you keep alerting switched on after the first week: sending only what matters, pausing it during maintenance, and knowing when the alerting itself has died.

1. The path of an alert

Prometheuschecks each rule; a condition that holds long enough "fires"
→
Alertmanagergroups, waits, routes, silences
→
A receiverTelegram, Slack, email, or any web address
→
Youone message, not forty
Prometheus decides that something is wrong. Alertmanager decides who hears about it, how soon, and how often.

The split matters. Without the middle box, one failed server produces a message per rule, per check, every 30 seconds. Alertmanager's job is to turn that stream into a small number of messages a person can act on.

2. Say how urgent each alert is

Add one label to every rule: how fast a person has to react. Two levels are enough to start.

SeverityMeaningWhich of our alerts
pageUsers are affected now, or will be within minutes. Wake someone.TargetDown, HostMemoryLow
ticketSomething is degrading. Look at it during working hours.RequestsQueueing, KVCacheNearlyFull, RequestsPreempted, DiskNearlyFull

Here is the rule file from part 7 with those labels, and one new rule at the end that section 8 explains. Replace your alerts.yaml with it:

groups:
  - name: llm-server
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} is not answering"

      - alert: RequestsQueueing
        expr: vllm:num_requests_waiting > 0
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "Requests have been waiting for a slot for 5 minutes"

      - alert: KVCacheNearlyFull
        expr: vllm:kv_cache_usage_perc > 0.9
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "The KV cache has been over 90 percent full for 10 minutes"

      - alert: RequestsPreempted
        expr: increase(vllm:num_preemptions_total[10m]) > 0
        labels:
          severity: ticket
        annotations:
          summary: "Requests were paused and recomputed in the last 10 minutes"

      - alert: HostMemoryLow
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Less than 10 percent of system memory is available"

      - alert: DiskNearlyFull
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "Less than 10 percent of the root disk is free"

      # Always firing, on purpose. If its message stops arriving, the alerting itself is broken.
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: heartbeat
        annotations:
          summary: "Alerting is alive"
$ promtool check rules alerts.yaml
Checking alerts.yaml
  SUCCESS: 7 rules found

3. Create a place for the messages to go

Any receiver works the same way. Telegram is used here because a bot is free, takes two minutes and rings a phone.

  1. Create a bot. In Telegram, open a chat with @BotFather and send /newbot. It asks for a name and a username, then gives you a token. The token is a password: anyone who has it can send messages as your bot.
  2. Start a chat with your bot and send it any message, such as "hello". A bot cannot write to you first.
  3. Find your chat ID. Open https://api.telegram.org/bot<token>/getUpdates in a browser, with your token in place of <token>. In the reply, find "chat" and note the number after "id".
  4. Save the token in a file, not in the configuration:
printf '%s' 'PASTE-YOUR-TOKEN-HERE' > telegram-token
chmod 644 telegram-token
About that file's permissionsThe official Alertmanager image runs as the user nobody, so the file must be readable by other users on the machine. Keep it in a folder that only you can enter (chmod 700 on the folder), keep it out of version control, and add telegram-token to .gitignore now.

4. Tell Alertmanager who gets what

Save this as alertmanager.yaml, with your own chat ID in place of 123456789:

route:
  receiver: tickets                 # anything not matched below
  group_by: [alertname, job]
  group_wait: 30s                   # collect related alerts before the first message
  group_interval: 5m                # then at most one update per group every 5 minutes
  repeat_interval: 4h               # remind while it is still firing
  routes:
    - matchers: [severity="page"]
      receiver: phone
      repeat_interval: 1h
    - matchers: [severity="heartbeat"]
      receiver: heartbeat
      group_wait: 0s
      repeat_interval: 5m

receivers:
  - name: phone
    telegram_configs:
      - bot_token_file: /etc/alertmanager/telegram-token
        chat_id: 123456789
        parse_mode: ""
        message: |-
          {{ if eq .Status "firing" }}🔴 FIRING{{ else }}🟢 RESOLVED{{ end }}: {{ .CommonLabels.alertname }}
          {{ range .Alerts }}{{ .Annotations.summary }}
          {{ end }}

  - name: tickets
    telegram_configs:
      - bot_token_file: /etc/alertmanager/telegram-token
        chat_id: 123456789
        disable_notifications: true   # arrives silently
        parse_mode: ""
        message: |-
          {{ if eq .Status "firing" }}🟠 {{ else }}🟢 resolved: {{ end }}{{ .CommonLabels.alertname }}
          {{ range .Alerts }}{{ .Annotations.summary }}
          {{ end }}

  - name: heartbeat
    webhook_configs:
      - url: https://heartbeat.example.com/ping/your-check-id
        send_resolved: false

Read it as a decision tree. Every alert enters at the top and takes the first matching branch:

$ amtool check-config alertmanager.yaml
Checking 'alertmanager.yaml'  SUCCESS
Found:
 - global config
 - route
 - 0 inhibit rules
 - 3 receivers
 - 0 templates

$ amtool config routes show --config.file=alertmanager.yaml
Routing tree:
.
└── default-route  receiver: tickets
    ├── {severity="page"}  receiver: phone
    └── {severity="heartbeat"}  receiver: heartbeat

$ amtool config routes test --config.file=alertmanager.yaml severity=page alertname=TargetDown
phone
$ amtool config routes test --config.file=alertmanager.yaml severity=ticket alertname=RequestsQueueing
tickets
$ amtool config routes test --config.file=alertmanager.yaml alertname=SomethingNew
tickets
amtool ships with Alertmanager. The last test shows the safety net: an alert with no severity still reaches someone.

The two Telegram receivers differ in one line. disable_notifications: true makes a message arrive without a sound, so a ticket waits quietly in the chat while a page rings.

For Slack, replace telegram_configs with slack_configs and give it api_url_file, a file holding an incoming-webhook address. For email, use email_configs. The route above stays the same.

5. Connect it to the stack from part 7

Add this service to compose.yaml, next to prometheus, and add alertmanager-data: under volumes: at the bottom:

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yaml:/etc/alertmanager/alertmanager.yml:ro
      - ./telegram-token:/etc/alertmanager/telegram-token:ro
      - alertmanager-data:/alertmanager
    ports:
      - "127.0.0.1:9093:9093"
    restart: unless-stopped

Then tell Prometheus where to send what fires. Add this block to prometheus.yaml, at the top level:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]
docker compose config -q && docker compose up -d

The first command prints nothing when the file is valid. Alertmanager's own page is then at http://localhost:9093, reachable from the server itself, like the rest of the monitoring.

6. Test the whole chain without breaking anything

Do not wait for a real failure to find out whether alerting works. This test runs on any machine, with no GPU and no vLLM. It needs three small files.

A stand-in for vLLM. It serves two metrics and reads the number of waiting requests from a file, so you can change it while it runs. Save as fake_vllm.py:

"""fake_vllm.py: pretend to be a vLLM server's /metrics page, so alerts can be tested without a GPU.

The number of waiting requests is read from waiting.txt each time, so you can change it while this runs.
"""
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path


class Metrics(BaseHTTPRequestHandler):
    def do_GET(self):
        waiting = Path("waiting.txt").read_text().strip() if Path("waiting.txt").exists() else "0"
        body = f"vllm:num_requests_waiting {waiting}\nvllm:kv_cache_usage_perc 0.42\n".encode()
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


HTTPServer(("127.0.0.1", 8000), Metrics).serve_forever()

A stand-in for the chat app. It prints each message it is sent. Save as catch.py:

"""catch.py: stand in for a chat app. Print each alert message Alertmanager delivers to port 5001."""
import json
from datetime import datetime
from http.server import BaseHTTPRequestHandler, HTTPServer


class Catch(BaseHTTPRequestHandler):
    def do_POST(self):
        message = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        for alert in message["alerts"]:
            print(f'{datetime.now():%H:%M:%S}  to {self.path[1:]:8} {alert["status"]:9} {alert["labels"]["alertname"]}: {alert["annotations"]["summary"]}', flush=True)
        self.send_response(200)
        self.end_headers()

    def log_message(self, *args):
        pass


HTTPServer(("127.0.0.1", 5001), Catch).serve_forever()

Test configurations with short timings. Save as test-alertmanager.yaml:

route:
  receiver: tickets
  group_by: [alertname, job]
  group_wait: 5s
  group_interval: 10s
  repeat_interval: 1h
  routes:
    - matchers: [severity="page"]
      receiver: phone
    - matchers: [severity="heartbeat"]
      receiver: heartbeat

receivers:
  - name: phone
    webhook_configs: [{url: "http://127.0.0.1:5001/phone"}]
  - name: tickets
    webhook_configs: [{url: "http://127.0.0.1:5001/tickets"}]
  - name: heartbeat
    webhook_configs: [{url: "http://127.0.0.1:5001/heartbeat", send_resolved: false}]

And as test-prometheus.yaml:

global:
  scrape_interval: 5s
  evaluation_interval: 5s

rule_files:
  - test-alerts.yaml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["127.0.0.1:9093"]

scrape_configs:
  - job_name: vllm
    static_configs:
      - targets: ["127.0.0.1:8000"]

Make a copy of your real rules in which every waiting period is 15 seconds, so you are testing the real conditions without the real patience:

sed 's/for: .*/for: 15s/' alerts.yaml > test-alerts.yaml

Start the four programs, each in its own terminal. If you installed Alertmanager from Ubuntu's or Debian's packages, the command is prometheus-alertmanager:

python3 catch.py
echo 3 > waiting.txt && python3 fake_vllm.py
alertmanager --config.file=test-alertmanager.yaml --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
prometheus --config.file=test-prometheus.yaml --web.listen-address=127.0.0.1:9091

Now play out an incident. Three requests are "waiting" from the start. After the first message arrives, clear the queue with echo 0 > waiting.txt. Then stop fake_vllm.py with Ctrl+C, as if the server had crashed. This is what catch.py printed in our run:

12:18:38  to tickets  firing    RequestsQueueing: Requests have been waiting for a slot for 5 minutes
12:18:58  to tickets  resolved  RequestsQueueing: Requests have been waiting for a slot for 5 minutes
12:19:18  to heartbeat firing    Watchdog: Alerting is alive
12:19:58  to phone    firing    TargetDown: vllm is not answering
We cleared the queue at 12:18:54 and stopped the server at 12:19:31. The queue alert went to the quiet channel and was withdrawn 4 seconds after the queue cleared. The crash went to the phone 27 seconds after it happened.

That one run proves five things: Prometheus reads the rules, the rules fire, Alertmanager receives them, the routing sends each severity to the right place, and "resolved" messages arrive. When it works here, the only untested link in production is the receiver itself, and you test that by stopping the real vLLM container for three minutes on a quiet afternoon.

7. How long until the message arrives

Four settings add up, and it helps to know which one you are waiting for.

SettingWhereWhat it doesOur value
forThe ruleHow long the condition must hold before the alert fires. Filters out brief spikes.2 to 10 minutes
group_waitAlertmanagerHow long to collect related alerts before the first message.30 seconds
group_intervalAlertmanagerThe shortest gap between updates about the same group.5 minutes
repeat_intervalAlertmanagerHow often to remind you while nothing has changed.1 hour for pages, 4 hours for tickets

So with the real settings, a crashed server reaches your phone about two and a half minutes after it stops answering: 2 minutes of for, then 30 seconds of group_wait, plus up to one check interval. Shorten for and you are woken by restarts that fix themselves. Lengthen it and users notice first. Two minutes is a reasonable place to begin.

8. Pause alerts during maintenance

When you restart the server on purpose, tell Alertmanager first. A silence mutes matching alerts for a set time and then removes itself, which is safer than switching alerting off and forgetting to switch it on again.

$ amtool --alertmanager.url=http://127.0.0.1:9093 silence add job=vllm --duration=2h --author=you --comment="Model reload"
d8ce8c10-f8ef-4a4f-8d3e-1bcd27933c13
$ amtool --alertmanager.url=http://127.0.0.1:9093 silence query
ID                                    Matchers    Ends At                  Created By  Comment
d8ce8c10-f8ef-4a4f-8d3e-1bcd27933c13  job="vllm"  2026-11-12 14:20:23 UTC  you         Model reload
$ amtool --alertmanager.url=http://127.0.0.1:9093 alert query --silenced
Alertname   Starts At                Summary                State
TargetDown  2026-11-12 12:19:53 UTC  vllm is not answering  suppressed
From the same test run. The alert still exists; it is just not being sent. To end a silence early: amtool silence expire and its ID.

With Docker, run the same commands inside the container: docker compose exec alertmanager amtool --alertmanager.url=http://localhost:9093 silence add …. The web page on port 9093 has a button for it too.

9. Who watches the watcher

The worst alerting failure is silent. If the monitoring machine loses power, no alert fires, and "no messages" looks the same as "all is well".

The Watchdog rule at the end of the rule file is the standard answer. Its condition is always true, so it fires all the time, on purpose. The heartbeat route sends it every 5 minutes to a service outside your server whose only job is to notice when the pings stop, and to tell you. Several such services exist, some free for one check; search for "dead man's switch monitoring" or "heartbeat monitoring". Replace the address in the heartbeat receiver with the one your service gives you.

Until you set one upRemove the heartbeat route and receiver, or the pings will go to an address that does not exist and fill Alertmanager's log with errors. The other two routes work without it.

10. Mistakes that make people turn alerting off

MistakeWhat happensInstead
Everything is urgentThe phone rings for a full disk at 3 a.m. Within a week the chat is muted.Page only when users are affected now
No for on a ruleA message for every brief spikeGive every threshold rule a waiting period
The token in the configuration fileIt ends up in a repositorybot_token_file, and the file in .gitignore
Alerts nobody can act on"KV cache nearly full", and then what?Put the first step in the annotation, or a link to your notes on the fix
Never testingThe first real incident is also the first testRun section 6 after every change to the rules or routes
Publishing port 9093Anyone can silence your alertsKeep it on 127.0.0.1, as in the Compose file

11. Where the series stands

You now have a server that tells you when it needs you, and stays quiet when it does not. The whole series, in order: a first server, Docker, Qwen, a coding model on A100s, many users, sizing, dashboards, and alerts.

When an alert does fire, two guides help you answer it: vLLM out of memory for the cache and memory alerts, and the multi-user guide for a queue that keeps growing.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

The rule file, both Alertmanager configurations and the Compose file were validated with promtool, amtool and docker compose config. The end-to-end test in section 6 was run for this article on Linux with Prometheus 2.45.3 and Alertmanager 0.26.0: the messages shown were really delivered to a local receiver, and the silence was really applied. Delivery to Telegram itself was not tested, because that needs a real bot and chat; the Telegram settings are from Alertmanager's documentation and passed amtool's check. The Compose service was checked for syntax, not started. Thresholds and timings are starting points. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy