vLLM alerts with Alertmanager: know before your users do
Send your vLLM server's alerts to your phone with Alertmanager and Telegram: routing by urgency, timing, silences, and a full test without a GPU.
At a glanceWork out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
Partly testedThe sizing script was run on real model files and its output is shown. No GPU was used, so nothing here is a measured speed.
"How many GPUs do we need for 200 users?" has no answer on a vendor's page, because it depends on your model, your prompts and how patient your users are. It does have a method. Part of it is arithmetic you can do before buying anything, and part is a measurement you must make on real hardware. This part gives you both, with every number shown.
| Question | Limited by | How you answer it |
|---|---|---|
| Does it fit? | GPU memory | Arithmetic, from two files |
| Is it fast enough? | GPU computation | A load test |
People often stop after the first question. A setup that fits can still be far too slow for the load, and a setup that is fast can still pause requests because memory runs out. You need both answers, and the larger GPU count wins.
Add up the weight files in the model's repository. That is close to what they occupy on the GPU when the model runs at the precision it was published in.
Mind the units. File listings use gigabytes of 1,000³ bytes. GPU tools report mebibytes, and a card sold as "80 GB" shows about 80 × 1,024 MiB. To compare, divide bytes by 1,073,741,824 to get GiB. Phind-CodeLlama-34B is 67.5 GB on disk, which is 62.9 GiB.
While a request is being answered, the model keeps two tensors, a key and a value, for every token of that request in every layer. This is the KV cache. Its size per token comes straight from config.json:
bytes per token = 2 × layers × key-value heads × head dimension × bytes per number
| Term | Field in config.json |
|---|---|
| layers | num_hidden_layers |
| key-value heads | num_key_value_heads, or num_attention_heads if the first is absent |
| head dimension | head_dim, or hidden_size ÷ num_attention_heads |
| bytes per number | 2 when torch_dtype is bfloat16 or float16 |
Applied to four models from this series:
| Model | Layers × KV heads × head dim | Per token | One full default context |
|---|---|---|---|
| Qwen2.5-7B-Instruct | 28 × 4 × 128 | 57,344 bytes | 1.8 GiB at 32,768 tokens |
| Qwen3-8B | 36 × 8 × 128 | 147,456 bytes | 5.6 GiB at 40,960 tokens |
| Phind-CodeLlama-34B-v2 | 48 × 8 × 128 | 196,608 bytes | 0.75 GiB at 4,096 tokens |
| Qwen2.5-Coder-32B-Instruct | 64 × 8 × 128 | 262,144 bytes | 8.0 GiB at 32,768 tokens |
Two things stand out. Qwen3-8B is close to Qwen2.5-7B in size but needs more than twice the cache per token, because it has more layers and twice the key-value heads. And context length matters as much as the model: one full-length Qwen2.5-Coder request costs more memory than ten full-length Phind requests.
cache memory = GPU memory − weights − runtime reserve
tokens in cache = cache memory ÷ bytes per token
full-length requests at once = tokens in cache ÷ context length
The runtime reserve covers what vLLM needs besides weights and cache: the working memory of a forward pass and its compiled graphs. On NVIDIA GPUs, the documentation says vLLM measures this itself at startup and gives the rest to the cache, so the true figure is only known on the machine. For planning, this series keeps back a tenth of the card. That is an assumption, chosen to be cautious.
This script does the arithmetic. Download a model's config.json and save the script as kv_budget.py:
import json, sys
cfg = json.load(open(sys.argv[1])) # the model's config.json
gpu_gib = float(sys.argv[2]) # GPU memory in GiB, added up over all GPUs used
weights_gib = float(sys.argv[3]) # total size of the weight files in GiB
reserve = 0.10 # share kept back for the runtime: an assumption
layers = cfg["num_hidden_layers"]
kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
head_dim = cfg.get("head_dim", cfg["hidden_size"] // cfg["num_attention_heads"])
bytes_per_value = 2 # 16-bit values (bfloat16 or float16)
per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
cache_gib = gpu_gib * (1 - reserve) - weights_gib
tokens = cache_gib * 2**30 / per_token
print(f"KV cache per token: {per_token:,} bytes")
print(f"Left for the cache: {cache_gib:.1f} GiB, about {tokens:,.0f} tokens")
for length in (4096, 8192, 16384, 32768):
print(f" full requests of {length:,} tokens: {tokens / length:.1f}")
Run for Phind-CodeLlama-34B on one 80 GB card, it printed:
$ python kv_budget.py config.json 80 62.85
KV cache per token: 196,608 bytes
Left for the cache: 9.1 GiB, about 49,971 tokens
full requests of 4,096 tokens: 12.2
full requests of 8,192 tokens: 6.1
full requests of 16,384 tokens: 3.0
full requests of 32,768 tokens: 1.5
If "Left for the cache" is negative, the model does not fit on that hardware.
Check it against the server. When vLLM starts, it logs the real numbers, in lines of this form from the documentation:
GPU KV cache size: 643,232 tokens
Maximum concurrency for 40,960 tokens per request: 15.70x
The first is the measured version of "tokens in cache". The second divides it by your --max-model-len. The documentation's own advice is to add GPUs if these are lower than you need. Once you have a machine, trust these lines over any estimate, including this article's.
Compare three ways to run Phind-CodeLlama-34B on A100 80GB cards, using the same script:
| Layout | Cache memory | Full 4,096-token requests |
|---|---|---|
| 1 GPU | 9.1 GiB | 12 |
| 2 GPUs as two separate servers | 2 × 9.1 GiB | 24 |
| 2 GPUs as one server with tensor parallelism | 81.2 GiB | 108 |
Model weightsKV cache, the room for requestsRuntime reserve
Two separate servers each load their own copy of the weights. One server across both cards loads the weights once, and the whole second card becomes cache. When a model nearly fills a GPU, splitting it across two gives far more room for requests than running two copies.
That is a statement about memory only. Tensor parallelism adds communication between the cards on every step, and whether the extra room turns into extra throughput is something the load test in step 8 decides. The two layouts also fail differently: with replicas, one server keeps answering if the other card fails.
"200 users" is not 200 simultaneous requests. People read, think and type between questions. The number that matters is how many requests are in progress at once:
average concurrency = users × requests per user per hour × seconds per request ÷ 3600
This is Little's law from queueing theory: the average number in a system is the arrival rate times the time each one spends there. A worked example with assumed inputs:
| Input | Value | Where it comes from |
|---|---|---|
| Users | 200 | People who will actually use it in a working day |
| Requests per user per hour | 30 | A guess to replace with your logs |
| Seconds per request | 8 | A guess to replace with a measurement |
| Average concurrency | 13.3 | 200 × 30 × 8 ÷ 3600 |
| Peak factor | 3 | An assumption: busy moments compared with the average |
| Peak concurrency to plan for | 40 | 13.3 × 3 |
The same inputs give the output volume. If an answer averages 400 tokens, 200 × 30 × 400 ÷ 3600 is about 670 tokens a second on average, and about 2,000 at peak.
The three guessed inputs dominate the result, so replace them as soon as you can. The proxy log from part 5 records every request with its duration and the user's name. After a week of a pilot with a few people, you have real numbers.
Automated clients change everything. A coding agent or a batch job sends requests back to back, with no reading time. Count each one as a user who never pauses.
Compare the peak concurrency with the memory table. For the example, 40 requests in progress at peak:
Real requests rarely fill the context, so the true capacity is higher than the worst case. Planning for the worst case is still right, because the alternative is preemption at the busiest moment: requests paused and recomputed exactly when everyone is waiting.
No formula predicts tokens per second for your model on your card with your prompts. vLLM ships a load generator for this. With a server running, a basic run looks like:
vllm bench serve \
--backend openai-chat \
--base-url http://127.0.0.1:8000 \
--endpoint /v1/chat/completions \
--model qwen3 \
--tokenizer Qwen/Qwen3-8B \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 256 \
--num-prompts 300 \
--max-concurrency 8 \
--percentile-metrics ttft,tpot,itl,e2el \
--metric-percentiles 50,95,99 \
--save-result
| Flag | Meaning |
|---|---|
--model, --tokenizer | The name the server answers to, and the real model whose tokenizer counts the tokens. They differ when you use --served-model-name. |
--random-input-len, --random-output-len | Prompt and answer length in tokens. Set them to what your users send and receive. |
--max-concurrency | How many requests are in progress at once. This is the dial you turn. |
--request-rate | The alternative dial: requests started per second, with random gaps like real traffic. The default sends everything at once. |
--percentile-metrics | Which timings to report: time to first token, time per output token, the gap between tokens, and end-to-end time. |
--save-result | Writes the results to a JSON file, so runs can be compared. |
The simplest place to run it is inside the server's container, with docker exec. For cleaner numbers, run it from a second machine and point --base-url at the server, so the load generator does not compete for the server's CPU.
The procedure. Decide first what "fast enough" means, for example a first token within 2 seconds for 95 percent of requests. Then repeat the run with --max-concurrency at 1, 2, 4, 8, 16, 32 and 64, and fill in a table:
| Concurrency | Output tokens per second | Time to first token, 95th percentile | Time per token, 95th percentile |
|---|---|---|---|
| 1 | measure | measure | measure |
| 8 | measure | measure | measure |
| 32 | measure | measure | measure |
The pattern to expect: total tokens per second climbs as concurrency rises, then flattens, while the waiting times grow. The highest concurrency that still meets your target is the capacity of that server. Call it C.
Read percentiles, not averages. An average hides the slow requests, and the slow requests are what people remember.
servers needed = peak concurrency ÷ C, rounded up
GPUs needed = servers × GPUs per server, plus one spare server if downtime matters
Continuing the example with a peak of 40. The values of C below are placeholders to show the arithmetic, not measurements:
| If one server meets the target up to C = | Servers for a peak of 40 |
|---|---|
| 8 | 5 |
| 16 | 3 |
| 32 | 2 |
Then apply the memory check from step 7 to each server, and take whichever requirement is larger.
The GPUs are not the whole machine:
| Lever | Effect | Cost |
|---|---|---|
Shorter --max-model-len | More requests fit in the same cache | Long inputs are rejected |
A cap on max_tokens | Requests finish sooner and free their slot | Long answers are cut |
| A quantized build | Smaller weights, so more cache or a smaller card | Lower precision |
| A smaller model | Less memory and more tokens per second | Weaker answers on hard tasks |
| Thinking off by default | Fewer output tokens per answer | Weaker multi-step reasoning |
One more is free. vLLM reuses the cache for prompts that start with the same text, such as a shared system prompt, and reports it in the vllm:prefix_cache_hits metric. Keeping the fixed part of your prompts at the start, and identical between requests, lets that work.
--dataset-name sharegpt and a dataset file is closer to real conversations.config.json, and the number of requests a card can hold.Last in the series: monitoring the server with Prometheus and Grafana.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
The memory formula and every figure in the tables were calculated from model configuration files read on Hugging Face on November 12, 2026; the script in this article was run on those files and its output is shown as printed. The 10 percent runtime reserve and the peak factor are assumptions. No GPU was available, so no throughput was measured and the benchmark commands are untested here. Flags follow the vLLM documentation. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.