Seongwoo Choi / Journal

How many GPUs for N users? A vLLM sizing method

At a glanceWork out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.

Partly testedThe sizing script was run on real model files and its output is shown. No GPU was used, so nothing here is a measured speed.

Series · part 6 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 12 sections
  1. 1. Two questions, answered separately
  2. 2. Memory, step one: the weights
  3. 3. Memory, step two: the cost of one token
  4. 4. Memory, step three: the budget
  5. 5. The result that surprises people
  6. 6. From users to requests at the same moment
  7. 7. First check: does the peak fit in memory?
  8. 8. Second check: measure the speed
  9. 9. Turn the measurements into a GPU count
  10. 10. Five ways to need fewer GPUs
  11. 11. Where this method stops
  12. 12. What you can do now

"How many GPUs do we need for 200 users?" has no answer on a vendor's page, because it depends on your model, your prompts and how patient your users are. It does have a method. Part of it is arithmetic you can do before buying anything, and part is a measurement you must make on real hardware. This part gives you both, with every number shown.

1. Two questions, answered separately

QuestionLimited byHow you answer it
Does it fit?GPU memoryArithmetic, from two files
Is it fast enough?GPU computationA load test

People often stop after the first question. A setup that fits can still be far too slow for the load, and a setup that is fast can still pause requests because memory runs out. You need both answers, and the larger GPU count wins.

2. Memory, step one: the weights

Add up the weight files in the model's repository. That is close to what they occupy on the GPU when the model runs at the precision it was published in.

Mind the units. File listings use gigabytes of 1,000³ bytes. GPU tools report mebibytes, and a card sold as "80 GB" shows about 80 × 1,024 MiB. To compare, divide bytes by 1,073,741,824 to get GiB. Phind-CodeLlama-34B is 67.5 GB on disk, which is 62.9 GiB.

3. Memory, step two: the cost of one token

While a request is being answered, the model keeps two tensors, a key and a value, for every token of that request in every layer. This is the KV cache. Its size per token comes straight from config.json:

bytes per token = 2 × layers × key-value heads × head dimension × bytes per number
TermField in config.json
layersnum_hidden_layers
key-value headsnum_key_value_heads, or num_attention_heads if the first is absent
head dimensionhead_dim, or hidden_size ÷ num_attention_heads
bytes per number2 when torch_dtype is bfloat16 or float16

Applied to four models from this series:

ModelLayers × KV heads × head dimPer tokenOne full default context
Qwen2.5-7B-Instruct28 × 4 × 12857,344 bytes1.8 GiB at 32,768 tokens
Qwen3-8B36 × 8 × 128147,456 bytes5.6 GiB at 40,960 tokens
Phind-CodeLlama-34B-v248 × 8 × 128196,608 bytes0.75 GiB at 4,096 tokens
Qwen2.5-Coder-32B-Instruct64 × 8 × 128262,144 bytes8.0 GiB at 32,768 tokens

Two things stand out. Qwen3-8B is close to Qwen2.5-7B in size but needs more than twice the cache per token, because it has more layers and twice the key-value heads. And context length matters as much as the model: one full-length Qwen2.5-Coder request costs more memory than ten full-length Phind requests.

4. Memory, step three: the budget

cache memory = GPU memory − weights − runtime reserve
tokens in cache = cache memory ÷ bytes per token
full-length requests at once = tokens in cache ÷ context length

The runtime reserve covers what vLLM needs besides weights and cache: the working memory of a forward pass and its compiled graphs. On NVIDIA GPUs, the documentation says vLLM measures this itself at startup and gives the rest to the cache, so the true figure is only known on the machine. For planning, this series keeps back a tenth of the card. That is an assumption, chosen to be cautious.

This script does the arithmetic. Download a model's config.json and save the script as kv_budget.py:

import json, sys

cfg = json.load(open(sys.argv[1]))   # the model's config.json
gpu_gib = float(sys.argv[2])         # GPU memory in GiB, added up over all GPUs used
weights_gib = float(sys.argv[3])     # total size of the weight files in GiB
reserve = 0.10                       # share kept back for the runtime: an assumption

layers = cfg["num_hidden_layers"]
kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
head_dim = cfg.get("head_dim", cfg["hidden_size"] // cfg["num_attention_heads"])
bytes_per_value = 2                  # 16-bit values (bfloat16 or float16)

per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
cache_gib = gpu_gib * (1 - reserve) - weights_gib
tokens = cache_gib * 2**30 / per_token

print(f"KV cache per token: {per_token:,} bytes")
print(f"Left for the cache: {cache_gib:.1f} GiB, about {tokens:,.0f} tokens")
for length in (4096, 8192, 16384, 32768):
    print(f"  full requests of {length:,} tokens: {tokens / length:.1f}")

Run for Phind-CodeLlama-34B on one 80 GB card, it printed:

$ python kv_budget.py config.json 80 62.85
KV cache per token: 196,608 bytes
Left for the cache: 9.1 GiB, about 49,971 tokens
  full requests of 4,096 tokens: 12.2
  full requests of 8,192 tokens: 6.1
  full requests of 16,384 tokens: 3.0
  full requests of 32,768 tokens: 1.5

If "Left for the cache" is negative, the model does not fit on that hardware.

Check it against the server. When vLLM starts, it logs the real numbers, in lines of this form from the documentation:

GPU KV cache size: 643,232 tokens
Maximum concurrency for 40,960 tokens per request: 15.70x

The first is the measured version of "tokens in cache". The second divides it by your --max-model-len. The documentation's own advice is to add GPUs if these are lower than you need. Once you have a machine, trust these lines over any estimate, including this article's.

5. The result that surprises people

Compare three ways to run Phind-CodeLlama-34B on A100 80GB cards, using the same script:

LayoutCache memoryFull 4,096-token requests
1 GPU9.1 GiB12
2 GPUs as two separate servers2 × 9.1 GiB24
2 GPUs as one server with tensor parallelism81.2 GiB108

1 GPU, 80 GiB12 full requests

weights

2 GPUs as two separate servers, 160 GiB24 full requests

weightsweights again

2 GPUs as one server, 160 GiB108 full requests

weightscache for requests

Model weightsKV cache, the room for requestsRuntime reserve

Phind-CodeLlama-34B on A100 80GB cards, drawn to scale from the figures in the table.

Two separate servers each load their own copy of the weights. One server across both cards loads the weights once, and the whole second card becomes cache. When a model nearly fills a GPU, splitting it across two gives far more room for requests than running two copies.

That is a statement about memory only. Tensor parallelism adds communication between the cards on every step, and whether the extra room turns into extra throughput is something the load test in step 8 decides. The two layouts also fail differently: with replicas, one server keeps answering if the other card fails.

6. From users to requests at the same moment

"200 users" is not 200 simultaneous requests. People read, think and type between questions. The number that matters is how many requests are in progress at once:

average concurrency = users × requests per user per hour × seconds per request ÷ 3600

This is Little's law from queueing theory: the average number in a system is the arrival rate times the time each one spends there. A worked example with assumed inputs:

InputValueWhere it comes from
Users200People who will actually use it in a working day
Requests per user per hour30A guess to replace with your logs
Seconds per request8A guess to replace with a measurement
Average concurrency13.3200 × 30 × 8 ÷ 3600
Peak factor3An assumption: busy moments compared with the average
Peak concurrency to plan for4013.3 × 3

The same inputs give the output volume. If an answer averages 400 tokens, 200 × 30 × 400 ÷ 3600 is about 670 tokens a second on average, and about 2,000 at peak.

The three guessed inputs dominate the result, so replace them as soon as you can. The proxy log from part 5 records every request with its duration and the user's name. After a week of a pilot with a few people, you have real numbers.

Automated clients change everything. A coding agent or a batch job sends requests back to back, with no reading time. Count each one as a user who never pauses.

7. First check: does the peak fit in memory?

Compare the peak concurrency with the memory table. For the example, 40 requests in progress at peak:

  • One A100 80GB holds about 12 full-length Phind requests. Not enough if requests are long.
  • Two A100 80GB with tensor parallelism hold about 108. Enough, with room to spare.

Real requests rarely fill the context, so the true capacity is higher than the worst case. Planning for the worst case is still right, because the alternative is preemption at the busiest moment: requests paused and recomputed exactly when everyone is waiting.

8. Second check: measure the speed

No formula predicts tokens per second for your model on your card with your prompts. vLLM ships a load generator for this. With a server running, a basic run looks like:

vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions \
  --model qwen3 \
  --tokenizer Qwen/Qwen3-8B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 300 \
  --max-concurrency 8 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,95,99 \
  --save-result
FlagMeaning
--model, --tokenizerThe name the server answers to, and the real model whose tokenizer counts the tokens. They differ when you use --served-model-name.
--random-input-len, --random-output-lenPrompt and answer length in tokens. Set them to what your users send and receive.
--max-concurrencyHow many requests are in progress at once. This is the dial you turn.
--request-rateThe alternative dial: requests started per second, with random gaps like real traffic. The default sends everything at once.
--percentile-metricsWhich timings to report: time to first token, time per output token, the gap between tokens, and end-to-end time.
--save-resultWrites the results to a JSON file, so runs can be compared.

The simplest place to run it is inside the server's container, with docker exec. For cleaner numbers, run it from a second machine and point --base-url at the server, so the load generator does not compete for the server's CPU.

The procedure. Decide first what "fast enough" means, for example a first token within 2 seconds for 95 percent of requests. Then repeat the run with --max-concurrency at 1, 2, 4, 8, 16, 32 and 64, and fill in a table:

ConcurrencyOutput tokens per secondTime to first token, 95th percentileTime per token, 95th percentile
1measuremeasuremeasure
8measuremeasuremeasure
32measuremeasuremeasure

The pattern to expect: total tokens per second climbs as concurrency rises, then flattens, while the waiting times grow. The highest concurrency that still meets your target is the capacity of that server. Call it C.

Read percentiles, not averages. An average hides the slow requests, and the slow requests are what people remember.

9. Turn the measurements into a GPU count

servers needed = peak concurrency ÷ C, rounded up
GPUs needed    = servers × GPUs per server, plus one spare server if downtime matters

Continuing the example with a peak of 40. The values of C below are placeholders to show the arithmetic, not measurements:

If one server meets the target up to C =Servers for a peak of 40
85
163
322

Then apply the memory check from step 7 to each server, and take whichever requirement is larger.

The GPUs are not the whole machine:

  • CPU: the documentation recommends at least 2 + N physical cores for N GPUs, and notes that a GPU that is never fully busy often means the CPU is the limit.
  • RAM: weights pass through system memory while loading. Plan for at least the size of the weight files; this is a rule of thumb, not a documented figure.
  • Disk: the weight files plus the container image, on a local disk.

10. Five ways to need fewer GPUs

LeverEffectCost
Shorter --max-model-lenMore requests fit in the same cacheLong inputs are rejected
A cap on max_tokensRequests finish sooner and free their slotLong answers are cut
A quantized buildSmaller weights, so more cache or a smaller cardLower precision
A smaller modelLess memory and more tokens per secondWeaker answers on hard tasks
Thinking off by defaultFewer output tokens per answerWeaker multi-step reasoning

One more is free. vLLM reuses the cache for prompts that start with the same text, such as a shared system prompt, and reports it in the vllm:prefix_cache_hits metric. Keeping the fixed part of your prompts at the start, and identical between requests, lets that work.

11. Where this method stops

  • The memory arithmetic covers standard transformer models with 16-bit weights. Models with unusual attention layouts, images or audio need their own check.
  • Random benchmark prompts never repeat, so they understate the benefit of prefix caching. A run with --dataset-name sharegpt and a dataset file is closer to real conversations.
  • A benchmark measures one moment. After launch, the dashboards in part 7 tell you whether the plan still holds.

12. What you can do now

  • Compute a model's memory per token from its config.json, and the number of requests a card can hold.
  • Explain why one server on two GPUs can hold several times more requests than two servers on the same cards.
  • Turn "N users" into a peak concurrency, and say which inputs are guesses.
  • Run a load test, read it by percentile, and turn the result into a GPU count.

Last in the series: monitoring the server with Prometheus and Grafana.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

The memory formula and every figure in the tables were calculated from model configuration files read on Hugging Face on November 12, 2026; the script in this article was run on those files and its output is shown as printed. The 10 percent runtime reserve and the peak factor are assumptions. No GPU was available, so no throughput was measured and the benchmark commands are untested here. Flags follow the vLLM documentation. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy