How many GPUs for N users? A vLLM sizing method
Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
At a glanceWrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.
Partly testedThe chat template was rendered and matched the model card. The model was not loaded: no A100 was available, so memory figures are calculated.
Qwen in part 3 worked out of the box. Many models do not. They were published before today's conventions, with a prompt format described in a README and nothing a server can read. Phind-CodeLlama-34B-v2 is a clean example: a strong coding model from 2023 with no chat template, weights in an older file format, and a size that makes GPU planning matter. Wrapping it teaches every step you need for any model that does not just work.
One honest note before you spend a day on it. Newer coding models exist, and Qwen2.5-Coder-32B from part 3 needs about the same hardware. Choose Phind if you already rely on it. Otherwise read this part for the method.
Four small files in the repository tell you almost everything. Open them in the browser on the model's "Files" tab.
| File | What it says | What that means for you |
|---|---|---|
README.md | License llama2. Trained with a sequence length of 4,096 tokens. Uses the Alpaca/Vicuna prompt format. | Read the Llama 2 license terms before commercial use. Plan for a 4,096-token context. |
config.json | 48 layers, hidden size 8,192, 64 attention heads, 8 key-value heads, 16-bit weights, 16,384 positions | These numbers give the memory per token in step 2. |
tokenizer_config.json | No chat_template entry. Maximum length 4,096. | The chat endpoint will not work until you supply a template. |
| File list | Seven pytorch_model-*.bin files, 67.5 GB in total, last changed in August 2023 | An older weight format. vLLM's default loader tries safetensors first and falls back to these. |
The .bin format deserves one caution. It is based on Python's pickle, which can run code when a file is loaded. Only load such files from a publisher you trust, and download them from the original repository, not a mirror.
The repository is not gated, so no access request or token is needed.
Two things must fit in GPU memory: the weights, and the KV cache that holds each request's tokens while it is being answered. The weights are 67.5 GB on disk, which is 62.9 GiB in the units GPUs report.
The cache cost per token follows from the config: 2 (a key and a value) × 48 layers × 8 key-value heads × 128 dimensions per head × 2 bytes = 196,608 bytes, about 0.19 MiB. A request that fills the 4,096-token context holds about 0.75 GiB.
With a tenth of each card kept back for the runtime, which is an assumption and not a vLLM setting, the layouts compare like this:
| Layout | Left for the cache | Full 4,096-token requests at once | Notes |
|---|---|---|---|
| 1 × A100 80GB | about 9 GiB | 12 | The simplest option |
| 2 × A100 40GB | about 9 GiB | 12 | Needs --tensor-parallel-size 2 |
| 2 × A100 80GB | about 81 GiB | 108 | One copy of the weights, a much larger cache |
| 1 × A100 40GB, AWQ build | about 19 GiB | 25 | 4-bit weights: 17 GiB, lower precision |
The third row is the one worth remembering. Doubling the GPUs raises the cache ninefold, because the weights are paid for once and everything else goes to requests. Part 6 turns this into a method for any model.
The model does not fit on a single 40 GB card at full precision. The fourth row uses a community-quantized build, TheBloke/Phind-CodeLlama-34B-v2-AWQ. The vLLM documentation says quantized models from the Hub can be used directly, at the cost of lower precision. It is a different publisher from the original, so judge it as you would any third-party file.
A chat request arrives as a list of messages with roles. The model was trained on plain text in one specific layout. The chat template is the translation between the two, written in the Jinja language. The vLLM documentation is direct about what happens without one: "the server will not be able to process chat and all chat requests will error."
The model card shows the layout the model expects:
### System Prompt
You are an intelligent programming assistant.
### User Message
Implement a linked list in C++
### Assistant
...
Save the following as phind.jinja. It writes each message under the right heading and, when the server asks for a reply, ends with the assistant heading so the model continues from there:
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
### System Prompt
{{ message['content'] }}
{% elif message['role'] == 'user' -%}
### User Message
{{ message['content'] }}
{% elif message['role'] == 'assistant' -%}
### Assistant
{{ message['content'] }}
{% endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
### Assistant
{% endif -%}
The hyphens inside {%- and -%} remove the whitespace next to a tag. They are what keeps the blank lines exactly where the model card has them. A model is sensitive to this: a missing newline will not cause an error, only worse answers that are hard to trace back.
You do not need the model to check a template. Render it on any computer with Python and compare the text with the model card. Save this as test_template.py next to the template:
import jinja2
template = jinja2.Environment().from_string(open("phind.jinja").read())
messages = [
{"role": "system", "content": "You are an intelligent programming assistant."},
{"role": "user", "content": "Implement a linked list in C++"},
]
text = template.render(messages=messages, add_generation_prompt=True)
expected = (
"### System Prompt\nYou are an intelligent programming assistant.\n\n"
"### User Message\nImplement a linked list in C++\n\n"
"### Assistant\n"
)
print(repr(text))
assert text == expected, "the template does not match the model card"
print("template matches the model card")
pip install jinja2
python test_template.py
This check was run for this article and passed, including a second test with an earlier assistant turn and a follow-up question. Printing with repr matters, because it makes every newline visible. Keep the script. Each time you edit the template, run it again before restarting a server that takes minutes to load.
From the folder that holds phind.jinja, on a machine with one A100 80GB:
docker run -d --name phind \
--runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v "$PWD/phind.jinja:/templates/phind.jinja:ro" \
-p 127.0.0.1:8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model Phind/Phind-CodeLlama-34B-v2 \
--served-model-name phind-codellama \
--chat-template /templates/phind.jinja \
--max-model-len 4096
| Flag | Why |
|---|---|
The second -v | Puts your template inside the container, read-only. |
--chat-template | The path to the template as the container sees it. The reference also accepts the template as a single-line string. |
--max-model-len 4096 | Matches the length the model was trained and tested with. The config would allow more, but nothing in the model card supports it. |
--served-model-name | A short, stable name for clients. |
For two GPUs, add one flag:
--tensor-parallel-size 2
Tensor parallelism splits every layer across the cards. The vLLM documentation recommends it when a model is too large for one GPU, or when you want more room for the cache. To use only some of a machine's GPUs, select them in Docker with --gpus '"device=0,1"'. The size must match the number of GPUs the container can see.
Expect a slow first start. The download is 67.5 GB, and loading reads all of it from disk. The documentation adds that with tensor parallelism each worker reads the whole model, so loading takes longer in proportion. Keep the cache on a local disk.
Follow the log with docker logs -f phind and look for the two lines that report the real cache size:
GPU KV cache size: ... tokens
Maximum concurrency for 4,096 tokens per request: ...x
Compare the second number with the table in step 2. It should be in the same range. If it is far lower, something else is using the card; check nvidia-smi.
Then ask a question:
curl -s http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "phind-codellama",
"messages": [
{"role": "system", "content": "You are an intelligent programming assistant."},
{"role": "user", "content": "Write a Python function that checks whether a string is a palindrome."}
],
"temperature": 0.1,
"top_p": 0.75,
"max_tokens": 512
}'
The temperature and top_p values are the ones the model card uses in its own evaluation code.
Judge the answer by its shape. It should be code and a short explanation, and it should stop. Two failure patterns point straight at the template:
"stop": ["### User Message"] to requests as a guard.--chat-template and the mount.Any client that lets you set an OpenAI-compatible base URL can use the server. It needs three values: the base URL ending in /v1, the model name phind-codellama, and an API key, which can be any text until you configure one.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
stream = client.chat.completions.create(
model="phind-codellama",
messages=[{"role": "user", "content": "Explain what this regex matches: ^\\d{3}-\\d{4}$"}],
max_tokens=400,
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
Streaming matters for a coding assistant. People tolerate a long answer that starts at once far better than a short one that appears after a silent wait.
Know two limits before you promise this to a team:
The server is still bound to the local machine. Opening it to a team safely, with keys and a proxy, is the subject of part 5.
| What you see | Cause | Fix |
|---|---|---|
| Chat requests return an error about a chat template | The template was not loaded | Check the mount and the --chat-template path |
| Out of memory while loading | The layout is too small, or the card is shared | Step 2; check nvidia-smi for other processes |
| An error about the number of GPUs | --tensor-parallel-size does not match the visible GPUs | Make the two agree |
| Loading seems stuck | 67.5 GB is being read, once per worker with tensor parallelism | Wait, and watch disk activity; use a local disk |
| A request is rejected as too long | Prompt plus max_tokens exceeds 4,096 | Shorten the input or lower max_tokens |
| Answers run on into invented turns | Template whitespace, or no stop sequence | Steps 4 and 6 |
Next: serving many users from one server.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
Model facts were read from the Phind-CodeLlama-34B-v2 repository on Hugging Face on November 12, 2026. The chat template in this article was rendered with Jinja 3.1.6 and its output matched the model card's prompt format exactly. The memory figures are calculated, not measured: no A100 was available and the model was not loaded. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.