Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
Wrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.
At a glancePick a Qwen model that fits your GPU, start it with the right vLLM flags, then use thinking mode, tool calling and structured output from the OpenAI client.
Not testedWritten from the Qwen and vLLM documentation. Nothing was run on a GPU, so the request examples are untested.
Qwen is a good first real model for your own server. The weights are openly licensed, the family runs from laptop-sized to datacenter-sized, and vLLM supports its two most useful extras: a reasoning mode you can switch on and off, and tool calling. This part serves Qwen3-8B and then uses every one of those features from a client.
These figures were read from each model's repository on Hugging Face. All three carry the Apache 2.0 license.
| Model | Weight files | Default context | Good for |
|---|---|---|---|
Qwen/Qwen2.5-7B-Instruct | 14.2 GiB | 32,768 tokens | General chat and tools on a 24 GB card |
Qwen/Qwen3-8B | 15.3 GiB | 40,960 tokens | The same, plus switchable thinking |
Qwen/Qwen2.5-Coder-32B-Instruct | 61.0 GiB | 32,768 tokens | Code, on an 80 GB card |
The weights are only part of the bill. Every token a request holds also needs working memory, called the KV cache. Part 6 derives the formula; for Qwen3-8B it comes to about 0.14 MiB per token, so one request that fills the default 40,960-token context needs about 5.6 GiB by itself.
That is why the default context is the first thing to change on a small card. With a tenth of the card kept back for the runtime, the estimate looks like this:
| GPU memory | Left for the cache | Full 8,192-token requests | Full 16,384-token requests |
|---|---|---|---|
| 24 GB | about 6 GiB | 5 | 2 |
| 48 GB | about 28 GiB | 24 | 12 |
| 80 GB | about 57 GiB | 50 | 25 |
These are worst cases, with every request at full length at once. Real requests are usually shorter, so more of them fit. Treat the table as a floor, and trust the number vLLM prints at startup over any estimate.
The Qwen documentation asks for vLLM 0.8.5 or newer for Qwen3. Using the Docker setup from part 2:
docker run -d --name qwen \
--runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 127.0.0.1:8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-8B \
--served-model-name qwen3 \
--max-model-len 16384 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser hermes
The last four flags are the ones specific to this model:
| Flag | What it does here |
|---|---|
--max-model-len 16384 | Caps prompt plus answer at 16,384 tokens, in place of the default 40,960. Raise it on a bigger card. |
--reasoning-parser qwen3 | Separates the model's thinking from its answer in the response. |
--enable-auto-tool-choice | Lets the model decide when to call a tool. vLLM only loads a tool parser when this flag is set. |
--tool-call-parser hermes | Reads Qwen's tool calls. Both the Qwen and vLLM documentation name hermes for Qwen2.5 and Qwen3-8B. |
Follow the log with docker logs -f qwen. Near the end of startup, look for two lines of this form, which the vLLM documentation shows as the thing to check:
GPU KV cache size: 643,232 tokens
Maximum concurrency for 40,960 tokens per request: 15.70x
Your numbers will differ. The first is the real size of the cache on your card. The second divides it by your context length, which is the measured version of the table in step 1. If it is below 1, the server cannot hold even one full-length request, and you should lower --max-model-len.
curl -s http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3",
"messages": [
{"role": "user", "content": "Explain what a KV cache is in two sentences."}
],
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20,
"max_tokens": 2048
}'
The sampling values are the ones the Qwen documentation uses for thinking mode. Send them with every request. vLLM takes its defaults from the model's generation_config.json, so the answers you get without them depend on a file you did not write.
The reply has two parts inside choices[0].message: the answer in content, and the thinking in a separate field.
A naming trap. The vLLM documentation says that field is now called reasoning and used to be called reasoning_content. The Qwen page still uses the old name. If your client reads the old field and always finds it empty, this is why.
Qwen3 thinks before it answers unless told not to. Thinking helps on multi-step problems and wastes time and tokens on simple ones. You can decide per request:
curl -s http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3",
"messages": [{"role": "user", "content": "Translate to French: good morning"}],
"temperature": 0.7,
"top_p": 0.8,
"top_k": 20,
"presence_penalty": 1.5,
"max_tokens": 512,
"chat_template_kwargs": {"enable_thinking": false}
}'
Note that the sampling values changed too. The Qwen documentation recommends different settings for the two modes: temperature 0.6 and top_p 0.95 with thinking, temperature 0.7 and top_p 0.8 without.
From the OpenAI Python client, fields that are not part of the OpenAI API go in extra_body:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="qwen3",
messages=[{"role": "user", "content": "Translate to French: good morning"}],
temperature=0.7,
top_p=0.8,
presence_penalty=1.5,
max_tokens=512,
extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)
If most of your traffic does not need thinking, make that the server's default and let individual requests turn it on. Add this to the server arguments:
--default-chat-template-kwargs '{"enable_thinking": false}'
The vLLM documentation states that a request's own chat_template_kwargs always wins over the server default.
Tool calling is a conversation in three steps. You describe the functions. The model answers with the name and arguments of the one it wants. Your code runs it and sends back the result, and the model writes the final answer. The model never runs anything itself.
import json
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
def get_weather(location: str, unit: str) -> str:
return f"18 degrees {unit} and cloudy in {location}" # replace with a real lookup
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City and country"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location", "unit"],
},
},
}]
messages = [{"role": "user", "content": "What is the weather in Seoul, in celsius?"}]
first = client.chat.completions.create(model="qwen3", messages=messages, tools=tools, tool_choice="auto")
reply = first.choices[0].message
if reply.tool_calls:
messages.append(reply)
for call in reply.tool_calls:
arguments = json.loads(call.function.arguments)
result = get_weather(**arguments)
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
final = client.chat.completions.create(model="qwen3", messages=messages, tools=tools)
print(final.choices[0].message.content)
else:
print(reply.content)
The first half follows the example in the vLLM documentation; the second half sends the result back in the standard OpenAI message format. Three details decide whether this works in practice:
tool_choice has four settings: auto lets the model decide, required forces some tool call, none forbids them, and naming one function forces that function.Other members of the family need a different parser. The vLLM documentation lists hermes for Qwen/Qwen2.5-* and Qwen/QwQ-32B, and qwen3_xml for the Qwen3-Coder models. The wrong parser does not raise an error. The tool call simply arrives as plain text in content.
When a program reads the answer, "usually valid" is not good enough. vLLM can constrain generation so the output must match a list of choices, a regular expression or a JSON schema. The parameters go in extra_body:
completion = client.chat.completions.create(
model="qwen3",
messages=[{"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}],
extra_body={
"structured_outputs": {"choice": ["positive", "negative"]},
"chat_template_kwargs": {"enable_thinking": False},
},
)
print(completion.choices[0].message.content)
Replace choice with json and a JSON schema to get an object in a fixed shape. The Qwen documentation adds one piece of advice: also describe the format you want in the prompt. The constraint guarantees the shape, and the prompt helps the model fill it sensibly.
Quantized builds. Qwen publishes pre-quantized versions that take less memory at some cost in precision. Only the name changes:
--model Qwen/Qwen3-8B-AWQ
--model Qwen/Qwen3-8B-FP8
The Qwen documentation notes that the FP8 builds run natively on newer GPU generations such as Ada Lovelace and Hopper, and that since vLLM 0.9.0 they also run on Ampere cards, which includes the A100, in a mixed mode.
Longer context. Qwen3 was trained with contexts up to 32,768 tokens. For much longer inputs the Qwen documentation describes a technique called YaRN, enabled with a --rope-scaling argument alongside a larger --max-model-len. It also warns that vLLM applies the scaling at every length, which can cost quality on short texts. Use it only when you really need long inputs; the exact argument is on the Qwen page.
| What you see | Cause | Fix |
|---|---|---|
| Out of memory at startup | The default 40,960-token context is too large for the card | Lower --max-model-len; this is the Qwen documentation's first suggestion |
| The tool call shows up as text | Tool flags missing, or the wrong parser | Check --enable-auto-tool-choice and the parser for your model |
| Thinking text appears inside the answer | No reasoning parser | Add --reasoning-parser qwen3 |
| The reasoning field is always empty | The client reads the old field name | Read reasoning, not reasoning_content |
| The answer stops before it starts | Thinking used up max_tokens | Raise max_tokens, or turn thinking off for that request |
| Answers differ from another tool running the same model | Different sampling defaults | Send temperature, top_p and top_k explicitly |
Qwen ships everything vLLM needs. The next part takes a model that does not: Phind-CodeLlama-34B on A100 GPUs, where you write the chat template yourself.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
Flags, parser names and sampling values come from the Qwen and vLLM documentation as read on November 12, 2026. Model sizes and layer counts were read from each model's files on Hugging Face, and the memory figures were calculated from them. Nothing was run on a GPU, so the request examples are untested here. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.