Seongwoo Choi / Journal

Serve Qwen with vLLM: chat, tool calling and thinking

At a glancePick a Qwen model that fits your GPU, start it with the right vLLM flags, then use thinking mode, tool calling and structured output from the OpenAI client.

Not testedWritten from the Qwen and vLLM documentation. Nothing was run on a GPU, so the request examples are untested.

You need
The Docker setup from part 2 and one NVIDIA GPU with 24 GB of memory or more.
You end up with
A Qwen3 server that chats, shows or hides its reasoning, and calls your functions.
Series · part 3 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 9 sections
  1. 1. Choose a model that fits
  2. 2. Start the server
  3. 3. Send a first request
  4. 4. Turn thinking on and off
  5. 5. Let the model call your functions
  6. 6. Force an answer into a shape
  7. 7. Smaller and longer: quantized builds and long context
  8. 8. When something looks wrong
  9. 9. What you can do now

Qwen is a good first real model for your own server. The weights are openly licensed, the family runs from laptop-sized to datacenter-sized, and vLLM supports its two most useful extras: a reasoning mode you can switch on and off, and tool calling. This part serves Qwen3-8B and then uses every one of those features from a client.

1. Choose a model that fits

These figures were read from each model's repository on Hugging Face. All three carry the Apache 2.0 license.

ModelWeight filesDefault contextGood for
Qwen/Qwen2.5-7B-Instruct14.2 GiB32,768 tokensGeneral chat and tools on a 24 GB card
Qwen/Qwen3-8B15.3 GiB40,960 tokensThe same, plus switchable thinking
Qwen/Qwen2.5-Coder-32B-Instruct61.0 GiB32,768 tokensCode, on an 80 GB card

The weights are only part of the bill. Every token a request holds also needs working memory, called the KV cache. Part 6 derives the formula; for Qwen3-8B it comes to about 0.14 MiB per token, so one request that fills the default 40,960-token context needs about 5.6 GiB by itself.

That is why the default context is the first thing to change on a small card. With a tenth of the card kept back for the runtime, the estimate looks like this:

GPU memoryLeft for the cacheFull 8,192-token requestsFull 16,384-token requests
24 GBabout 6 GiB52
48 GBabout 28 GiB2412
80 GBabout 57 GiB5025

These are worst cases, with every request at full length at once. Real requests are usually shorter, so more of them fit. Treat the table as a floor, and trust the number vLLM prints at startup over any estimate.

2. Start the server

The Qwen documentation asks for vLLM 0.8.5 or newer for Qwen3. Using the Docker setup from part 2:

docker run -d --name qwen \
  --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 127.0.0.1:8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B \
  --served-model-name qwen3 \
  --max-model-len 16384 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The last four flags are the ones specific to this model:

FlagWhat it does here
--max-model-len 16384Caps prompt plus answer at 16,384 tokens, in place of the default 40,960. Raise it on a bigger card.
--reasoning-parser qwen3Separates the model's thinking from its answer in the response.
--enable-auto-tool-choiceLets the model decide when to call a tool. vLLM only loads a tool parser when this flag is set.
--tool-call-parser hermesReads Qwen's tool calls. Both the Qwen and vLLM documentation name hermes for Qwen2.5 and Qwen3-8B.

Follow the log with docker logs -f qwen. Near the end of startup, look for two lines of this form, which the vLLM documentation shows as the thing to check:

GPU KV cache size: 643,232 tokens
Maximum concurrency for 40,960 tokens per request: 15.70x

Your numbers will differ. The first is the real size of the cache on your card. The second divides it by your context length, which is the measured version of the table in step 1. If it is below 1, the server cannot hold even one full-length request, and you should lower --max-model-len.

3. Send a first request

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [
      {"role": "user", "content": "Explain what a KV cache is in two sentences."}
    ],
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "max_tokens": 2048
  }'

The sampling values are the ones the Qwen documentation uses for thinking mode. Send them with every request. vLLM takes its defaults from the model's generation_config.json, so the answers you get without them depend on a file you did not write.

The reply has two parts inside choices[0].message: the answer in content, and the thinking in a separate field.

A naming trap. The vLLM documentation says that field is now called reasoning and used to be called reasoning_content. The Qwen page still uses the old name. If your client reads the old field and always finds it empty, this is why.

4. Turn thinking on and off

Qwen3 thinks before it answers unless told not to. Thinking helps on multi-step problems and wastes time and tokens on simple ones. You can decide per request:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [{"role": "user", "content": "Translate to French: good morning"}],
    "temperature": 0.7,
    "top_p": 0.8,
    "top_k": 20,
    "presence_penalty": 1.5,
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Note that the sampling values changed too. The Qwen documentation recommends different settings for the two modes: temperature 0.6 and top_p 0.95 with thinking, temperature 0.7 and top_p 0.8 without.

From the OpenAI Python client, fields that are not part of the OpenAI API go in extra_body:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="qwen3",
    messages=[{"role": "user", "content": "Translate to French: good morning"}],
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    max_tokens=512,
    extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

If most of your traffic does not need thinking, make that the server's default and let individual requests turn it on. Add this to the server arguments:

--default-chat-template-kwargs '{"enable_thinking": false}'

The vLLM documentation states that a request's own chat_template_kwargs always wins over the server default.

5. Let the model call your functions

Tool calling is a conversation in three steps. You describe the functions. The model answers with the name and arguments of the one it wants. Your code runs it and sends back the result, and the model writes the final answer. The model never runs anything itself.

import json
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

def get_weather(location: str, unit: str) -> str:
    return f"18 degrees {unit} and cloudy in {location}"   # replace with a real lookup

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City and country"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["location", "unit"],
        },
    },
}]

messages = [{"role": "user", "content": "What is the weather in Seoul, in celsius?"}]
first = client.chat.completions.create(model="qwen3", messages=messages, tools=tools, tool_choice="auto")
reply = first.choices[0].message

if reply.tool_calls:
    messages.append(reply)
    for call in reply.tool_calls:
        arguments = json.loads(call.function.arguments)
        result = get_weather(**arguments)
        messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
    final = client.chat.completions.create(model="qwen3", messages=messages, tools=tools)
    print(final.choices[0].message.content)
else:
    print(reply.content)

The first half follows the example in the vLLM documentation; the second half sends the result back in the standard OpenAI message format. Three details decide whether this works in practice:

  • Descriptions are the interface. The model chooses a tool from its name, description and parameter descriptions. Write them as carefully as you would a prompt.
  • Arguments are text until you parse them. Validate them before running anything, especially for tools that write or delete.
  • tool_choice has four settings: auto lets the model decide, required forces some tool call, none forbids them, and naming one function forces that function.

Other members of the family need a different parser. The vLLM documentation lists hermes for Qwen/Qwen2.5-* and Qwen/QwQ-32B, and qwen3_xml for the Qwen3-Coder models. The wrong parser does not raise an error. The tool call simply arrives as plain text in content.

6. Force an answer into a shape

When a program reads the answer, "usually valid" is not good enough. vLLM can constrain generation so the output must match a list of choices, a regular expression or a JSON schema. The parameters go in extra_body:

completion = client.chat.completions.create(
    model="qwen3",
    messages=[{"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}],
    extra_body={
        "structured_outputs": {"choice": ["positive", "negative"]},
        "chat_template_kwargs": {"enable_thinking": False},
    },
)
print(completion.choices[0].message.content)

Replace choice with json and a JSON schema to get an object in a fixed shape. The Qwen documentation adds one piece of advice: also describe the format you want in the prompt. The constraint guarantees the shape, and the prompt helps the model fill it sensibly.

7. Smaller and longer: quantized builds and long context

Quantized builds. Qwen publishes pre-quantized versions that take less memory at some cost in precision. Only the name changes:

--model Qwen/Qwen3-8B-AWQ
--model Qwen/Qwen3-8B-FP8

The Qwen documentation notes that the FP8 builds run natively on newer GPU generations such as Ada Lovelace and Hopper, and that since vLLM 0.9.0 they also run on Ampere cards, which includes the A100, in a mixed mode.

Longer context. Qwen3 was trained with contexts up to 32,768 tokens. For much longer inputs the Qwen documentation describes a technique called YaRN, enabled with a --rope-scaling argument alongside a larger --max-model-len. It also warns that vLLM applies the scaling at every length, which can cost quality on short texts. Use it only when you really need long inputs; the exact argument is on the Qwen page.

8. When something looks wrong

What you seeCauseFix
Out of memory at startupThe default 40,960-token context is too large for the cardLower --max-model-len; this is the Qwen documentation's first suggestion
The tool call shows up as textTool flags missing, or the wrong parserCheck --enable-auto-tool-choice and the parser for your model
Thinking text appears inside the answerNo reasoning parserAdd --reasoning-parser qwen3
The reasoning field is always emptyThe client reads the old field nameRead reasoning, not reasoning_content
The answer stops before it startsThinking used up max_tokensRaise max_tokens, or turn thinking off for that request
Answers differ from another tool running the same modelDifferent sampling defaultsSend temperature, top_p and top_k explicitly

9. What you can do now

  • Pick a Qwen model and a context length from your GPU's memory, and confirm the choice in the startup log.
  • Switch thinking per request or per server, with the right sampling values for each mode.
  • Run a complete tool-calling loop, and know which parser each Qwen family needs.
  • Constrain output to a fixed set of answers or a schema.

Qwen ships everything vLLM needs. The next part takes a model that does not: Phind-CodeLlama-34B on A100 GPUs, where you write the chat template yourself.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

Flags, parser names and sampling values come from the Qwen and vLLM documentation as read on November 12, 2026. Model sizes and layer counts were read from each model's files on Hugging Face, and the memory figures were calculated from them. Nothing was run on a GPU, so the request examples are untested here. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy