Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
Wrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.
At a glanceInstall vLLM, start an OpenAI-compatible server with one command, call it from Python, and run it in Docker. With the flags that control GPU memory.
Not testedWritten from the vLLM documentation. Nothing was run: there was no GPU.
vLLM is a server for running open-weight language models on your own GPUs. It speaks the OpenAI API protocol, so an application written for the OpenAI client can point at a vLLM server by changing one URL. This guide gets a first server running and covers the handful of settings that decide whether a model fits.
The quickstart lists Linux and Python 3.11 to 3.14 as prerequisites. On a Mac with Apple Silicon, the documentation points to a separate build called vLLM-Metal. A server hosts one model at a time, so plan one server per model.
If the goal is to try a model on a laptop, a desktop tool is simpler; see what to check before installing Ollama on Windows. vLLM is the choice when several users or programs need to call the same model at once.
For NVIDIA GPUs the documentation recommends uv to create the environment:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
--torch-backend=auto lets uv pick the PyTorch build that matches the installed CUDA driver. vllm download-kernels fetches precompiled kernels in advance. Without it vLLM downloads and compiles them at startup, which the documentation says can add several minutes on newer GPUs.
One command downloads the model from Hugging Face and serves it:
vllm serve Qwen/Qwen2.5-1.5B-Instruct
The server listens on http://localhost:8000 by default; --host and --port change that. Check that it is up by listing the models:
curl http://localhost:8000/v1/models
One default is easy to miss. If the model repository contains a generation_config.json, vLLM applies it, so sampling defaults such as temperature may be the model author's and not vLLM's. Pass --generation-config vllm to ignore that file.
The official openai package works unchanged. Point it at the server and use the served model's name:
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
)
chat_response = client.chat.completions.create(
model="Qwen/Qwen2.5-1.5B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Tell me a joke."},
],
)
print("Chat response:", chat_response)
The key is "EMPTY" because the server does not check keys unless you ask it to. The same request works with curl against /v1/chat/completions.
vLLM publishes an image, vllm/vllm-openai, that starts the same server:
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-0.6B
The volume keeps downloaded models between runs. --ipc=host gives the container access to the host's shared memory, which PyTorch uses to pass data between processes; --shm-size is the alternative. Server arguments go after the image name.
The image runs as root by default. It also ships a built-in vllm user, which you select with --user 2000:0. In that case mount the model cache under /home/vllm/.cache/huggingface instead of /root, and make the directory writable by group 0. For a fixed deployment, replace latest with a version tag.
| Flag | What it controls |
|---|---|
--max-model-len | Context length, prompt plus output. Derived from the model if unset. auto picks the largest length that fits in GPU memory. |
--gpu-memory-utilization | Fraction of GPU memory this server may use, from 0 to 1. Set 0.5 on each of two servers to share one GPU. |
--served-model-name | The name clients send in the model field. Defaults to the model path. |
--tensor-parallel-size | Number of tensor-parallel groups, used to split one model across GPUs. Default 1. |
--generation-config vllm | Ignore the model repository's sampling defaults. |
When a model fails to load for lack of memory, the first two are the ones to change: a shorter context needs less memory, and a lower utilization leaves room for something else on the card. Why a model that fits can still be slow is covered in AI GPU memory capacity and bandwidth differences.
--api-key, or the VLLM_API_KEY environment variable, makes the server require a key. You can pass several keys, which helps with rotation.
The reference page for the flag carries a warning that is easy to skip. The key only protects endpoints under the /v1, /v2, /inference and /cohere prefixes. Other endpoints on the same server stay unauthenticated, including /invocations, which exposes the same inference. The documentation's wording is "Do not rely on --api-key alone to secure vLLM."
In practice that means keeping the port off the public internet. Bind the server to a private interface, and put it behind a reverse proxy or gateway that authenticates every path before it is reachable from outside.
This quickstart is the first part of a series that builds the server out step by step:
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
This guide follows the vLLM documentation as read on November 12, 2026. Nothing in it was run for this article, because vLLM needs a GPU machine that was not available. Compare the commands with vllm serve --help on your installed version. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.