Seongwoo Choi / Journal

Serve Phind-CodeLlama-34B on A100 GPUs with vLLM

At a glanceWrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.

Partly testedThe chat template was rendered and matched the model card. The model was not loaded: no A100 was available, so memory figures are calculated.

You need
The Docker setup from part 2, about 70 GB of free disk, and one A100 80GB or two A100 40GB cards.
You end up with
A private coding assistant behind an OpenAI-style chat endpoint, with a template you wrote and tested.
Series · part 4 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 9 sections
  1. 1. Read the model's files before you download 67 GB
  2. 2. Decide the GPU layout
  3. 3. Write the chat template
  4. 4. Test the template before the GPU is involved
  5. 5. Launch the server
  6. 6. Verify the load and the first answer
  7. 7. Connect people and tools
  8. 8. When it does not work
  9. 9. What you can do now

Qwen in part 3 worked out of the box. Many models do not. They were published before today's conventions, with a prompt format described in a README and nothing a server can read. Phind-CodeLlama-34B-v2 is a clean example: a strong coding model from 2023 with no chat template, weights in an older file format, and a size that makes GPU planning matter. Wrapping it teaches every step you need for any model that does not just work.

One honest note before you spend a day on it. Newer coding models exist, and Qwen2.5-Coder-32B from part 3 needs about the same hardware. Choose Phind if you already rely on it. Otherwise read this part for the method.

1. Read the model's files before you download 67 GB

Four small files in the repository tell you almost everything. Open them in the browser on the model's "Files" tab.

FileWhat it saysWhat that means for you
README.mdLicense llama2. Trained with a sequence length of 4,096 tokens. Uses the Alpaca/Vicuna prompt format.Read the Llama 2 license terms before commercial use. Plan for a 4,096-token context.
config.json48 layers, hidden size 8,192, 64 attention heads, 8 key-value heads, 16-bit weights, 16,384 positionsThese numbers give the memory per token in step 2.
tokenizer_config.jsonNo chat_template entry. Maximum length 4,096.The chat endpoint will not work until you supply a template.
File listSeven pytorch_model-*.bin files, 67.5 GB in total, last changed in August 2023An older weight format. vLLM's default loader tries safetensors first and falls back to these.

The .bin format deserves one caution. It is based on Python's pickle, which can run code when a file is loaded. Only load such files from a publisher you trust, and download them from the original repository, not a mirror.

The repository is not gated, so no access request or token is needed.

2. Decide the GPU layout

Two things must fit in GPU memory: the weights, and the KV cache that holds each request's tokens while it is being answered. The weights are 67.5 GB on disk, which is 62.9 GiB in the units GPUs report.

The cache cost per token follows from the config: 2 (a key and a value) × 48 layers × 8 key-value heads × 128 dimensions per head × 2 bytes = 196,608 bytes, about 0.19 MiB. A request that fills the 4,096-token context holds about 0.75 GiB.

With a tenth of each card kept back for the runtime, which is an assumption and not a vLLM setting, the layouts compare like this:

LayoutLeft for the cacheFull 4,096-token requests at onceNotes
1 × A100 80GBabout 9 GiB12The simplest option
2 × A100 40GBabout 9 GiB12Needs --tensor-parallel-size 2
2 × A100 80GBabout 81 GiB108One copy of the weights, a much larger cache
1 × A100 40GB, AWQ buildabout 19 GiB254-bit weights: 17 GiB, lower precision

The third row is the one worth remembering. Doubling the GPUs raises the cache ninefold, because the weights are paid for once and everything else goes to requests. Part 6 turns this into a method for any model.

The model does not fit on a single 40 GB card at full precision. The fourth row uses a community-quantized build, TheBloke/Phind-CodeLlama-34B-v2-AWQ. The vLLM documentation says quantized models from the Hub can be used directly, at the cost of lower precision. It is a different publisher from the original, so judge it as you would any third-party file.

3. Write the chat template

A chat request arrives as a list of messages with roles. The model was trained on plain text in one specific layout. The chat template is the translation between the two, written in the Jinja language. The vLLM documentation is direct about what happens without one: "the server will not be able to process chat and all chat requests will error."

The model card shows the layout the model expects:

### System Prompt
You are an intelligent programming assistant.

### User Message
Implement a linked list in C++

### Assistant
...

Save the following as phind.jinja. It writes each message under the right heading and, when the server asks for a reply, ends with the assistant heading so the model continues from there:

{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
### System Prompt
{{ message['content'] }}

{% elif message['role'] == 'user' -%}
### User Message
{{ message['content'] }}

{% elif message['role'] == 'assistant' -%}
### Assistant
{{ message['content'] }}

{% endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
### Assistant
{% endif -%}

The hyphens inside {%- and -%} remove the whitespace next to a tag. They are what keeps the blank lines exactly where the model card has them. A model is sensitive to this: a missing newline will not cause an error, only worse answers that are hard to trace back.

4. Test the template before the GPU is involved

You do not need the model to check a template. Render it on any computer with Python and compare the text with the model card. Save this as test_template.py next to the template:

import jinja2

template = jinja2.Environment().from_string(open("phind.jinja").read())
messages = [
    {"role": "system", "content": "You are an intelligent programming assistant."},
    {"role": "user", "content": "Implement a linked list in C++"},
]
text = template.render(messages=messages, add_generation_prompt=True)

expected = (
    "### System Prompt\nYou are an intelligent programming assistant.\n\n"
    "### User Message\nImplement a linked list in C++\n\n"
    "### Assistant\n"
)
print(repr(text))
assert text == expected, "the template does not match the model card"
print("template matches the model card")
pip install jinja2
python test_template.py

This check was run for this article and passed, including a second test with an earlier assistant turn and a follow-up question. Printing with repr matters, because it makes every newline visible. Keep the script. Each time you edit the template, run it again before restarting a server that takes minutes to load.

5. Launch the server

From the folder that holds phind.jinja, on a machine with one A100 80GB:

docker run -d --name phind \
  --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$PWD/phind.jinja:/templates/phind.jinja:ro" \
  -p 127.0.0.1:8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Phind/Phind-CodeLlama-34B-v2 \
  --served-model-name phind-codellama \
  --chat-template /templates/phind.jinja \
  --max-model-len 4096
FlagWhy
The second -vPuts your template inside the container, read-only.
--chat-templateThe path to the template as the container sees it. The reference also accepts the template as a single-line string.
--max-model-len 4096Matches the length the model was trained and tested with. The config would allow more, but nothing in the model card supports it.
--served-model-nameA short, stable name for clients.

For two GPUs, add one flag:

  --tensor-parallel-size 2

Tensor parallelism splits every layer across the cards. The vLLM documentation recommends it when a model is too large for one GPU, or when you want more room for the cache. To use only some of a machine's GPUs, select them in Docker with --gpus '"device=0,1"'. The size must match the number of GPUs the container can see.

Expect a slow first start. The download is 67.5 GB, and loading reads all of it from disk. The documentation adds that with tensor parallelism each worker reads the whole model, so loading takes longer in proportion. Keep the cache on a local disk.

6. Verify the load and the first answer

Follow the log with docker logs -f phind and look for the two lines that report the real cache size:

GPU KV cache size: ... tokens
Maximum concurrency for 4,096 tokens per request: ...x

Compare the second number with the table in step 2. It should be in the same range. If it is far lower, something else is using the card; check nvidia-smi.

Then ask a question:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "phind-codellama",
    "messages": [
      {"role": "system", "content": "You are an intelligent programming assistant."},
      {"role": "user", "content": "Write a Python function that checks whether a string is a palindrome."}
    ],
    "temperature": 0.1,
    "top_p": 0.75,
    "max_tokens": 512
  }'

The temperature and top_p values are the ones the model card uses in its own evaluation code.

Judge the answer by its shape. It should be code and a short explanation, and it should stop. Two failure patterns point straight at the template:

  • The model keeps going and writes its own "### User Message". It did not learn to stop at that point, or the layout is slightly off. Rerun the template test, then add "stop": ["### User Message"] to requests as a guard.
  • The answer ignores the system prompt or reads like a continuation of your question. The template is probably not being applied. Check the path in --chat-template and the mount.

7. Connect people and tools

Any client that lets you set an OpenAI-compatible base URL can use the server. It needs three values: the base URL ending in /v1, the model name phind-codellama, and an API key, which can be any text until you configure one.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
stream = client.chat.completions.create(
    model="phind-codellama",
    messages=[{"role": "user", "content": "Explain what this regex matches: ^\\d{3}-\\d{4}$"}],
    max_tokens=400,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Streaming matters for a coding assistant. People tolerate a long answer that starts at once far better than a short one that appears after a silent wait.

Know two limits before you promise this to a team:

  • No tool calling. The model card describes plain instruction and answer pairs. Agent tools that depend on function calling need a model trained for it, such as the Qwen models in part 3.
  • A 4,096-token context. That is roughly a few hundred lines of code plus the answer. It suits questions about a function or a file, not a whole repository.

The server is still bound to the local machine. Opening it to a team safely, with keys and a proxy, is the subject of part 5.

8. When it does not work

What you seeCauseFix
Chat requests return an error about a chat templateThe template was not loadedCheck the mount and the --chat-template path
Out of memory while loadingThe layout is too small, or the card is sharedStep 2; check nvidia-smi for other processes
An error about the number of GPUs--tensor-parallel-size does not match the visible GPUsMake the two agree
Loading seems stuck67.5 GB is being read, once per worker with tensor parallelismWait, and watch disk activity; use a local disk
A request is rejected as too longPrompt plus max_tokens exceeds 4,096Shorten the input or lower max_tokens
Answers run on into invented turnsTemplate whitespace, or no stop sequenceSteps 4 and 6

9. What you can do now

  • Read a model's files and predict whether it will serve as is.
  • Write a chat template from a model card and prove it correct without a GPU.
  • Estimate how many concurrent requests a GPU layout can hold, and check the estimate against the startup log.
  • Recognize template problems from the shape of a bad answer.

Next: serving many users from one server.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

Model facts were read from the Phind-CodeLlama-34B-v2 repository on Hugging Face on November 12, 2026. The chat template in this article was rendered with Jinja 3.1.6 and its output matched the model card's prompt format exactly. The memory figures are calculated, not measured: no A100 was available and the model was not loaded. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy