vLLM quickstart: serve a model with an OpenAI-style API
Install vLLM, start an OpenAI-compatible server with one command, call it from Python, and run it in Docker. With the flags that control GPU memory.
At a glanceHow one vLLM server answers many people at once, which limits decide who waits, why --api-key is not enough, and a tested nginx proxy with per-user keys.
Partly testedThe nginx configuration was run against a stand-in server: keys, blocked paths, streaming and rate limits. vLLM itself was not run: there was no GPU.
A model server that works for you is not yet a service for a team. Two questions remain. What happens when ten people ask at the same moment? And how do you let colleagues in without letting everyone else in? This part answers both, and ends with a proxy configuration that was tested for this article.
vLLM does not answer requests one after another. A language model writes one token per step, and one step can produce the next token for many requests together. New requests join the running batch at the next step and finished ones leave. This is called continuous batching, and it is the main reason a single GPU can serve a group.
Each request goes through two phases, and the vLLM documentation treats them differently:
By default the scheduler gives priority to requests that are already writing, and fits new prompts into the capacity that is left, splitting long prompts into chunks. The effect is that one person pasting a very long file does not freeze everyone else's answers in the middle of a sentence.
So ten simultaneous requests do not take ten times as long. They share the GPU, and each gets its tokens somewhat more slowly than it would alone. How much more slowly depends on the model and the card, and can only be measured; part 6 shows how.
| Limit | Set by | What happens at the limit |
|---|---|---|
| Context per request | --max-model-len | A request that is too long is rejected |
| Requests in one step | --max-num-seqs | Extra requests wait in a queue |
| Tokens in one step | --max-num-batched-tokens | Long prompts are read in chunks over several steps |
| Working memory | GPU memory left after the weights | Some running requests are paused and redone later |
The last row is the one that hurts. When the KV cache runs out, vLLM pauses a request to free memory and recomputes it when there is room. The log shows a warning that a request "is preempted ... because there is not enough KV cache space". The request still completes, but its user waits much longer.
The documentation lists the remedies in this order: leave --gpu-memory-utilization unset or raise it, lower --max-num-seqs or --max-num-batched-tokens so fewer requests run at once, or add GPUs with tensor parallelism so more memory is free for the cache.
The token limit is a trade-off you can tune. According to the documentation, a smaller --max-num-batched-tokens, such as 2048, keeps running answers smooth, because fewer new prompts interrupt them. A larger one gets new requests their first token sooner. For the best throughput it recommends values above 8192, especially for small models on large GPUs.
A rule that follows from the table: a queue is healthier than a pause. If you see preemption warnings, lower --max-num-seqs until they stop. Requests then wait their turn briefly, which costs far less than redoing work.
| What the user notices | Metric at /metrics | Usual cause when it is bad |
|---|---|---|
| Nothing happens after pressing Enter | vllm:request_queue_time_seconds | More requests than --max-num-seqs allows |
| A pause before the first word | vllm:time_to_first_token_seconds | Long prompts, or a queue |
| Text arrives slowly | vllm:inter_token_latency_seconds | Too many requests sharing each step |
| A long stall in the middle | vllm:num_preemptions | The KV cache is full |
Two gauges show the live state: vllm:num_requests_running and vllm:num_requests_waiting. Part 7 puts all of these on a dashboard. Until then, curl -s http://127.0.0.1:8000/metrics | grep num_requests on the server is enough to see whether a queue is forming.
Check the CPU as well. The documentation counts one API process, one engine process and one worker per GPU, and recommends at least 2 + N physical cores for N GPUs. When cores are short, the GPU sits partly idle while requests wait.
vLLM can require a key: pass --api-key or set the VLLM_API_KEY environment variable. Do set it. But read the limit in the documentation before relying on it. The key protects only paths under /v1, /v2, /inference and /cohere. The same server also answers on paths with no check at all, and the documentation lists them. Among them:
/invocations, which runs the same inference as /v1 without a key/pause and /abort_requests, which stop generation for everyone/metrics, which reveals the model name, load and usage/tokenize, /health, /version and /loadThe documentation's conclusion is plain: "Do not rely exclusively on --api-key for securing access to vLLM." Its recommended design is a reverse proxy that allows only the endpoints you choose, blocks the rest, and adds its own authentication, rate limiting and logging. One key shared by a whole team also has a practical problem. You cannot tell who used what, and you cannot remove one person without changing the key for all.
The design has three rules:
Save this as vllm.conf:
# Each person or program gets its own key. Unknown keys map to an empty name.
map $http_authorization $client {
default "";
"Bearer team-alice-CHANGE-ME" "alice";
"Bearer team-bob-CHANGE-ME" "bob";
}
# Per client: 2 new requests a second on average, bursts of 10.
limit_req_zone $client zone=per_client:10m rate=2r/s;
limit_req_status 429;
log_format llm '$time_iso8601 client=$client status=$status seconds=$request_time path=$uri';
upstream vllm {
least_conn;
server vllm:8000;
}
server {
listen 443 ssl;
server_name llm.example.internal;
ssl_certificate /etc/nginx/certs/fullchain.pem;
ssl_certificate_key /etc/nginx/certs/privkey.pem;
access_log /var/log/nginx/llm.log llm;
client_max_body_size 2m;
# Only these three paths reach vLLM.
location ~ ^/v1/(chat/completions|completions|models)$ {
if ($client = "") { return 401; }
limit_req zone=per_client burst=10 nodelay;
proxy_pass http://vllm;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Authorization "Bearer SERVER-KEY-CHANGE-ME";
proxy_set_header X-Client $client;
# Streaming: hand tokens on as they arrive, and allow long answers.
proxy_buffering off;
proxy_read_timeout 600s;
}
# Everything else stops here, including /metrics, /invocations and /pause.
location / { return 404; }
}
What each block is for:
map turns a key into a name. To add a person, add a line. To remove one, delete the line and reload nginx. Nobody else is affected.limit_req counts requests per name, so one script in a loop cannot crowd out the team. Requests over the limit get status 429.log_format writes one line per request with the person's name and the duration. That is your usage record.proxy_buffering off is essential. With buffering on, nginx collects the answer and delivers it in lumps, and streaming appears broken.proxy_read_timeout defaults to 60 seconds, which a long answer can exceed.Generate real keys with openssl rand -hex 24. This file now contains secrets, so restrict who can read it and keep it out of version control.
Then the Compose file. Note that the vllm service has no ports section at all:
services:
vllm:
image: vllm/vllm-openai:latest
command:
- --model
- Qwen/Qwen3-8B
- --served-model-name
- qwen3
- --max-model-len
- "16384"
environment:
VLLM_API_KEY: ${VLLM_API_KEY:?put VLLM_API_KEY in the .env file}
HF_TOKEN: ${HF_TOKEN:-}
volumes:
- ${HOME}/.cache/huggingface:/root/.cache/huggingface
- vllm-cache:/root/.cache/vllm
ipc: host
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
proxy:
image: nginx:stable
ports:
- "443:443"
volumes:
- ./vllm.conf:/etc/nginx/conf.d/default.conf:ro
- ./certs:/etc/nginx/certs:ro
depends_on:
- vllm
restart: unless-stopped
volumes:
vllm-cache:
Put VLLM_API_KEY=... in a .env file next to it, with the same value as SERVER-KEY-CHANGE-ME in vllm.conf. Place the certificate and its private key in a certs folder under the names used in the configuration. Use a certificate from your organization's authority or a public one; for a first test on a private network, a self-signed one will do.
docker compose up -d
docker compose exec proxy nginx -t
Run these from another machine. Each line states the answer you should get. The -k flag is only for a self-signed test certificate.
URL=https://llm.example.internal
# No key: 401
curl -sk -o /dev/null -w "%{http_code}\n" $URL/v1/models
# A valid key on an allowed path: 200
curl -sk -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer team-alice-CHANGE-ME" $URL/v1/models
# A valid key on paths that must stay closed: 404 each
for path in /metrics /invocations /pause /health /tokenize; do
curl -sk -o /dev/null -w "$path %{http_code}\n" -H "Authorization: Bearer team-alice-CHANGE-ME" $URL$path
done
This configuration was run for this article against a stand-in server that echoed what it received. The results:
| Test | Result |
|---|---|
| No key, and a wrong key | 401 both times |
Alice's key on /v1/models | 200; the server behind received the server key and the name "alice", not Alice's key |
/metrics, /invocations, /pause, /health, /tokenize, /v1/embeddings | 404 each, with a valid key |
| A streamed reply sent in four parts, half a second apart | Arrived at 0.0, 0.5, 1.0 and 1.5 seconds, so nothing was held back |
| 40 requests fired at once with one key | 11 accepted, 29 answered with 429 |
| A second key during that burst | 200, unaffected |
The stand-in server is not vLLM, so this proves the proxy's behaviour, not the model's. Repeat the three commands above on your own setup before giving anyone a key.
Clients then use the proxy's address and their personal key:
from openai import OpenAI
client = OpenAI(base_url="https://llm.example.internal/v1", api_key="team-alice-CHANGE-ME")
There are two ways to add GPUs, and they solve different problems.
| Tensor parallelism | Replicas | |
|---|---|---|
| What it is | One server, one model split across GPUs | Several independent servers, each with a full copy of the model |
| Use it when | The model does not fit on one GPU, or the cache is too small | The model fits on one GPU and you need more throughput or a spare |
| How | --tensor-parallel-size N | One container per GPU, plus a load balancer |
| If one GPU fails | The whole server is down | The others keep answering |
The proxy above is already a load balancer. For replicas, add a second service to the Compose file, a copy of vllm named vllm2. Give each service one card by replacing count: all with device_ids: ["0"] and device_ids: ["1"]. Then add a line to the upstream block:
upstream vllm {
least_conn;
server vllm:8000 max_fails=3 fail_timeout=30s;
server vllm2:8000 max_fails=3 fail_timeout=30s;
}
least_conn sends each new request to the server with the fewest open connections. That suits model serving better than taking turns, because requests differ so much in length. This layout is the one in vLLM's own nginx guide. Start one replica first and let it finish downloading the model, so the second finds the weights in the shared cache.
max_tokens. A request without a ceiling can hold a slot until the context is full.n parameter. One request can ask for many answers at once. vLLM limits it with the VLLM_MAX_N_SEQUENCES environment variable, whose default is 16,384. The documentation suggests a value such as 64 or 128 for exposed servers.VLLM_SERVER_DEV_MODE=1 on a shared server. The documentation marks its extra endpoints as dangerous.--api-key leaves open.How many GPUs a given number of users needs is the next question: GPU sizing for vLLM.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
The vLLM behaviour described here comes from its documentation as read on November 12, 2026. The nginx configuration was run for this article with nginx 1.24.0 against a small stand-in server, not against vLLM: keys, blocked paths, streaming and the rate limit were tested and the results are reported below. The Compose file was checked with docker compose config. No GPU was used. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.