Seongwoo Choi / Journal

vLLM for a team: batching, API keys and a safe proxy

At a glanceHow one vLLM server answers many people at once, which limits decide who waits, why --api-key is not enough, and a tested nginx proxy with per-user keys.

Partly testedThe nginx configuration was run against a stand-in server: keys, blocked paths, streaming and rate limits. vLLM itself was not run: there was no GPU.

You need
A working vLLM container from parts 2 to 4, a host name for the server, and a TLS certificate for it.
You end up with
One HTTPS endpoint where each person has their own key, a rate limit and a usage log, and nothing else is reachable.
Series · part 5 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 9 sections
  1. 1. What happens when ten people ask at once
  2. 2. The four limits that decide who waits
  3. 3. What users feel, and the number behind it
  4. 4. Why the built-in API key is not enough
  5. 5. Build the proxy
  6. 6. Test the door before you hand out keys
  7. 7. When one server is not enough
  8. 8. House rules for clients
  9. 9. What you can do now

A model server that works for you is not yet a service for a team. Two questions remain. What happens when ten people ask at the same moment? And how do you let colleagues in without letting everyone else in? This part answers both, and ends with a proxy configuration that was tested for this article.

1. What happens when ten people ask at once

vLLM does not answer requests one after another. A language model writes one token per step, and one step can produce the next token for many requests together. New requests join the running batch at the next step and finished ones leave. This is called continuous batching, and it is the main reason a single GPU can serve a group.

Each request goes through two phases, and the vLLM documentation treats them differently:

  • Prefill: the model reads the prompt. This is heavy computation, and a long prompt takes noticeable time.
  • Decode: the model writes the answer token by token. Each step is light, but the request holds memory for every token it has seen so far.

By default the scheduler gives priority to requests that are already writing, and fits new prompts into the capacity that is left, splitting long prompts into chunks. The effect is that one person pasting a very long file does not freeze everyone else's answers in the middle of a sentence.

So ten simultaneous requests do not take ten times as long. They share the GPU, and each gets its tokens somewhat more slowly than it would alone. How much more slowly depends on the model and the card, and can only be measured; part 6 shows how.

2. The four limits that decide who waits

LimitSet byWhat happens at the limit
Context per request--max-model-lenA request that is too long is rejected
Requests in one step--max-num-seqsExtra requests wait in a queue
Tokens in one step--max-num-batched-tokensLong prompts are read in chunks over several steps
Working memoryGPU memory left after the weightsSome running requests are paused and redone later

The last row is the one that hurts. When the KV cache runs out, vLLM pauses a request to free memory and recomputes it when there is room. The log shows a warning that a request "is preempted ... because there is not enough KV cache space". The request still completes, but its user waits much longer.

The documentation lists the remedies in this order: leave --gpu-memory-utilization unset or raise it, lower --max-num-seqs or --max-num-batched-tokens so fewer requests run at once, or add GPUs with tensor parallelism so more memory is free for the cache.

The token limit is a trade-off you can tune. According to the documentation, a smaller --max-num-batched-tokens, such as 2048, keeps running answers smooth, because fewer new prompts interrupt them. A larger one gets new requests their first token sooner. For the best throughput it recommends values above 8192, especially for small models on large GPUs.

A rule that follows from the table: a queue is healthier than a pause. If you see preemption warnings, lower --max-num-seqs until they stop. Requests then wait their turn briefly, which costs far less than redoing work.

3. What users feel, and the number behind it

What the user noticesMetric at /metricsUsual cause when it is bad
Nothing happens after pressing Entervllm:request_queue_time_secondsMore requests than --max-num-seqs allows
A pause before the first wordvllm:time_to_first_token_secondsLong prompts, or a queue
Text arrives slowlyvllm:inter_token_latency_secondsToo many requests sharing each step
A long stall in the middlevllm:num_preemptionsThe KV cache is full

Two gauges show the live state: vllm:num_requests_running and vllm:num_requests_waiting. Part 7 puts all of these on a dashboard. Until then, curl -s http://127.0.0.1:8000/metrics | grep num_requests on the server is enough to see whether a queue is forming.

Check the CPU as well. The documentation counts one API process, one engine process and one worker per GPU, and recommends at least 2 + N physical cores for N GPUs. When cores are short, the GPU sits partly idle while requests wait.

4. Why the built-in API key is not enough

vLLM can require a key: pass --api-key or set the VLLM_API_KEY environment variable. Do set it. But read the limit in the documentation before relying on it. The key protects only paths under /v1, /v2, /inference and /cohere. The same server also answers on paths with no check at all, and the documentation lists them. Among them:

  • /invocations, which runs the same inference as /v1 without a key
  • /pause and /abort_requests, which stop generation for everyone
  • /metrics, which reveals the model name, load and usage
  • /tokenize, /health, /version and /load

The documentation's conclusion is plain: "Do not rely exclusively on --api-key for securing access to vLLM." Its recommended design is a reverse proxy that allows only the endpoints you choose, blocks the rest, and adds its own authentication, rate limiting and logging. One key shared by a whole team also has a practical problem. You cannot tell who used what, and you cannot remove one person without changing the key for all.

5. Build the proxy

Alice, Bob, a scripteach with a personal key
→
nginx proxychecks the key, limits the rate, writes the log
→
vLLMno published port, one server key
Three paths pass through. Everything else, such as /metrics and /pause, stops at the proxy with a 404.

The design has three rules:

  • vLLM publishes no port. Only the proxy can reach it, over a private Docker network.
  • Each person or program has a personal key. The proxy checks it, then replaces it with the server's own key before passing the request on.
  • Three paths are allowed. Everything else returns 404 at the proxy and never reaches vLLM.

Save this as vllm.conf:

# Each person or program gets its own key. Unknown keys map to an empty name.
map $http_authorization $client {
    default                            "";
    "Bearer team-alice-CHANGE-ME"      "alice";
    "Bearer team-bob-CHANGE-ME"        "bob";
}

# Per client: 2 new requests a second on average, bursts of 10.
limit_req_zone $client zone=per_client:10m rate=2r/s;
limit_req_status 429;

log_format llm '$time_iso8601 client=$client status=$status seconds=$request_time path=$uri';

upstream vllm {
    least_conn;
    server vllm:8000;
}

server {
    listen 443 ssl;
    server_name llm.example.internal;
    ssl_certificate     /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;

    access_log /var/log/nginx/llm.log llm;
    client_max_body_size 2m;

    # Only these three paths reach vLLM.
    location ~ ^/v1/(chat/completions|completions|models)$ {
        if ($client = "") { return 401; }
        limit_req zone=per_client burst=10 nodelay;

        proxy_pass http://vllm;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Authorization "Bearer SERVER-KEY-CHANGE-ME";
        proxy_set_header X-Client $client;

        # Streaming: hand tokens on as they arrive, and allow long answers.
        proxy_buffering off;
        proxy_read_timeout 600s;
    }

    # Everything else stops here, including /metrics, /invocations and /pause.
    location / { return 404; }
}

What each block is for:

  • map turns a key into a name. To add a person, add a line. To remove one, delete the line and reload nginx. Nobody else is affected.
  • limit_req counts requests per name, so one script in a loop cannot crowd out the team. Requests over the limit get status 429.
  • log_format writes one line per request with the person's name and the duration. That is your usage record.
  • proxy_buffering off is essential. With buffering on, nginx collects the answer and delivers it in lumps, and streaming appears broken.
  • proxy_read_timeout defaults to 60 seconds, which a long answer can exceed.

Generate real keys with openssl rand -hex 24. This file now contains secrets, so restrict who can read it and keep it out of version control.

Then the Compose file. Note that the vllm service has no ports section at all:

services:
  vllm:
    image: vllm/vllm-openai:latest
    command:
      - --model
      - Qwen/Qwen3-8B
      - --served-model-name
      - qwen3
      - --max-model-len
      - "16384"
    environment:
      VLLM_API_KEY: ${VLLM_API_KEY:?put VLLM_API_KEY in the .env file}
      HF_TOKEN: ${HF_TOKEN:-}
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    ipc: host
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  proxy:
    image: nginx:stable
    ports:
      - "443:443"
    volumes:
      - ./vllm.conf:/etc/nginx/conf.d/default.conf:ro
      - ./certs:/etc/nginx/certs:ro
    depends_on:
      - vllm
    restart: unless-stopped

volumes:
  vllm-cache:

Put VLLM_API_KEY=... in a .env file next to it, with the same value as SERVER-KEY-CHANGE-ME in vllm.conf. Place the certificate and its private key in a certs folder under the names used in the configuration. Use a certificate from your organization's authority or a public one; for a first test on a private network, a self-signed one will do.

docker compose up -d
docker compose exec proxy nginx -t

6. Test the door before you hand out keys

Run these from another machine. Each line states the answer you should get. The -k flag is only for a self-signed test certificate.

URL=https://llm.example.internal

# No key: 401
curl -sk -o /dev/null -w "%{http_code}\n" $URL/v1/models

# A valid key on an allowed path: 200
curl -sk -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer team-alice-CHANGE-ME" $URL/v1/models

# A valid key on paths that must stay closed: 404 each
for path in /metrics /invocations /pause /health /tokenize; do
  curl -sk -o /dev/null -w "$path %{http_code}\n" -H "Authorization: Bearer team-alice-CHANGE-ME" $URL$path
done

This configuration was run for this article against a stand-in server that echoed what it received. The results:

TestResult
No key, and a wrong key401 both times
Alice's key on /v1/models200; the server behind received the server key and the name "alice", not Alice's key
/metrics, /invocations, /pause, /health, /tokenize, /v1/embeddings404 each, with a valid key
A streamed reply sent in four parts, half a second apartArrived at 0.0, 0.5, 1.0 and 1.5 seconds, so nothing was held back
40 requests fired at once with one key11 accepted, 29 answered with 429
A second key during that burst200, unaffected

The stand-in server is not vLLM, so this proves the proxy's behaviour, not the model's. Repeat the three commands above on your own setup before giving anyone a key.

Clients then use the proxy's address and their personal key:

from openai import OpenAI

client = OpenAI(base_url="https://llm.example.internal/v1", api_key="team-alice-CHANGE-ME")

7. When one server is not enough

There are two ways to add GPUs, and they solve different problems.

Tensor parallelismReplicas
What it isOne server, one model split across GPUsSeveral independent servers, each with a full copy of the model
Use it whenThe model does not fit on one GPU, or the cache is too smallThe model fits on one GPU and you need more throughput or a spare
How--tensor-parallel-size NOne container per GPU, plus a load balancer
If one GPU failsThe whole server is downThe others keep answering

The proxy above is already a load balancer. For replicas, add a second service to the Compose file, a copy of vllm named vllm2. Give each service one card by replacing count: all with device_ids: ["0"] and device_ids: ["1"]. Then add a line to the upstream block:

upstream vllm {
    least_conn;
    server vllm:8000  max_fails=3 fail_timeout=30s;
    server vllm2:8000 max_fails=3 fail_timeout=30s;
}

least_conn sends each new request to the server with the fewest open connections. That suits model serving better than taking turns, because requests differ so much in length. This layout is the one in vLLM's own nginx guide. Start one replica first and let it finish downloading the model, so the second finds the weights in the shared cache.

8. House rules for clients

  • Always set max_tokens. A request without a ceiling can hold a slot until the context is full.
  • Stream. People wait more patiently for text that is arriving, and a dropped connection frees its slot.
  • Retry on 429 with a pause. A client that retries at once makes a queue longer.
  • Cap the n parameter. One request can ask for many answers at once. vLLM limits it with the VLLM_MAX_N_SEQUENCES environment variable, whose default is 16,384. The documentation suggests a value such as 64 or 128 for exposed servers.
  • Never set VLLM_SERVER_DEV_MODE=1 on a shared server. The documentation marks its extra endpoints as dangerous.

9. What you can do now

  • Explain why ten users do not mean ten times the wait, and what batching costs each of them.
  • Tell a queue from a preemption in the log and in the metrics, and choose the flag that fixes each.
  • Name the paths that --api-key leaves open.
  • Run a proxy that gives each person a key, a limit and a log line, and prove it blocks what it should.
  • Choose between tensor parallelism and replicas for a second GPU.

How many GPUs a given number of users needs is the next question: GPU sizing for vLLM.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

The vLLM behaviour described here comes from its documentation as read on November 12, 2026. The nginx configuration was run for this article with nginx 1.24.0 against a small stand-in server, not against vLLM: keys, blocked paths, streaming and the rate limit were tested and the results are reported below. The Compose file was checked with docker compose config. No GPU was used. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy