vLLM quickstart: serve a model with an OpenAI-style API
Install vLLM, start an OpenAI-compatible server with one command, call it from Python, and run it in Docker. With the flags that control GPU memory.
Series · 8 parts · 76 min read in total
From the first request to a monitored server for a whole team: Docker on NVIDIA GPUs, Qwen and Phind-CodeLlama, many users at once, GPU sizing, dashboards and alerts.
Install vLLM, start an OpenAI-compatible server with one command, call it from Python, and run it in Docker. With the flags that control GPU memory.
Install the NVIDIA Container Toolkit, prove the GPU works inside a container, start vLLM with docker run, then make it a Compose service that survives reboots.
Pick a Qwen model that fits your GPU, start it with the right vLLM flags, then use thinking mode, tool calling and structured output from the OpenAI client.
Wrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.
How one vLLM server answers many people at once, which limits decide who waits, why --api-key is not enough, and a tested nginx proxy with per-user keys.
Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
One Compose file for vLLM, node exporter, the NVIDIA DCGM exporter, Prometheus and Grafana, with the queries and alerts for CPU, RAM, GPU and the model server.
Send your vLLM server's alerts to your phone with Alertmanager and Telegram: routing by urgency, timing, silences, and a full test without a GPU.