Seongwoo Choi / Journal

vLLM quickstart: serve a model with an OpenAI-style API

At a glanceInstall vLLM, start an OpenAI-compatible server with one command, call it from Python, and run it in Docker. With the flags that control GPU memory.

Not testedWritten from the vLLM documentation. Nothing was run: there was no GPU.

You need
A Linux machine with an NVIDIA GPU and Python 3.11 or newer.
You end up with
A model answering OpenAI-style requests on your own machine.
Series · part 1 of 8Run your own LLM server with vLLM
  1. vLLM quickstart: serve a model with an OpenAI-style API
  2. Run vLLM in Docker on an NVIDIA GPU, step by step
  3. Serve Qwen with vLLM: chat, tool calling and thinking
  4. Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
  5. vLLM for a team: batching, API keys and a safe proxy
  6. How many GPUs for N users? A vLLM sizing method
  7. Monitor vLLM with Prometheus, Grafana and DCGM in Docker
  8. vLLM alerts with Alertmanager: know before your users do

About this series →

In this article 8 sections
  1. 1. Before you install
  2. 2. Install
  3. 3. Start the server
  4. 4. Call it from Python
  5. 5. Run it in Docker
  6. 6. The flags to learn first
  7. 7. Before anyone else can reach it
  8. 8. Where to go from here

vLLM is a server for running open-weight language models on your own GPUs. It speaks the OpenAI API protocol, so an application written for the OpenAI client can point at a vLLM server by changing one URL. This guide gets a first server running and covers the handful of settings that decide whether a model fits.

1. Before you install

The quickstart lists Linux and Python 3.11 to 3.14 as prerequisites. On a Mac with Apple Silicon, the documentation points to a separate build called vLLM-Metal. A server hosts one model at a time, so plan one server per model.

If the goal is to try a model on a laptop, a desktop tool is simpler; see what to check before installing Ollama on Windows. vLLM is the choice when several users or programs need to call the same model at once.

2. Install

For NVIDIA GPUs the documentation recommends uv to create the environment:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

--torch-backend=auto lets uv pick the PyTorch build that matches the installed CUDA driver. vllm download-kernels fetches precompiled kernels in advance. Without it vLLM downloads and compiles them at startup, which the documentation says can add several minutes on newer GPUs.

3. Start the server

One command downloads the model from Hugging Face and serves it:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

The server listens on http://localhost:8000 by default; --host and --port change that. Check that it is up by listing the models:

curl http://localhost:8000/v1/models

One default is easy to miss. If the model repository contains a generation_config.json, vLLM applies it, so sampling defaults such as temperature may be the model author's and not vLLM's. Pass --generation-config vllm to ignore that file.

4. Call it from Python

The official openai package works unchanged. Point it at the server and use the served model's name:

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

chat_response = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a joke."},
    ],
)
print("Chat response:", chat_response)

The key is "EMPTY" because the server does not check keys unless you ask it to. The same request works with curl against /v1/chat/completions.

5. Run it in Docker

vLLM publishes an image, vllm/vllm-openai, that starts the same server:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

The volume keeps downloaded models between runs. --ipc=host gives the container access to the host's shared memory, which PyTorch uses to pass data between processes; --shm-size is the alternative. Server arguments go after the image name.

The image runs as root by default. It also ships a built-in vllm user, which you select with --user 2000:0. In that case mount the model cache under /home/vllm/.cache/huggingface instead of /root, and make the directory writable by group 0. For a fixed deployment, replace latest with a version tag.

6. The flags to learn first

FlagWhat it controls
--max-model-lenContext length, prompt plus output. Derived from the model if unset. auto picks the largest length that fits in GPU memory.
--gpu-memory-utilizationFraction of GPU memory this server may use, from 0 to 1. Set 0.5 on each of two servers to share one GPU.
--served-model-nameThe name clients send in the model field. Defaults to the model path.
--tensor-parallel-sizeNumber of tensor-parallel groups, used to split one model across GPUs. Default 1.
--generation-config vllmIgnore the model repository's sampling defaults.

When a model fails to load for lack of memory, the first two are the ones to change: a shorter context needs less memory, and a lower utilization leaves room for something else on the card. Why a model that fits can still be slow is covered in AI GPU memory capacity and bandwidth differences.

7. Before anyone else can reach it

--api-key, or the VLLM_API_KEY environment variable, makes the server require a key. You can pass several keys, which helps with rotation.

The reference page for the flag carries a warning that is easy to skip. The key only protects endpoints under the /v1, /v2, /inference and /cohere prefixes. Other endpoints on the same server stay unauthenticated, including /invocations, which exposes the same inference. The documentation's wording is "Do not rely on --api-key alone to secure vLLM."

In practice that means keeping the port off the public internet. Bind the server to a private interface, and put it behind a reverse proxy or gateway that authenticates every path before it is reachable from outside.

8. Where to go from here

This quickstart is the first part of a series that builds the server out step by step:

  • Part 2 runs vLLM in Docker on an NVIDIA GPU as a service that survives reboots.
  • Part 3 serves Qwen with tool calling and switchable thinking.
  • Part 4 wraps Phind-CodeLlama-34B, a model with no chat template, on A100 GPUs.
  • Part 5 opens the server to a team with per-user keys and a proxy.
  • Part 6 works out how many GPUs a number of users needs.
  • Part 7 adds dashboards for CPU, memory, GPUs and the server itself.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

This guide follows the vLLM documentation as read on November 12, 2026. Nothing in it was run for this article, because vLLM needs a GPU machine that was not available. Compare the commands with vllm serve --help on your installed version. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy