How many GPUs for N users? A vLLM sizing method
Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
How much graphics memory a model needs, and what to do when it runs out.
Number 14 this week: 121 points on Hacker News in the past 7 days and about 27 news articles a day.
Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
Choose a GPU for serving language models by memory first, then interconnect, then price per token: NVIDIA's specifications, cloud prices and worked sums.
vLLM runs out of memory in three different ways: at startup, under load, and in system RAM. Six checks that tell them apart, with the fix for each.
Send your vLLM server's alerts to your phone with Alertmanager and Telegram: routing by urgency, timing, silences, and a full test without a GPU.
One Compose file for vLLM, node exporter, the NVIDIA DCGM exporter, Prometheus and Grafana, with the queries and alerts for CPU, RAM, GPU and the model server.
Wrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.
Pick a Qwen model that fits your GPU, start it with the right vLLM flags, then use thinking mode, tool calling and structured output from the OpenAI client.
Install the NVIDIA Container Toolkit, prove the GPU works inside a container, start vLLM with docker run, then make it a Compose service that survives reboots.
Install vLLM, start an OpenAI-compatible server with one command, call it from Python, and run it in Docker. With the flags that control GPU memory.
When a local AI model running in Ollama responds slowly, check its execution status, GPU placement, context length, and memory use by other programs before changing models.
Capacity determines whether a model fits, while bandwidth affects token generation speed; weights, KV cache, and workspace all contribute to memory requirements.
AI & technologyIntel disclosed technical details of its data-center inference GPU, Intel GPU D-series, at OCP Global Summit on October 12, featuring up to 480GB LPDDR5X memory, 1.5TB/s bandwidth, and 1.3 petaflops at FP8.