How many GPUs for N users? A vLLM sizing method
Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
Serving a trained model to real users: speed, memory and cost.
Number 10 this week: 749 points on Hacker News in the past 7 days and about 17 news articles a day.
Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.
Choose a GPU for serving language models by memory first, then interconnect, then price per token: NVIDIA's specifications, cloud prices and worked sums.
AI & technologyIntel disclosed technical details of its data-center inference GPU, Intel GPU D-series, at OCP Global Summit on October 12, featuring up to 480GB LPDDR5X memory, 1.5TB/s bandwidth, and 1.3 petaflops at FP8.