Seongwoo Choi / Journal

Start with part 1

Serve Phind-CodeLlama-34B on A100 GPUs with vLLM

Wrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.

How many GPUs for N users? A vLLM sizing method

Work out GPU memory from a model's config file, turn a number of users into concurrent requests, then load-test with vllm bench serve to find the real limit.