Serve Phind-CodeLlama-34B on A100 GPUs with vLLM
Wrap a model that ships without a chat template: read its files, write and test the template, pick an A100 layout, and launch a coding assistant for a team.
At a glanceChoose a GPU for serving language models by memory first, then interconnect, then price per token: NVIDIA's specifications, cloud prices and worked sums.
Not testedWritten from NVIDIA's specifications and public cloud prices. No GPU was rented or timed.
The A100, L40S and H100 are the three NVIDIA cards you are most likely to be offered for serving a language model, whether you rent by the hour or inherit a server. Marketing pages rank them by speed. For inference, speed is the third question. The first is whether your model fits, and the second is whether the cards can work together. This guide takes them in that order.
| A100 80GB | L40S | H100 | |
|---|---|---|---|
| Architecture | Ampere | Ada Lovelace | Hopper |
| Memory | 80 GB | 48 GB | 80 GB, or 94 GB on the NVL model |
| Memory bandwidth | 1,935 to 2,039 GB/s | 864 GB/s | 3.35 to 3.9 TB/s |
| NVLink between cards | Yes | No | Yes |
| Split into smaller GPUs (MIG) | Up to 7 | No | Up to 7 |
| FP8 listed in the specifications | No | Yes | Yes |
| Maximum power | 300 W, or 400 W for the SXM model | 350 W | Up to 700 W for SXM; 350 to 400 W for NVL |
All figures are from NVIDIA's product pages. An A100 with 40 GB also exists in older servers; everything below about the A100 80GB applies to it with half the memory.
Newer generations exist as well. Lambda's price list already includes the B200 with 180 GB per GPU. The older cards remain the practical choice for many teams because they are cheaper and easier to get.
Memory decides which models you can run at all, and how many requests fit beside them. Every request holds working memory, the KV cache, for as long as it is being answered. The method is in the sizing guide; here are its results for two models, keeping a tenth of each card in reserve.
Model weightsKV cache, the room for requestsRuntime reserve
| Model, at 16-bit precision | One L40S (48 GB) | One A100 or H100 (80 GB) | One H100 NVL (94 GB) |
|---|---|---|---|
| 7B to 8B class, about 15 GiB | Fits, 24 requests | Fits, 50 requests | Fits, with more room still |
| 32B to 34B class, about 61 to 63 GiB | Does not fit | Fits, 5 requests | Fits, 11 requests |
"Requests" here means requests of 8,192 tokens held at the same moment, a worst case. Two lessons follow. For small models, the 48 GB card is enough, and extra memory only buys concurrency. For 30B-class models at full precision, 80 GB is the entry ticket, and even there the room for requests is tight.
Quantized builds change the table. A 4-bit build of a 34B model is about 17 GiB and fits on a 48 GB card with room to spare, at some cost in precision.
When one card is not enough, you combine several, and here the three differ sharply. Splitting a model with tensor parallelism makes the cards exchange data at every step. NVLink is the fast connection built for that. The A100 and H100 have it; NVIDIA's page for the L40S lists NVLink support as "No".
The vLLM documentation draws the consequence directly. For GPUs without NVLink it names the L40S as its example and recommends pipeline parallelism, which splits the model by layers, over tensor parallelism, for higher throughput and lower communication overhead.
| Plan | A100 or H100 | L40S |
|---|---|---|
| One model on one card | Simple | Simple |
| Several copies, one per card, behind a load balancer | Works well | Works well |
| One large model across cards | Tensor parallelism over NVLink | Possible, with pipeline parallelism; expect more overhead |
| Several small services on one card | MIG, up to 7 slices | No MIG; share with --gpu-memory-utilization |
So the L40S is at its best running models that fit on one card, replicated as needed. It is a poor base for a model that must span cards.
NVIDIA lists FP8 performance for the L40S and H100, not for the A100. The Qwen documentation describes the practical effect in vLLM: FP8 model builds run natively on Ada Lovelace and Hopper cards, and since vLLM 0.9.0 they also run on Ampere cards such as the A100 in a mixed mode. If you plan to serve FP8 builds to save memory, the two newer architectures are the natural home.
Prices on the day this was written, per GPU per hour, on demand:
| Card | Runpod, Secure Cloud | Runpod, Community Cloud | Lambda, 8-GPU instances |
|---|---|---|---|
| A100 80GB | $1.79 | from $1.19 | $3.19 |
| H100 | $2.89 (PCIe), $3.99 (SXM) | from $1.99 | $4.59 |
| L40S | Not listed on the pages read; look it up at your provider | ||
The two providers sell different things, single cards against large multi-GPU machines with fast networking, so compare within a column, not across. Within each, the H100 costs about 1.4 to 1.6 times the A100.
Two sums turn a price into a decision.
Price per GiB of memory. If memory is your limit, this is the number. At $1.79 an A100 80GB costs about 2.2 cents per GiB per hour; at $2.89 an H100 costs about 3.6 cents. The same memory is cheaper on the older card.
Price per million tokens. If throughput is your limit, this is the number:
cost per million output tokens = price per hour ÷ (tokens per second × 3600) × 1,000,000
An example with invented throughput, to show the arithmetic only: a $1.79 card producing 1,000 tokens a second costs $0.50 per million tokens. A $2.89 card producing 2,000 costs $0.40. The dearer card wins if it is at least 1.6 times as fast on your workload, and loses otherwise.
A card that is on all month costs real money: at $1.79 an hour, about $1,290 for 30 days; at $2.89, about $2,080. If your traffic is concentrated in working hours, stopping the server at night matters more than the choice of card.
Hourly rental makes the comparison cheap. Rent each candidate for an hour and run the same test on both:
vllm bench serve with prompt and answer lengths like your real traffic, at the concurrency you expect. The command is in the sizing guide.Two hours of rental answers a question that could otherwise cost you months on the wrong hardware.
| Your situation | Look first at | Why |
|---|---|---|
| Models up to about 8B, or larger ones quantized | L40S | 48 GB is enough on one card |
| A 30B-class model at full precision, lowest cost | A100 80GB | The cheapest way to get 80 GB |
| The same model, and latency is what users complain about | H100 | More bandwidth; confirm with a test |
| A model that must span several cards | A100 or H100 | NVLink |
| FP8 model builds | L40S or H100 | Native support |
| Several small services on one card | A100 or H100 | MIG |
| Long contexts with many users on a 30B-class model | H100 NVL, or two 80 GB cards | The room for requests is the limit |
Whichever card you choose, the next step is the same: work out how many you need. That is the subject of How many GPUs for N users?
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
Specifications are from NVIDIA's product pages and prices from Runpod's and Lambda's public pricing pages, all read on November 12, 2026; prices change often, so check them again before deciding. The memory figures are calculated from model files with a 10 percent runtime reserve, which is an assumption. No GPU was rented or benchmarked for this article, so it gives no tokens-per-second figures; the last section shows how to get your own. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.