Seongwoo Choi / Journal

A100 vs L40S vs H100 for LLM inference: how to choose

At a glanceChoose a GPU for serving language models by memory first, then interconnect, then price per token: NVIDIA's specifications, cloud prices and worked sums.

Not testedWritten from NVIDIA's specifications and public cloud prices. No GPU was rented or timed.

In this article 7 sections
  1. 1. The three cards, side by side
  2. 2. First question: does your model fit?
  3. 3. Second question: can the cards work together?
  4. 4. A detail that matters for quantized models
  5. 5. Third question: what does it cost?
  6. 6. Get your own number in an afternoon
  7. 7. A short guide to choosing

The A100, L40S and H100 are the three NVIDIA cards you are most likely to be offered for serving a language model, whether you rent by the hour or inherit a server. Marketing pages rank them by speed. For inference, speed is the third question. The first is whether your model fits, and the second is whether the cards can work together. This guide takes them in that order.

1. The three cards, side by side

A100 80GBL40SH100
ArchitectureAmpereAda LovelaceHopper
Memory80 GB48 GB80 GB, or 94 GB on the NVL model
Memory bandwidth1,935 to 2,039 GB/s864 GB/s3.35 to 3.9 TB/s
NVLink between cardsYesNoYes
Split into smaller GPUs (MIG)Up to 7NoUp to 7
FP8 listed in the specificationsNoYesYes
Maximum power300 W, or 400 W for the SXM model350 WUp to 700 W for SXM; 350 to 400 W for NVL

All figures are from NVIDIA's product pages. An A100 with 40 GB also exists in older servers; everything below about the A100 80GB applies to it with half the memory.

Newer generations exist as well. Lambda's price list already includes the B200 with 180 GB per GPU. The older cards remain the practical choice for many teams because they are cheaper and easier to get.

From our YouTube channel, AI2Now. The player loads only when you press play.

2. First question: does your model fit?

Memory decides which models you can run at all, and how many requests fit beside them. Every request holds working memory, the KV cache, for as long as it is being answered. The method is in the sizing guide; here are its results for two models, keeping a tenth of each card in reserve.

Qwen3-8B on one L40S, 48 GiB24 requests of 8,192 tokens

weightscache

Qwen3-8B on one A100 or H100, 80 GiB50 requests of 8,192 tokens

weightscache

Qwen2.5-Coder-32B on one A100 or H100, 80 GiB5 requests of 8,192 tokens

weights

Model weightsKV cache, the room for requestsRuntime reserve

Drawn to scale. A 32B model at full precision does not fit on one 48 GB card at all.
Model, at 16-bit precisionOne L40S (48 GB)One A100 or H100 (80 GB)One H100 NVL (94 GB)
7B to 8B class, about 15 GiBFits, 24 requestsFits, 50 requestsFits, with more room still
32B to 34B class, about 61 to 63 GiBDoes not fitFits, 5 requestsFits, 11 requests

"Requests" here means requests of 8,192 tokens held at the same moment, a worst case. Two lessons follow. For small models, the 48 GB card is enough, and extra memory only buys concurrency. For 30B-class models at full precision, 80 GB is the entry ticket, and even there the room for requests is tight.

Quantized builds change the table. A 4-bit build of a 34B model is about 17 GiB and fits on a 48 GB card with room to spare, at some cost in precision.

3. Second question: can the cards work together?

When one card is not enough, you combine several, and here the three differ sharply. Splitting a model with tensor parallelism makes the cards exchange data at every step. NVLink is the fast connection built for that. The A100 and H100 have it; NVIDIA's page for the L40S lists NVLink support as "No".

The vLLM documentation draws the consequence directly. For GPUs without NVLink it names the L40S as its example and recommends pipeline parallelism, which splits the model by layers, over tensor parallelism, for higher throughput and lower communication overhead.

PlanA100 or H100L40S
One model on one cardSimpleSimple
Several copies, one per card, behind a load balancerWorks wellWorks well
One large model across cardsTensor parallelism over NVLinkPossible, with pipeline parallelism; expect more overhead
Several small services on one cardMIG, up to 7 slicesNo MIG; share with --gpu-memory-utilization

So the L40S is at its best running models that fit on one card, replicated as needed. It is a poor base for a model that must span cards.

4. A detail that matters for quantized models

NVIDIA lists FP8 performance for the L40S and H100, not for the A100. The Qwen documentation describes the practical effect in vLLM: FP8 model builds run natively on Ada Lovelace and Hopper cards, and since vLLM 0.9.0 they also run on Ampere cards such as the A100 in a mixed mode. If you plan to serve FP8 builds to save memory, the two newer architectures are the natural home.

5. Third question: what does it cost?

Prices on the day this was written, per GPU per hour, on demand:

CardRunpod, Secure CloudRunpod, Community CloudLambda, 8-GPU instances
A100 80GB$1.79from $1.19$3.19
H100$2.89 (PCIe), $3.99 (SXM)from $1.99$4.59
L40SNot listed on the pages read; look it up at your provider

The two providers sell different things, single cards against large multi-GPU machines with fast networking, so compare within a column, not across. Within each, the H100 costs about 1.4 to 1.6 times the A100.

Two sums turn a price into a decision.

Price per GiB of memory. If memory is your limit, this is the number. At $1.79 an A100 80GB costs about 2.2 cents per GiB per hour; at $2.89 an H100 costs about 3.6 cents. The same memory is cheaper on the older card.

Price per million tokens. If throughput is your limit, this is the number:

cost per million output tokens = price per hour ÷ (tokens per second × 3600) × 1,000,000

An example with invented throughput, to show the arithmetic only: a $1.79 card producing 1,000 tokens a second costs $0.50 per million tokens. A $2.89 card producing 2,000 costs $0.40. The dearer card wins if it is at least 1.6 times as fast on your workload, and loses otherwise.

Why the H100 may earn its priceIts memory bandwidth is about 1.7 times the A100's, and writing tokens is limited mainly by how fast the card can read the model. That makes a real speed-up plausible. How large it is for your model and prompts is something only a test can tell you.

A card that is on all month costs real money: at $1.79 an hour, about $1,290 for 30 days; at $2.89, about $2,080. If your traffic is concentrated in working hours, stopping the server at night matters more than the choice of card.

6. Get your own number in an afternoon

Hourly rental makes the comparison cheap. Rent each candidate for an hour and run the same test on both:

  1. Start the same model with the same flags on each card. The steps are in vLLM in Docker.
  2. Run vllm bench serve with prompt and answer lengths like your real traffic, at the concurrency you expect. The command is in the sizing guide.
  3. Note the output tokens per second and the 95th-percentile time to first token on each.
  4. Put the tokens per second into the formula above.

Two hours of rental answers a question that could otherwise cost you months on the wrong hardware.

7. A short guide to choosing

Your situationLook first atWhy
Models up to about 8B, or larger ones quantizedL40S48 GB is enough on one card
A 30B-class model at full precision, lowest costA100 80GBThe cheapest way to get 80 GB
The same model, and latency is what users complain aboutH100More bandwidth; confirm with a test
A model that must span several cardsA100 or H100NVLink
FP8 model buildsL40S or H100Native support
Several small services on one cardA100 or H100MIG
Long contexts with many users on a 30B-class modelH100 NVL, or two 80 GB cardsThe room for requests is the limit

Whichever card you choose, the next step is the same: work out how many you need. That is the subject of How many GPUs for N users?

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

Specifications are from NVIDIA's product pages and prices from Runpod's and Lambda's public pricing pages, all read on November 12, 2026; prices change often, so check them again before deciding. The memory figures are calculated from model files with a 10 percent runtime reserve, which is an assumption. No GPU was rented or benchmarked for this article, so it gives no tokens-per-second figures; the last section shows how to get your own. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy