Seongwoo Choi / Journal

AI GPU memory capacity and bandwidth differences: Why it can be slow even when the model fits

At a glanceCapacity determines whether a model fits, while bandwidth affects token generation speed; weights, KV cache, and workspace all contribute to memory requirements.

In this article 8 sections
  1. 1. Capacity and bandwidth answer different questions
  2. 2. Two figures to read from the official specifications
  3. 3. Start by calculating weight size
  4. 4. Calculate the upper limit of token speed
  5. 5. Why it can be slow even when capacity is sufficient
  6. 6. What to check before buying or configuring
  7. 7. Limitations of this calculation
  8. References

Whether an AI model can be loaded onto a GPU and how quickly it produces answers are different questions. Memory capacity (GB) determines whether the model fits, while memory bandwidth (TB/s) greatly affects the upper limit of token generation speed. This article organizes the calculation process by model size, using the two figures on NVIDIA's official H100 specifications page.

1. Capacity and bandwidth answer different questions

Capacity is about whether the model fits. Weight size is calculated by multiplying the number of parameters by the size of the data type, and then adding the KV cache (space for remembering earlier parts of a conversation) and workspace.

Bandwidth is about how quickly the weights can be read. When generating text one token at a time, the GPU spends most of its time reading weights. Therefore, even for the same model, higher bandwidth raises the upper limit of generation speed.

2. Two figures to read from the official specifications

The specifications table on NVIDIA's H100 product page lists 80GB of GPU memory and 3.35TB/s of memory bandwidth, while another column lists 94GB and 3.9TB/s. Check the table headers directly to confirm what configurations the two columns represent. These are vendor-published specifications and do not mean that the same speeds will be achieved on an actual server.

3. Start by calculating weight size

1) Weight size = number of parameters × bytes per data type. Calculate FP16 as 2 bytes, FP8 as 1 byte, and INT4 as 0.5 bytes.

- 70B FP16: 70 × 2 = 140GB. The weights alone do not fit on a single 80GB GPU.

- 70B FP8: 70 × 1 = 70GB. It fits within 80GB, but only about 10GB remains for the KV cache.

- 70B INT4: 70 × 0.5 = 35GB. There is substantial room left, but you must separately check how much quantization changes quality.

- 8B FP16: 8 × 2 = 16GB. It fits comfortably on one card.

4. Calculate the upper limit of token speed

2) Assuming batch size 1 (processing only one request at a time) and that the entire set of weights is read once to generate one token, the upper limit is bandwidth ÷ weight size. Actual speed is lower because of computation, KV cache reads, and software optimization.

- H100 80GB (3.35TB/s), 70B FP8: 3,350GB/s ÷ 70GB ≈ up to 47 tokens/second

- H100 80GB (3.35TB/s), 70B INT4: 3,350 ÷ 35 ≈ up to 95 tokens/second

- H100 80GB (3.35TB/s), 8B FP16: 3,350 ÷ 16 ≈ up to 209 tokens/second

- 94GB · 3.9TB/s column, 70B FP8: 3,900 ÷ 70 ≈ up to 55 tokens/second, with about 24GB of memory remaining after the weights.

5. Why it can be slow even when capacity is sufficient

First, if the weights barely fit in GPU memory, there is not enough space for the KV cache, making it difficult to process long documents or multiple requests simultaneously. Second, distributing the model across multiple GPUs adds communication time for data transfers between GPUs. Third, quantization (a method of reducing weights to smaller data types) reduces capacity requirements, but its effect on quality and support in the execution tool must be checked separately.

6. What to check before buying or configuring

1) Write down the model's parameter count and the data type you will use, then calculate the weight size.

2) Add the KV cache and workspace to estimate the required memory. If you plan to use a long context, leave more room.

3) Calculate the batch-1 speed upper limit as bandwidth ÷ weight size.

4) Measure performance in the execution tool you will actually use, with the same model and input length, and compare it with the upper limit.

5) If you plan to run a local model on a personal PC, also check the storage space and installation order in Specifications to check before installing Ollama on Windows and how to change the model storage location .

7. Limitations of this calculation

The figures in this article are theoretical upper limits based on specifications-table values and simple arithmetic. They should not be interpreted as actual performance. I have no results from directly measuring this GPU, and actual speed varies by benchmark conditions and software version for each model. For API usage, cost calculations can refer to the method of dividing usage into individual categories, as in How to calculate Gemini Flash API costs: Separating input and output tokens .

References

- NVIDIA H100 GPU specifications page (lists GPU memory of 80GB and 94GB and memory bandwidth of 3.35TB/s and 3.9TB/s): https://www.nvidia.com/en-us/data-center/h100/ (checked 2026-11-11 KST)

- Calculation method: Weight size and the token-speed upper limit are values calculated directly using the formulas above.

Sources and original article

Read the Korean original on Naver

This article reflects public information available when it was written. Specifications, prices and schedules can change; check the linked official documentation.

This English edition was written and translated with AI assistance, with sources provided. For errors or additional information, please request a correction. Editorial policy