Estimate GPU Memory for LLM Inference

A model's weight file fits inside the advertised GPU capacity, yet inference runs out of memory when a second request arrives. The missing quantity is often live request state. Plan for weights, the key/value cache, temporary workspaces and runtime overhead together.

Calculate a useful lower bound

As a first approximation, weight memory equals the number of parameters multiplied by bytes per parameter. Seven billion parameters at two bytes each require about 14 GB in decimal units, or roughly 13 GiB. This excludes quantization metadata, auxiliary tensors, allocations and any duplicated copies. A four-bit format is not a guarantee of exactly half a byte of total runtime memory per parameter.

For a conventional decoder with cached keys and values, a rough per-token KV estimate is two times the layer count, times the number of KV heads, times the head dimension, times bytes per element. Use the model's KV head count, which can differ from its query head count. Architectures and serving engines can store state differently.

Work the example

With 32 layers, 8 KV heads, a head dimension of 128 and two-byte cache elements, the estimate is 131,072 bytes per token. A single 4,096-token sequence would use about 512 MiB of KV cache under these assumptions. Four such active sequences approach 2 GiB before additional overhead. Count both prompt and generated tokens that remain in context.

Paged allocation can reduce fragmentation and permit sharing in supported cases. It does not make occupied token state free.

Validate under the real request mix

Measure peak allocated memory during prompt processing and decoding. Include long prompts, maximum outputs and simultaneous requests. Record time to first token, subsequent token rate and failures, not only GPU utilization. Leave operating headroom and specify what happens at capacity: queue, reject or reduce admitted concurrency. An explicit admission policy is preferable to discovering the limit through production crashes.

Further reading

Kwon et al.: PagedAttention

Related guide: Measure Storage Latency Before Buying Faster Disks.