Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

How much VRAM does an LLM need? A calculator for 8B to 405B

How much VRAM an LLM needs: parameters times bytes per weight, plus the KV cache for your context. Worked numbers for 8B to 405B models, in a Python script.

How much VRAM does an LLM need? A calculator for 8B to 405B

To run an LLM you need enough GPU memory (VRAM, or unified memory on a Mac) for the weights plus the KV cache, plus some headroom for the runtime. Weights are parameters times bytes per weight: about 2 GB per billion parameters at 16 bits and roughly 0.6 GB per billion at 4-bit Q4_K_M. The KV cache adds memory that grows with context length, from 4 GiB to 16 GiB at 32k tokens for the models below.

I ran the snippets below on 2026-10-31 with Python 3.13.11 and no packages.

What takes up memory when you run an LLM?

Three things, in order of size for a typical local setup:

  1. Weights. Every parameter, stored at some precision. This is fixed for a given model file. A dense model reads all of them for every token it generates.
  2. The KV cache. For each token in the context, every layer stores a key and a value vector per KV head, so the model doesn’t recompute them. It grows linearly with context length and with the number of concurrent requests. The mechanism is in what is the KV cache.
  3. Runtime overhead. Activations for the current step, scratch buffers, the CUDA or Metal context, and fragmentation. It varies by engine and batch size, so I budget headroom for it rather than a formula.

The formulas for the first two:

  • weights = parameters × bits per weight ÷ 8
  • KV cache = 2 × layers × KV heads × head dim × bytes per value × tokens

Bits per weight is 16 for BF16 or FP16, 8.5 for llama.cpp’s Q8_0, and about 4.9 for Q4_K_M. Those last two are measured averages from the llama.cpp quantize README, and they’re above 8 and 4 because each block of weights also stores a scale. What is quantization explains where the extra half bit comes from.

How much VRAM do 8B, 32B, 70B and 405B models need?

Here’s the calculator. Layer counts and KV heads come from the Llama 3 paper (Table 3) and the published config.json files for Qwen3 32B and Mixtral 8x7B. Parameter counts are the nominal ones in the model names.

# vram.py: memory to run an LLM = weights + KV cache. Python 3 standard library only.
GiB = 2 ** 30

# (nominal params in billions, layers, KV heads, head dim), from papers and config.json files
models = {
    "Llama 3.1 8B":   (8,   32, 8, 128),
    "Qwen3 32B":      (32,  64, 8, 128),
    "Mixtral 8x7B":   (47,  32, 8, 128),   # all 47B must be loaded, though 13B are active
    "Llama 3 70B":    (70,  80, 8, 128),
    "Llama 3 405B":   (405, 126, 8, 128),
}
# bits per weight: 16-bit, and llama.cpp's measured Q8_0 and Q4_K_M averages for Llama 3.1 8B
formats = {"BF16": 16, "Q8_0": 8.5008, "Q4_K_M": 4.8944}

def weights_gib(params_b, bpw):
    return params_b * 1e9 * bpw / 8 / GiB

def kv_gib(layers, kv_heads, head_dim, tokens, bytes_per=2):
    return 2 * layers * kv_heads * head_dim * bytes_per * tokens / GiB   # K and V, 16-bit

print(f"{'model':<15}" + "".join(f"{f:>9}" for f in formats) + f"{'KV 8k':>8}{'KV 32k':>8}{'Q4_K_M+32k':>12}")
for name, (p, L, kv, hd) in models.items():
    w = {f: weights_gib(p, b) for f, b in formats.items()}
    k8, k32 = kv_gib(L, kv, hd, 8192), kv_gib(L, kv, hd, 32768)
    print(f"{name:<15}" + "".join(f"{v:>9.1f}" for v in w.values())
          + f"{k8:>8.1f}{k32:>8.1f}{w['Q4_K_M'] + k32:>12.1f}")

# Full fine-tuning with mixed-precision Adam: 16 bytes per parameter (ZeRO paper), before activations.
print()
for name, (p, *_ ) in list(models.items())[:2]:
    print(f"{name}: full fine-tune model states = {p * 1e9 * 16 / GiB:.0f} GiB")
model               BF16     Q8_0   Q4_K_M   KV 8k  KV 32k  Q4_K_M+32k
Llama 3.1 8B        14.9      7.9      4.6     1.0     4.0         8.6
Qwen3 32B           59.6     31.7     18.2     2.0     8.0        26.2
Mixtral 8x7B        87.5     46.5     26.8     1.0     4.0        30.8
Llama 3 70B        130.4     69.3     39.9     2.5    10.0        49.9
Llama 3 405B       754.4    400.8    230.8     3.9    15.8       246.5

Llama 3.1 8B: full fine-tune model states = 119 GiB
Qwen3 32B: full fine-tune model states = 477 GiB

All columns are GiB. Two sanity checks against published numbers. The llama.cpp README measures Llama 3.1 8B at 14.96 GiB in F16 and 4.58 GiB in Q4_K_M; the script gives 14.9 and 4.6. It lists 70B at Q4_K_M as 43.1 GB, which is 40.1 GiB; the script gives 39.9. Close enough to plan with. Q4_K_M’s average bits per weight shifts a little from model to model, so treat the rows above as plus or minus a few percent.

And the 405B row matches what Meta reported: in the Llama 3 paper the BF16 405B “does not fit in the GPU memory of a single machine with 8 Nvidia H100 GPUs”, whose 80GB each add up to 640GB. The script says 754 GiB of weights alone. Meta served it across 16 GPUs on two machines.

Why does the KV cache matter so much?

Because it’s the part people forget, and it multiplies. The KV columns above are for one sequence. Serve eight users at 32k tokens each with Llama 3 70B and the cache alone is 80 GiB, double the Q4_K_M weights. Every model in the table uses grouped-query attention with 8 KV heads, which already keeps the cache small; a model with full multi-head attention would need several times more per token.

For a single person running a model locally, context length is the knob. Most local runtimes let you set it, and asking for the model’s maximum context reserves memory you may never use.

Do MoE models need less VRAM?

No, they need less compute. Mixtral 8x7B has 47B parameters but uses only 13B active parameters per token, because a router sends each token to 2 of 8 experts per layer. All 47B still have to be in memory, since the next token may pick different experts. So for sizing, an MoE model costs its total parameter count, while its speed behaves more like a model of its active size. I go into the routing in what is mixture of experts.

What fits on a 24GB Mac?

Malhar’s machine is an M4 MacBook Air with 24GB of unified memory, and the same arithmetic applies to it with one difference: on Apple silicon the GPU and CPU share one pool, so “VRAM” is whatever macOS and your open apps leave free, not the full 24GB.

Reading the table as arithmetic, not as a benchmark:

  • 8B at Q4_K_M or Q8_0 fits with room for long context: 4.6 or 7.9 GiB of weights, plus 4 GiB of cache at 32k.
  • 32B at Q4_K_M is 18.2 GiB of weights. With 32k of context it’s 26.2 GiB, which doesn’t fit. With a short context it’s on the edge, competing with the OS and a browser.
  • Mixtral 8x7B and 70B don’t fit at 4 bits. You’d need a 2 or 3-bit quant, and that’s a different quality trade.

Fitting is only half of it. Decoding reads every active weight for every token, so memory bandwidth caps speed. Apple lists the M4 at 120GB/s of memory bandwidth. Dividing bandwidth by model bytes gives a ceiling on tokens per second for a dense model, not a measurement: roughly 24 for an 8B at Q4_K_M (4.9GB) and about 6 for a 32B at Q4_K_M (19.5GB). Real numbers come in below that ceiling. LLM latency explained has the full reasoning, and how to run LLMs locally covers the tools.

How much VRAM do you need to fine-tune?

Far more than to run. The ZeRO paper (Rajbhandari et al., 2019) breaks down mixed-precision Adam: 2 bytes per parameter for FP16 weights, 2 for gradients, and 12 for the optimizer’s FP32 copy of the weights, momentum and variance. That’s 16 bytes per parameter before activations. Their example: a 1.5B GPT-2 needs 3GB for its FP16 weights but “at least 24 GB” to train. The script’s last lines apply the same rule: 119 GiB for an 8B model.

That gap is why parameter-efficient fine-tuning exists. QLoRA freezes a 4-bit base model and trains small adapters, which brought 65B fine-tuning down to “a single 48GB GPU”. What is LoRA has the details.

What I would do

Before downloading anything, I’d do the two multiplications in this post: parameters times bits per weight for the file, and the KV formula from the model’s config.json times the context I actually need. Then I’d leave headroom for the runtime and, on a Mac, for everything else the machine is doing. If the total is close to the limit, I’d cut context before cutting bits, because a smaller cache costs nothing in quality for prompts that fit.

For picking a model on fixed hardware, my order is: the largest model that fits at Q4_K_M with the context I need, then a step up in quant if there’s room left. Going below 4 bits to squeeze in a bigger model is the last resort, and I’d only do it after testing on my own prompts.

For a team serving users, the calculation flips: weights are paid once, and the KV cache is paid per concurrent request. That’s the number that sets how many GPUs you buy, and engines like vLLM exist mostly to manage it.

FAQ

How much VRAM do I need for a 7B or 8B model?

About 15 GiB at 16 bits, 8 GiB at Q8_0 and under 5 GiB at Q4_K_M for the weights, plus about 1 GiB of KV cache per 8k tokens of context for Llama 3.1 8B.

How much VRAM does a 70B model need?

About 130 GiB at 16 bits and about 40 GiB at Q4_K_M for the weights, plus 2.5 GiB of KV cache per 8k tokens for Llama 3 70B. At 4 bits that’s two 24GB GPUs or a machine with 64GB or more of unified memory.

Is unified memory on a Mac the same as VRAM?

For sizing, yes: the GPU uses the same memory pool as the CPU. But the OS and your apps share it, so the usable amount is less than the number on the box, and Macs have less bandwidth than datacenter GPUs.

What is a quick rule of thumb for LLM memory?

Billions of parameters times 2 for GB at 16 bits, times about 0.6 at Q4_K_M, then add the KV cache for your context and some headroom.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.