What is vLLM? PagedAttention and LLM serving, explained
vLLM is an open-source LLM serving engine built on PagedAttention, which stores the KV cache in small blocks like OS pages. How it works, with a Python demo.

vLLM is an open-source engine for serving large language models on GPUs, built around PagedAttention: an attention algorithm that stores each request’s KV cache in small fixed-size blocks instead of one big contiguous slab, the way an operating system pages virtual memory. Less wasted memory means more requests fit in a batch, and a bigger batch means more tokens per second from the same GPU.
It started at UC Berkeley’s Sky Computing Lab, and the paper by Kwon et al. (SOSP 2023) reported 2 to 4x higher throughput than the serving systems of the time at the same latency. Today it’s the default answer when someone asks “how do I self-host an open-weight model behind an API”. I ran the demo below on 2026-10-31 with Python 3.13.11 and no packages.
What problem does vLLM solve?
LLM serving is limited by memory, and most of that memory is the KV cache: the keys and values a transformer stores for every token it has seen, so each new token doesn’t recompute the past. The paper’s opening example is a 13B parameter model on an A100 with 40GB: about 65% of memory holds the weights, and close to 30% holds the KV cache. Weights are fixed. The KV cache is the part you manage, and how well you manage it decides how many requests you can batch.
The systems the paper measured stored each request’s cache in one contiguous chunk, because deep learning frameworks want tensors contiguous. But a request’s output length isn’t known in advance. So they reserved space for the maximum possible length (2,048 tokens in their example) and wasted memory three ways:
- Reserved slots. Space held for tokens that haven’t been generated yet, and that other requests can’t use meanwhile.
- Internal fragmentation. Space reserved for tokens that will never be generated, because the request stopped early.
- External fragmentation. Gaps between differently sized chunks that the allocator can’t hand out.
Their profiling found that only 20.4% to 38.2% of KV cache memory in those systems held actual token states. With vLLM, the figure was 96.3%.
How does PagedAttention work?
The paper states the analogy directly: think of blocks as pages, tokens as bytes and requests as processes.
- Split the cache into blocks. Each block holds the keys and values for a fixed number of tokens. vLLM’s default block size is 16, which the authors picked because it is “large enough to efficiently utilize the GPU and small enough to avoid significant internal fragmentation in most workloads”.
- Keep a block table per request. Like a page table, it maps the request’s logical blocks (tokens 0 to 15, 16 to 31, and so on) to physical blocks anywhere in GPU memory. They don’t need to be adjacent.
- Allocate on demand. A new physical block is taken from the free list only when the last one fills. The most a request can waste is one partly filled block.
- Teach attention to follow the table. The PagedAttention kernel fetches key and value blocks through the block table while computing attention, so the non-contiguous layout costs little.
Because every block is the same size, external fragmentation disappears. Because blocks are allocated as tokens arrive, reserved and internal waste shrink to under one block per request.
Here’s that difference as a simulation. A pool of 65,536 token slots, 200 requests with varied prompt and output lengths, one allocator that reserves 2,048 slots per request and one that hands out 16-token blocks:
# paged.py: contiguous KV reservation vs PagedAttention-style blocks. Python 3 stdlib only.
import math, random
random.seed(7)
POOL = 65_536 # KV cache slots on the GPU, one slot = one token's keys and values
MAX_LEN = 2048 # contiguous systems reserve the maximum length up front
BLOCK = 16 # vLLM's default block size
# 200 requests with prompt and output lengths that vary a lot, as chat traffic does.
reqs = [(random.randint(20, 600), random.randint(10, 700)) for _ in range(200)]
def contiguous(reqs):
# Reserve MAX_LEN slots per request; admit requests until the pool is full.
n = min(len(reqs), POOL // MAX_LEN)
used = sum(p + o for p, o in reqs[:n])
return n, used, n * MAX_LEN
def paged(reqs):
# Allocate fixed-size blocks only as tokens arrive; admit while blocks remain.
free, n, used, held = POOL // BLOCK, 0, 0, 0
for p, o in reqs:
need = math.ceil((p + o) / BLOCK)
if need > free:
break
free -= need; n += 1; used += p + o; held += need * BLOCK
return n, used, held
for name, fn in (("contiguous (reserve 2048)", contiguous), ("paged (16-token blocks)", paged)):
n, used, held = fn(reqs)
print(f"{name:<26} requests in batch {n:>3} slots held {held:>6} "
f"holding real tokens {used / held:6.1%}")
# Prefix sharing: 8 parallel samples of one 512-token prompt.
prompt_blocks = 512 // BLOCK
print(f"\n8 samples, shared 512-token prompt: {8 * prompt_blocks} prompt blocks unshared, "
f"{prompt_blocks} shared")
Output (Python 3.13.11):
contiguous (reserve 2048) requests in batch 32 slots held 65536 holding real tokens 31.6%
paged (16-token blocks) requests in batch 97 slots held 65504 holding real tokens 99.0%
8 samples, shared 512-token prompt: 256 prompt blocks unshared, 32 shared
Same memory, three times the batch. The toy simplifies one thing in vLLM’s favour and one against it: it sizes paged requests by their final length (a real engine grows them block by block and may have to preempt when memory runs out), and the contiguous allocator gets a perfectly packed pool with no external fragmentation. The 31.6% lands inside the paper’s measured 20.4% to 38.2% range, which is a coincidence of my random lengths, not a reproduction.
Why does sharing blocks matter?
Once the cache is a table of pointers, two requests can point at the same physical block. The paper uses this for:
- Parallel sampling. Ask for several completions of one prompt and the prompt’s blocks are stored once. A block gets copied only when one sequence writes into a shared block, which is copy-on-write, as in an OS
fork(). - Beam search. Beams share long common prefixes, and those shared blocks are reference counted instead of copied.
- Shared prefixes. A system prompt used by many requests can map to the same cached blocks.
The measured savings were 6.1% to 9.8% of memory for parallel sampling and 37.6% to 55.2% for beam search on the Alpaca dataset, and higher on ShareGPT (16.2% to 30.5% and 44.3% to 66.3%).
The modern form of the third item is automatic prefix caching. vLLM hashes every full block from its parent block’s hash, its tokens and extras such as a LoRA ID or cache salt. A new request whose prefix hashes match reuses those blocks and skips their prefill. Eviction is LRU over free blocks. It’s the same mechanism hosted APIs sell as prompt caching.
What else is in vLLM besides PagedAttention?
PagedAttention was the paper. The engine has since grown a long list, per its README:
- Continuous batching. The paper builds on iteration-level scheduling from Orca (Yu et al., OSDI 2022): after every decoding step, finished requests leave the batch and waiting ones join, so nobody waits for the slowest request in a batch.
- Chunked prefill. Long prompts are split so their prefill doesn’t stall everyone else’s decode. The optimization docs say it’s on by default whenever possible.
- Quantization in many formats (FP8, INT8, INT4, GPTQ, AWQ, GGUF and others). See what is quantization.
- Speculative decoding, including n-gram and EAGLE methods.
- Optimized attention kernels, including FlashAttention and FlashInfer.
- Tensor, pipeline, data, expert and context parallelism for models that span GPUs.
- An OpenAI-compatible API server, plus multi-LoRA serving and structured outputs.
Getting a server up is one command. The quickstart lists Linux as the OS and starts a server at http://localhost:8000:
uv pip install vllm
vllm serve Qwen/Qwen2.5-1.5B-Instruct
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}'
Any client that speaks the OpenAI API can point at that URL, which is most of why teams adopt it.
What are the knobs that matter?
Three settings decide most of what you’ll see in production:
gpu_memory_utilization. vLLM pre-allocates this fraction of GPU memory up front, and whatever the weights don’t take becomes the KV block pool. Raise it for more cache; lower it if something else shares the GPU.max_num_seqs. The cap on concurrent sequences. If the logs show preemptions, the docs suggest lowering it or giving the cache more memory. In the current V1 engine, preempted requests are recomputed by default rather than swapped to CPU memory.max_num_batched_tokens. Tokens processed per step. The docs say smaller values (they give 2048) improve inter-token latency because fewer prefills interrupt decodes, while larger values help time to first token and throughput.
Those last two are a direct trade between latency and throughput, which I break down in LLM latency explained.
Where does vLLM fit, and where doesn’t it?
vLLM is built for many concurrent users on datacenter accelerators. Its README lists NVIDIA, AMD and Intel GPUs, x86, ARM and PowerPC CPUs, and hardware plugins for TPUs, Gaudi, Ascend, Apple Silicon and others. The quickstart’s prerequisites name Linux and point Mac users to vLLM-Metal.
For one person on a laptop, its strengths don’t apply. A single user has a batch of one, so there’s nothing for paged memory and continuous batching to pack. Ollama, llama.cpp or MLX are the better tools there, covered in how to run LLMs locally.
It’s also not the only serving engine, and I haven’t benchmarked the alternatives against it. Published comparisons go stale with every release, so test any engine on your own model and traffic shape before choosing.
What I would do
To self-host an open-weight model for an internal app, I’d start with vllm serve and its OpenAI-compatible endpoint, so application code stays the same as with a hosted API and switching back is a config change.
Before load testing, I’d compute the KV cache budget: GPU memory times gpu_memory_utilization, minus the weights, divided by KV bytes per token (the formula is in the KV cache post). That tells me roughly how many tokens of context the server can hold across all users, which is the number that decides batch size.
Then I’d replay a sample of real prompts, not synthetic ones, because the gains depend on length distribution and prefix reuse. Put the system prompt and tool definitions first so prefix caching can hit, and watch preemption counts in the metrics. From a data-platform seat, the thing I’d circle is multi-tenancy: if tenants share a server, use the cache salt so one tenant’s cached prefix can’t be matched by another’s request.
FAQ
What does vLLM stand for?
The README and paper don’t expand the name. It’s the project name for the serving engine from Berkeley’s Sky Computing Lab, introduced in the PagedAttention paper.
Is vLLM faster than Hugging Face Transformers?
For serving many concurrent requests, generally yes, because continuous batching and paged KV memory let it batch far more requests. The paper’s comparisons were against FasterTransformer and Orca, reporting 2 to 4x throughput at the same latency. Measure on your workload.
Can I run vLLM on a Mac?
The quickstart lists Linux as the supported OS and mentions vLLM-Metal for Apple Silicon GPUs. For a single user on a Mac, Ollama, llama.cpp or MLX are simpler choices.
What is the difference between PagedAttention and FlashAttention?
PagedAttention is about where the KV cache lives: fixed-size blocks mapped through a table, so memory isn’t wasted between requests. FlashAttention is about computing attention without materializing the full score matrix. vLLM uses both.
Does vLLM change model outputs?
PagedAttention is exact; the paper reports throughput gains without affecting model accuracy. Optional features such as quantization or an FP8 KV cache are lossy and need their own evaluation.
Sources
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, September 12, 2023 (SOSP 2023). Memory breakdown in section 1, waste figures in Figure 2, block size in section 7.2, sharing savings in section 6.3
- Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022
- vLLM, GitHub README (read 2026-10-31)
- vLLM docs, Quickstart (read 2026-10-31)
- vLLM docs, Automatic prefix caching (read 2026-10-31)
- vLLM docs, Optimization and tuning (read 2026-10-31)