Why vLLM is fast
Serving a language model to many people at once is mostly a memory problem. Watch a KV cache fill, fragment and page, then race four serving strategies.
A language model writes one token at a time. To pick each new token it needs to “attend” to every token before it, and recomputing all of that from scratch at every step would be hopelessly slow. So the server keeps intermediate results for every token so far, the keys and values, in GPU memory. That store is the KV cache. It grows by one token’s worth with every token generated, for every conversation in progress.
That makes serving a memory problem. The more conversations whose caches fit in GPU memory at once, the more people you serve per second on the same expensive GPU.
How big is the cache?
It’s easy to underestimate. Pick a model and a workload:
In the vLLM paper’s own example, a 13-billion-parameter model on a 40 GB A100, the weights take about 65% of GPU memory and the KV cache close to 30%. How well that 30% is managed decides how many people you can serve.
The waste
Before vLLM, serving systems gave each request one contiguous slab of memory sized for the longest answer it might give, for example 2,048 tokens. Most answers are much shorter, and nobody knows in advance how long each will be. The slab sits mostly empty. The holes left between slabs of different sizes are too small to reuse.
The vLLM authors measured this. In the systems they compared against, only 20.4% to 38.2% of KV cache memory held actual token data. vLLM held 96.3%.
What vLLM changed
PagedAttention borrows the oldest trick in operating systems: virtual memory. The cache is cut into small fixed-size blocks (16 tokens here) that are handed out only as a conversation actually grows, and they don’t need to sit next to each other. Waste is at most one partly filled block per conversation.
Continuous batching (Orca calls it iteration-level scheduling) fixes the other leak. A static batch waits for its slowest member to finish before anyone new starts, so seats sit empty. Continuous batching refills a seat the moment a conversation ends.
Run it. Each square is one block. A solid colour means tokens are stored there, a pale tint means reserved but still empty, and grey means free. Try “reserve the max” first, then switch to pages.
Why does reserving memory for the longest possible answer waste so much?
Answer lengths vary widely and are unknown at the start. A reservation sized for the worst case is mostly empty for the typical case. The vLLM paper measured 20-38% of cache memory actually used in such systems.
With paged blocks of 16 tokens, what's the most memory a single conversation can waste?
Blocks are handed out only when the previous one is full, so only the newest block can be partly empty. That's at most 15 empty token slots per conversation, whatever its length.
Paged memory runs out mid-answer. What did you see the simulator (and vLLM) do?
Preemption trades some repeated work for not refusing new requests. Watch the evictions counter: it isn't free, because the evicted conversation's tokens are generated again.
Everything above ran in your browser. Now do it on a real cluster withthe tilt platform. You need Docker, kind and Tilt, about 15 GB of free disk and 8 GB of spare RAM. Setup guide · Can my machine run it?
A real vLLM server needs a supported GPU. The tilt platform runs Ollama instead, which serves small models on an ordinary CPU. Turn it on in tilt-config.json (flux_apps.ollama.enabled: true) and see Run your own model. With a GPU:
pip install vllm
vllm serve Qwen/Qwen2.5-0.5B-InstructRead the startup log: vLLM reports its KV cache size in tokens, and the maximum concurrency that leaves at the model’s full context length. Divide the tokens by the block size (16 tokens by default) for the number of blocks. That number is this whole lesson.
Sources
Missions and quizzes above earn their own XP. This marks the lesson done.