2 posts tagged with "Memory Management"

When serving Large Language Models (LLMs) in production, you quickly run into a counterintuitive reality: serving LLMs is rarely bottlenecked by raw GPU compute power (FLOPs). It is almost always bottlenecked by GPU High Bandwidth Memory (HBM).

An NVIDIA H100 GPU costs upwards of $30,000 and comes with 80 GB of VRAM. A 70-billion parameter model like LLaMA-3 in FP16 precision consumes 140 GB of memory just to load model weights across multiple GPUs.

Whatever precious VRAM remains must hold the dynamic KV Cache (Key-Value Cache) for all concurrent incoming user requests.

In traditional inference servers (like naive HuggingFace Transformers or early FasterTransformer), up to 60% to 80% of that KV-cache memory was completely wasted due to fragmentation and over-allocation.

Then came vLLM and PagedAttention (developed by UC Berkeley researchers in 2023). By resurrecting a 60-year-old fundamental concept from classical Operating Systems — Virtual Memory Paging — they revolutionized AI serving, slashing memory waste to under 4% and boosting throughput by 2x to 4x.

Here is how PagedAttention works from first principles, under the hood.

A very common misconception in Operating Systems is that every page fault means the OS must access the disk.
That is not always true.

A page fault simply means that the required page is not usable in the current context.
This can happen for multiple reasons.

In many cases, the page is already present in main memory, but: