One post tagged with "Large Language Models"

When serving Large Language Models (LLMs) in production, you quickly run into a counterintuitive reality: serving LLMs is rarely bottlenecked by raw GPU compute power (FLOPs). It is almost always bottlenecked by GPU High Bandwidth Memory (HBM).

An NVIDIA H100 GPU costs upwards of $30,000 and comes with 80 GB of VRAM. A 70-billion parameter model like LLaMA-3 in FP16 precision consumes 140 GB of memory just to load model weights across multiple GPUs.

Whatever precious VRAM remains must hold the dynamic KV Cache (Key-Value Cache) for all concurrent incoming user requests.

In traditional inference servers (like naive HuggingFace Transformers or early FasterTransformer), up to 60% to 80% of that KV-cache memory was completely wasted due to fragmentation and over-allocation.

Then came vLLM and PagedAttention (developed by UC Berkeley researchers in 2023). By resurrecting a 60-year-old fundamental concept from classical Operating Systems — Virtual Memory Paging — they revolutionized AI serving, slashing memory waste to under 4% and boosting throughput by 2x to 4x.

Here is how PagedAttention works from first principles, under the hood.