## The Memory Crisis in High-Concurrency LLM Serving When serving Large Language Models (LLMs) like LLaMA-3, Mistral, or DeepSeek, the primary performance bottleneck is not raw FLOPS compute—it is **GPU High Bandwidth Memory (HBM) capacity**. During token generation, the model saves Key and Value tensors for all previous tokens in what is called the **KV Cache**. Prior to vLLM, standard serving systems (like HuggingFace Transformers) suffered from **60% to 80% wasted GPU memory**: - **Internal Fragmentation:** Memory was reserved upfront for a request's maximum output length (e.g. 2,048 tokens), even if the model finished in 50 tokens. - **External Fragmentation:** Memory allocations had to be contiguous, leaving small unusable gaps between allocations. Then researchers at UC Berkeley released **[`vllm-project/vllm`](https://github.com/vllm-project/vllm)** (40,000+ ⭐), demonstrating that this is the exact problem operating system architects solved in the 1960s with **Virtual Memory and Paging**. --- ## 1. How PagedAttention Works PagedAttention divides the continuous KV cache of a request into fixed-size **KV Blocks** (typically 16 or 32 tokens). - Tensors are stored in non-contiguous physical GPU memory pages. - A **Block Table** maps logical token sequences to physical GPU memory addresses, identical to an OS Page Table. - As new tokens generate, vLLM allocates new physical blocks on demand without pre-reserving maximum lengths! ```text Logical Token Stream: [Token 0 ... 15] -> Block Table -> Physical GPU Block #42 [Token 16 ... 31] -> Block Table -> Physical GPU Block #108 [Token 32 ... 47] -> Block Table -> Physical GPU Block #12 ``` Because memory is allocated in discrete block chunks, wasted memory drops to **less than 4%** (only the unused slots in the final block of a sequence). --- ## 2. Serving 24x Higher Throughput with vLLM in Python ```python from vllm import LLM, SamplingParams # Configure model with PagedAttention engine llm = LLM( model="meta-llama/Meta-Llama-3-8B-Instruct", gpu_memory_utilization=0.90, max_model_len=4096, tensor_parallel_size=1 ) sampling_params = SamplingParams( temperature=0.7, top_p=0.95, max_tokens=256 ) prompts = [ "Explain monotonic clock vs wall clock in Python.", "Why does shadcn/ui avoid npm package publishing?", "How does vLLM optimize GPU KV cache allocation?" ] # Continuous batching processes all prompts concurrently with zero wasted memory outputs = llm.generate(prompts, sampling_params) for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Generated {len(output.outputs[0].token_ids)} tokens.") ``` --- ## 3. Copy-on-Write for Parallel Sampling and Beam Search When generating multiple candidate responses for the same prompt, traditional systems duplicated the prompt's KV cache multiple times. PagedAttention enables **Copy-on-Write**: Multiple output streams share the exact same prompt KV blocks in GPU memory. Only when their generated tokens diverge does vLLM allocate new physical blocks, slashing memory requirements for beam search by up to 55%. --- ## 4. Key Takeaways from `vllm-project/vllm` 1. **Classic Computer Science Principles Endure:** When facing modern AI scaling challenges, OS fundamentals (virtual memory, page tables, continuous batching) often hold the optimal solution. 2. **Eliminate Contiguous Memory Requirements:** Non-contiguous block allocation eliminates memory fragmentation. 3. **Explore the Repository:** [github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)