High-Throughput LLM Serving with vLLM & PagedAttention: Production Architecture & Optimization

1. The KV Cache Bottleneck in LLM Serving

In large language model inference, computation is bifurcated into two distinct phases:

1. Prefill Phase (Compute-Bound): The entire input prompt is processed concurrently in a single forward pass, populating the initial Key-Value (KV) cache. This phase heavily saturates the GPU's Tensor Cores.
2. Decoding Phase (Memory-Bandwidth Bound): Tokens are generated autoregressively one by one. For every single token generated, the model must read all model parameters and the accumulated KV cache from high-bandwidth GPU memory (HBM) into SRAM.

In legacy serving frameworks, memory allocation for the KV cache was notoriously inefficient. Systems allocated contiguous buffers sized to the model's maximum theoretical context window (e.g., 8k or 32k tokens). Because actual user prompts vary wildly in length, up to 70% to 80% of GPU VRAM was squandered on unallocated memory reservations, severely capping the number of concurrent requests a server could handle.

2. Virtual Memory Paging Applied to GPUs: PagedAttention

Developed by researchers at UC Berkeley and the vLLM team, PagedAttention solves KV memory fragmentation by borrowing the foundational concept of virtual memory and paging from operating systems.

Rather than requiring contiguous physical VRAM allocations:

• Block Partitioning: The KV cache is divided into fixed-size physical memory blocks, each holding keys and values for a small number of tokens (typically 16 or 32 tokens).
• Block Tables: A lightweight block table maps logical token positions in a sequence to non-contiguous physical blocks anywhere in GPU memory.
• Zero Waste: Physical blocks are allocated dynamically on-demand only as tokens are generated. Waste is strictly confined to the final incomplete block of a sequence (reducing fragmentation to under 4%).

Furthermore, PagedAttention unlocks near-instantaneous parallel sampling and beam search. When multiple generation candidates branch from a single prompt, they share physical blocks for the prompt prefix. New blocks are allocated only when tokens diverge (Copy-on-Write), shrinking memory overhead for multi-branch exploration by up to 55%.

3. Continuous (Iteration-Level) Batching Mechanics

In traditional static batching, requests were grouped together and executed until the longest request finished generating. Shorter requests that finished early sat idle in GPU memory, wasting valuable compute cycles.

vLLM implements Continuous Batching (Orca architecture). Rather than waiting at the batch level, scheduling happens at the iteration level. After every single token generation cycle, completed requests are immediately evicted from the batch and returned to the client, while newly arrived requests from the queue are injected into the active batch without interrupting ongoing generations.

4. Production vLLM Docker Deployment with FP8 Quantization

To deploy a production-grade inference server serving DeepSeek-R1-Distill-Qwen-32B across dual NVIDIA RTX 3090/4090 GPUs, execute:

docker run --gpus all 
  -p 8000:8000 
  --ipc=host 
  -v /data/models:/root/.cache/huggingface 
  vllm/vllm-openai:latest 
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B 
  --tensor-parallel-size 2 
  --max-model-len 16384 
  --gpu-memory-utilization 0.92 
  --enable-chunked-prefill 
  --kv-cache-dtype fp8 
  --trust-remote-code

Key configuration flags:

• --tensor-parallel-size 2: Splits model weights across 2 GPUs using NCCL communication.
• --gpu-memory-utilization 0.92: Allocates 92% of free VRAM strictly to PagedAttention blocks after model weights are loaded.
• --kv-cache-dtype fp8: Compresses KV cache to 8-bit floats, halving memory footprint and doubling concurrent user capacity.

5. Chunked Prefill & Latency SLA Management

In real-world API services, arriving requests have asymmetric lengths. If a client submits a 64k token document for summarization, computing the prefill would monopolize the GPUs for 800ms, causing extreme latency spikes (jitter) for 50 other interactive chatbot users waiting for single-token decodes.

vLLM's Chunked Prefill breaks large prefill prompts into smaller chunks (e.g. 512 tokens). In each execution iteration, the engine pairs a 512-token prefill chunk with ongoing decoding requests, maintaining predictable sub-30ms Inter-Token Latencies (ITL) while absorbing massive prompt spikes.

6. Throughput Benchmarks: vLLM vs HuggingFace TGI vs Ollama

The table below demonstrates benchmark throughput serving Meta Llama 3.3 70B Instruct on an 8x NVIDIA A100 (80GB) node under 64 concurrent user streams:

To forecast hardware requirements and compare hosting costs against commercial API subscriptions, explore our LLM GPU VRAM Calculator and Token Cost Estimator.

7. Frequently Asked Questions