1. Mathematical Foundations of LLM Memory Sizing
Determining whether a foundation model can execute entirely within local GPU memory without spilling across the PCIe bus requires calculating four discrete memory allocations:
VRAM_total = M_weights + M_KV + M_CUDA + M_activations
A. Model Weights Memory (M_weights)
The baseline static memory consumed by the neural network's parameters. Calculated as:
M_weights = (P × bpp / 8) × 1.04
Where P is total parameter count (e.g. 8.03 billion for Llama 3.1 8B), bpp is bits per parameter (16.0 for FP16, 4.5 for Q4_K_M GGUF), and the 1.04 multiplier accounts for tensor metadata and embedding tables.
B. Key-Value (KV) Cache Memory (M_KV)
During autoregressive generation, the model caches keys and values from past tokens to avoid re-evaluating preceding attention matrices. Under standard Multi-Head Attention (MHA):
M_KV = 2 × n_layers × (d_model × GQA_ratio) × L_context × b × (bpp_kv / 8)
Where GQA_ratio reflects modern Grouped-Query Attention (e.g. 0.25 for Llama 3.1 8B, which uses 8 KV heads for 32 query heads), drastically shrinking cache footprint. For radical architectures like DeepSeek-V3's Multi-Head Latent Attention (MLA), keys and values are compressed into a compact latent vector, reducing KV memory by over 93%.
C. CUDA Runtime & Context Buffer (M_CUDA)
Initializing the PyTorch CUDA runtime, NCCL inter-GPU communication buffers, and cuBLAS execution spaces consumes approximately 600MB to 1.2GB of VRAM regardless of model size.
2. Memory Bandwidth vs Compute: Understanding Tokens/Sec
A common misconception among hardware builders is that GPU compute (TFLOPS) determines LLM generation speed. In reality, during single-stream autoregressive decoding, Large Language Models are strictly memory-bandwidth bound.
For every single token generated, the GPU must transfer the entire quantized model weights across the memory bus from VRAM to SRAM. The theoretical maximum decoding speed is governed by:
Max Tokens/sec = Memory Bandwidth (GB/s) / Model Size (GB)
| Hardware Platform | Memory Bus & Type | Bandwidth (GB/s) | Llama 3.1 8B (Q4 ~5GB) | Llama 3.3 70B (Q4 ~42GB) |
|---|---|---|---|---|
| NVIDIA RTX 4070 (12GB) | 192-bit GDDR6X | 504 GB/s | ~75 tok/s | OOM (Exceeds VRAM) |
| NVIDIA RTX 4090 (24GB) | 384-bit GDDR6X | 1,008 GB/s | ~130 tok/s | OOM (Exceeds VRAM) |
| Dual RTX 3090 (48GB NVLink) | 2x 384-bit GDDR6X | 1,872 GB/s | ~140 tok/s | ~24 tok/s |
| Apple M4 Max (128GB Unified) | 512-bit LPDDR5X | 546 GB/s | ~65 tok/s | ~12 tok/s |
| NVIDIA H100 (80GB SXM5) | 5120-bit HBM3 | 3,350 GB/s | ~320 tok/s | ~72 tok/s |
Notice why dual RTX 3090/4090 cards or high-bandwidth H100 nodes dominate throughput: higher memory bandwidth directly translates into linear increases in token generation speed. For deployment tutorials, read our guide on Deploying Local DeepSeek with Ollama & Open-WebUI.
3. Frequently Asked Questions
What happens if a model slightly exceeds my GPU VRAM?
When model weights spill past physical VRAM, inference runtimes like Ollama and llama.cpp offload remaining layers to CPU system RAM over the PCIe bus. Because system RAM bandwidth (DDR5 ~60 GB/s) is 15x slower than GPU VRAM (GDDR6X ~1,000 GB/s), generation speeds collapse from 50+ tokens/sec down to 1–3 tokens/sec.
Is Apple Silicon Unified Memory better than dedicated NVIDIA GPUs?
Apple Silicon Mac Studio and MacBook Pro systems feature unified memory configurations up to 192GB or 256GB. While memory bandwidth (400–800 GB/s) is lower than enterprise HBM3 GPUs, unified memory allows running massive 70B and 671B MoE models locally on a single desktop without purchasing a multi-thousand dollar GPU rack.
How does context length impact VRAM during long conversations?
As conversation history extends from 8k to 32k or 64k tokens, the KV cache grows dynamically. At FP16 precision on a 70B model, a 64k context requires over 18GB of VRAM strictly for the KV cache. To avoid out-of-memory errors, enable FP8 or Q4 KV caching in your inference engine.