Understand LLM performance
before you run the model.
AMD LLM Lab combines real MI300X measurements with predictive modeling to explore VRAM, throughput, precision and hardware feasibility across enterprise open-weight models.
Direct empirical observations on Instinct MI300X
Qwen2.5 open weights: 0.5B, 1.5B, 3B, 7B, 14B, 32B
BF16, FP16, INT8, INT4, INT2 sweeps
5.3 TB/s memory bandwidth on gfx942
Performance Landscape
Explore measured throughput (tok/s) and peak VRAM (GB) across parameter sizes and precisions.
Hover any observation to view model identity, precision, context length, and exact measured output.
Model Scaling Behavior
How memory requirements scale linearly while throughput exhibits non-linear execution characteristics.
Linear Memory vs Convex Latency
On AMD Instinct MI300X, peak allocation scales predictably at 1.309 GB per billion parameters plus a 7.22 GB driver & ROCm runtime baseline.
VRAM_GB = 1.3087 × Params_B + 7.2168
Held-out model MAE: ±6.75 GB (R² = 0.536)
Precision Analysis
Comparing BF16, FP16, INT8, INT4, and INT2 behavior across measured MI300X runs.
Lower bit-width precision does not unconditionally yield higher throughput on current software stacks. On 7B models, INT4 under Quanto shows a 29.9% memory reduction with a 70.0% generation speed penalty compared to native BF16, while on 32B models INT4 delivers a 56.4% memory cut.
What can your machine run?
Compare your local system specs against model requirements categorized into four deployment tiers.
Windows / Linux
Host system
Host CPU
? threads
Available
System memory
None / Integrated
Host device
Manual Entry
Planning budget
FASTAPI PROBE
Local host telemetry
| Model | Precision / Stack | Estimated Peak VRAM | Status Tier | Technical Assessment |
|---|---|---|---|---|
| Qwen2.5-0.5B (0.5B) | BF16 / none (Transformers) | 5.1 GB | Not practical | Requires 5.1 GB; exceeds available system memory |
| Qwen2.5-1.5B (1.5B) | BF16 / none (Transformers) | 8.4 GB | Not practical | Requires 8.4 GB; exceeds available system memory |
| Qwen2.5-3B (3.086B) | BF16 / none (Transformers) | 10.1 GB | Not practical | Requires 10.1 GB; exceeds available system memory |
| Qwen2.5-7B (7.615B) | INT4 / qint4 (Optimum Quanto) | 12.3 GB | Not practical | Requires 12.3 GB; exceeds available system memory |
| Qwen2.5-7B (7.615B) | BF16 / none (Transformers) | 18.2 GB | Not practical | Requires 18.2 GB; exceeds available system memory |
| Qwen2.5-14B (14.77B) | INT4 / qint4 (Optimum Quanto) | 22.8 GB | Not practical | Requires 22.8 GB; exceeds available system memory |
| Qwen2.5-14B (14.77B) | BF16 / none (Transformers) | 37.3 GB | Not practical | Requires 37.3 GB; exceeds available system memory |
| Qwen2.5-32B (32.76B) | INT4 / qint4 (Optimum Quanto) | 45.2 GB | Not practical | Requires 45.2 GB; exceeds available system memory |
| Qwen2.5-32B (32.76B) | BF16 / none (Transformers) | 103.8 GB | Not practical | Requires 103.8 GB; exceeds available system memory |
Research Insights
Derived directly from the 57 measured MI300X benchmark runs.
Model Size and Memory Scaling
Peak VRAM consumption is primarily governed by parameter count with a constant baseline driver allocation of ~7.2 GB. Across all model sizes (0.5B to 32B), the scaling slope is 1.309 GB per billion parameters in BF16. Context length (512 to 8192) causes less than 8% total memory variation.
Precision & Quantization Dynamics
Quantization yields non-linear benefits. On smaller models (1.5B), INT4 increases measured VRAM by +10.8% and decreases throughput by 62.3% due to quantization container overheads. However, on larger models (14B and 32B), INT4 drops memory by 38.9% and 56.4% respectively, making large models fit onto smaller hardware.
Throughput & Configuration Trade-offs
Inference throughput depends heavily on model size and runtime kernel support. The 0.5B model achieves 78 tok/s, whereas 32B BF16 runs at ~12 tok/s on single-stream inference. Memory efficiency (tokens per GB) peaks at 15.1 tok/GB on smaller models and drops to 0.12 tok/GB on 32B.