llmrun.dev estimates whether an open-weight model fits a given GPU, Mac or mini PC, and how fast it decodes at each quantization. This post describes the decode-speed model for Mixture-of-Experts checkpoints, since that is where size-based estimates fail most.
Why size-based estimates fail for MoE
Single-stream decode is memory-bandwidth-bound: every generated token reads the weights once. For dense models the estimate is tok/s ≈ bandwidth / model size × efficiency, with a per-backend efficiency of 0.65 (CUDA), 0.60 (ROCm), 0.50 (oneAPI) and 0.70 (Metal). For MoE this is far too pessimistic. gpt-oss-20b (20.9B total, 3.6B active, 12.9 GB in memory) comes out at ≈51 tok/s on an RTX 4090, while llama-bench tg128 reports ≈222.
Model
- Bytes read per token are scaled by active / total parameters (applied when active < 0.9 × total; otherwise the dense formula is used unchanged).
- An additive per-layer cost covers routing and expert dispatch: 0.055 ms per layer on CUDA, 0.2 ms on Metal, ROCm and oneAPI.
seconds/token = active_GB / (bandwidth × efficiency) + n_layers × t_layer
Calibration
The constants were fitted to llama-bench tg128 results for gpt-oss-20b and gpt-oss-120b on 8 configurations (RTX 4090, RTX 5060 Ti, RTX 5080, RTX PRO 6000, RX 7900 XT, M1 Max, M4 Max, M2 Ultra). Each result is kept as a regression fixture with an 18% tolerance. Two examples:
| Hardware / model | Estimate | llama-bench |
|---|---|---|
| RTX 4090 / gpt-oss-20b | 212 tok/s | 222 tok/s |
| M2 Ultra / gpt-oss-120b | 81.5 tok/s | 80 tok/s |
On the M2 Ultra the per-layer term accounts for more than half of the time per token, which is why scaling bandwidth alone overshoots on Apple Silicon.
Prefill is estimated separately as compute-bound (dense FP16 TFLOPS × MFU / (2 × active params)). Fit includes a KV cache sized per architecture (sliding-window and hybrid layers are handled) with a 16K-token reserve.
Known limitations
- CPU/RAM offload is not modelled.
- Decode ignores KV-cache reads, so long-context speeds are optimistic.
- Calibration covers only the gpt-oss family; MoE shapes with many fine-grained experts or large shared experts are not validated.
- Stated error is about ±20% for decode and ±30% for prefill.
Methodology and formulas: How It Works — VRAM, Fit & Performance | llmrun
The site is free and needs no sign-up; some pages carry affiliate links to cloud GPU rentals.
Feedback is welcome, in particular llama-bench tg128 numbers for other MoE architectures (Qwen3-MoE, DeepSeek-style shared experts) that would test the per-layer term.