Oh. It depends on the backend you’re using, but that seems to be about the expected speed:
482 tokens in about 12 seconds works out to roughly 40 tokens/s. If those 12 seconds are the actual generation/decode time, that number does not look suspiciously high for Qwen3.8-27B on a 3090.
As a sanity check rather than an exact apples-to-apples benchmark, the current crowd-sourced llamabench.ai results for Qwen3.8-27B on an RTX 3090 include non-speculative runs at 38.2, 40.0, 41.1, and 45.6 tok/s. Those runs use llama.cpp and somewhat different quant/cache settings, so I would not treat them as a reproduction of your Ollama run, but your ~40.2 tok/s sits right in that range.
The cheapest way to verify what you are seeing is probably:
ollama run <your-model> --verbose
ollama ps
For the first command, the useful lines are approximately:
prompt eval count: ...
prompt eval duration: ...
prompt eval rate: ...
eval count: ...
eval duration: ...
eval rate: ... tokens/s
Ollama’s API usage documentation makes the distinction explicit:
prompt_eval_* is processing the input prompt.eval_count is the number of generated output tokens.eval_duration is the time spent generating those output tokens.total_duration also includes other work such as model loading and prompt processing.
So if Ollama itself reports an eval rate around 40 tok/s, there is not much ambiguity left: you really are getting about 40 generated tokens/s.
ollama ps gives the other high-value check. According to the Ollama FAQ, 100% GPU means the model is loaded entirely on the GPU, while a CPU/GPU percentage indicates that it is split between system RAM and VRAM.
A useful decision tree is therefore:
Is ~12 s the generation time / Ollama eval_duration?
|
+-- Yes
| |
| +-- ~482 / ~12 s = ~40 tok/s
| |
| +-- ollama ps says 100% GPU
| | |
| | +-- This looks quite normal for this model/GPU class.
| |
| +-- CPU/GPU split
| |
| +-- Then context size/offloading becomes important
| before comparing the number with other benchmarks.
|
+-- No / not sure
|
+-- Use --verbose (or the API metrics) once and compare eval_rate,
rather than dividing visible tokens by wall-clock time.
In other words, I would not worry that the 3090 has somehow produced an impossible result. Its age is a bit misleading here: 24 GB of VRAM is still a very useful configuration for local inference, because a quantized model of this size can fit without having to spill a large part of it into CPU memory.
Why comparing this to an older '27B model on a 3090' can be misleading
Qwen3.8-27B is also not just a conventional 27B full-attention Transformer with a new name.
The official Qwen3.8-27B model card describes:
- 27B parameters
- 64 language-model layers
- a layout of
16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))
- MTP (Multi-Token Prediction) training
- native 262K context
So it is a hybrid recurrent/attention architecture. That does not mean “DeltaNet explains exactly why you get 40 tok/s”, but it does mean an old benchmark for an unrelated 27B full-attention model is not a particularly strong prediction for this model.
Qwen’s own model card also warns that inference throughput can vary significantly across serving frameworks. That matters here because:
GPU + parameter count + nominal quant
is not enough to define a benchmark.
At minimum, these can change the result:
backend / backend version
exact GGUF
quantization
GPU offload
context length
KV/state-cache settings
Flash Attention
prompt length
prefill vs decode
speculative decoding / MTP
sampling/application overhead
So a result from llama.cpp, Ollama, vLLM, or another backend should usually be treated as a range/check rather than an exact prediction unless those conditions match.
There is an additional timing detail here: Ollama added Qwen3.8-27B support in v0.32.12, released on August 14, 2026, and v0.32.13 immediately included another Qwen3.8-specific change. This is a very new model/runtime combination, so older “3090 + ~27B” reports can easily be measuring a materially different software path.
About the GGUF you pulled from Hugging Face
There is a small artifact-identification wrinkle that is probably worth keeping separate from the performance question.
Hugging Face’s Ollama integration documentation supports tags of the form:
hf.co/<user>/<repository>:<quantization>
and also allows using a complete GGUF filename when an exact artifact matters.
At the moment, the Unsloth repository recommends:
ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
The current UD-Q4_K_M file is 16.5 GB, and the current mmproj-F16.gguf is 928 MB.
Those sizes match the numbers in your log remarkably closely:
model: ~16.5 GB
projector: ~928 MB
So the current UD-Q4_K_M + F16 projector is a very plausible match.
I would still call that a strong size match, not proof of the exact historical artifact, because the contents of a model repository and its quantization aliases can change over time. If exact reproducibility ever matters, using/recording the full GGUF filename or digest is safer than relying only on a short quantization label.
This distinction does not really change the answer to your speed question; it just makes later benchmark comparisons less ambiguous.
What about MTP / speculative decoding?
I would keep MTP separate from the explanation of your current ~40 tok/s result.
Qwen3.8-27B was trained with Multi-Token Prediction, but:
"the model has MTP capability"
and
"this particular inference run is using MTP/speculative decoding"
are not the same statement.
There is already enough evidence to explain ~40 tok/s without speculative decoding: the 3090/Qwen3.8-27B community results linked above include several non-speculative runs in almost exactly that range.
For comparison, Ollama exposes a separate current model variant, qwen3.8:27b-mtp-q4_K_M, whose parameters explicitly include:
draft_num_predict: 4
Likewise, llama.cpp documents speculative decoding as an explicit inference mechanism: draft tokens are proposed and then verified by the target model.
That can raise decode throughput substantially when draft acceptance and the rest of the configuration cooperate. The same 3090 benchmark page currently has MTP runs well above the non-speculative ~40 tok/s range.
But I would not use those faster MTP numbers as the baseline for your run unless you have confirmed that MTP is actually active.
A simple way to keep the comparison clean is:
normal decode <-> normal decode
MTP/speculative <-> MTP/speculative
rather than mixing both sets of numbers.
Why two people can report very different 'tokens/s' for the same GPU
There are really two separate performance phases in ordinary autoregressive inference:
1. Prompt processing / prefill
The existing prompt is processed, often many tokens at once.
Ollama reports this separately as:
prompt eval rate
2. Token generation / decode
The model repeatedly generates the next token.
Ollama reports this as:
eval rate
For an interactive chat, decode rate is usually what someone means when they say “the model generates at 40 tok/s”, but prompt processing has a large effect on time-to-first-token and therefore on how fast the application feels.
So these are quite different comparisons:
"I get 40 tok/s decode"
"The entire request, including a giant prompt, finishes at an average of 40 tok/s"
"I counted 482 visible words/tokens and divided by stopwatch time"
The first one is the cleanest model-generation benchmark.
Context length is another reason benchmark numbers move. If a configuration fits entirely in VRAM at one context size but needs CPU offload or substantially more cache/state memory at another, the performance regime can change.
That is why ollama ps is useful: it provides a direct observation instead of requiring you to guess from the nominal GGUF file size.
For a reproducible comparison, something like this is enough:
GPU: RTX 3090 24 GB
Ollama version: ...
model: exact tag / GGUF
context: ...
processor: 100% GPU (or split)
prompt eval rate: ...
eval rate: ...
MTP/speculative: on/off
You do not need all of that just to answer “is 40 tok/s plausible?”, but it makes later comparisons much more useful.
One more Qwen3.8-specific detail: thinking tokens
The Qwen3.8-27B model card says that the model thinks by default before producing its direct response.
That means “generated tokens” and “the tokens I visibly counted in the final answer” are not necessarily identical concepts, depending on the frontend/API and how thinking is displayed.
For benchmarking, I would therefore use Ollama’s own eval_count and eval_duration rather than manually counting the final visible answer.
Again, this is mostly a measurement/detail issue rather than evidence of a problem with your setup.
If the performance changes later
I would not troubleshoot anything further while it is producing clean output at ~40 tok/s. But if the behavior changes, the symptom gives you a reasonably cheap next branch:
| Symptom |
First thing I would check |
| Decode suddenly much slower |
ollama ps, context length, CPU/GPU split |
| First token is slow but generation is fast |
prompt evaluation / prompt size |
| Short chats are fast, long chats become slow |
context/cache/state size and GPU residency |
| Enabling MTP gives little/no gain |
whether MTP is actually active and draft acceptance |
| Speed is normal but output becomes malformed |
Ollama/backend version before assuming hardware failure |
| Two benchmarks disagree badly |
backend, exact GGUF, context, cache, MTP, and measurement boundary |
That keeps several different problems separate instead of treating every performance change as “the GPU got slower”.
So, for the result you posted, my default interpretation would simply be:
yes, ~40 tok/s is believable.
If ollama run ... --verbose reports roughly that eval rate and ollama ps says 100% GPU, I would consider the basic mystery solved. The fact that a 3090 is an older card does not prevent it from still being a very capable local-LLM card when a ~27B quantized model fits comfortably inside its 24 GB VRAM.
