"Combining them into one heroic speedup would make a better headline and a worse benchmark."
"The machine immediately taught me that capacity estimates are just admission tickets."
"Useful in production, poison in a kernel comparison."
Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.
The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.
It has gotten so much worse over the last month. The default writing style of the Claude 5 model series in Claude Code is some sort of jiberish jargon.
Apparently we've blown way past the Turing test and approaching AGI and yet LLM-generated text still sticks out like a sore thumb. Maybe LLMs aren't that good at writing after all.
I think it's fair to say they're better at a paragraph or so than most humans. And have been for quite some time, which is probably why their use in writing has exploded.
Long form though? Still pretty bad. Probably getting worse in practice, as people have them write larger and larger chunks of text without paying any more attention to the result.
Can you please try and see how many tokens you get with some form of concurrency. Pretty much ALL the benchmarks I've seen on the more accessible cards are just single request.
Because concurrency with a single "accessible" card quickly diminishes. I have dual 3090s, and on Qwen 3.6 35B A3B at 80k a single card with max concurrent set to 4 will get 70-80tps single request, 50-60 TOTAL tps with 2, 45-50 with 3, and around 40tps with all 4 going.
I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.
I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other.
As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.
I haven’t tried any larger models, but a 12B model on my Ampere A5000 gets around 4-5x the aggregate throughput with concurrency. I have the maximum context configured to 32k, but my actual requests are usually around 2-4k tokens.
No idea how that compares to running a larger model and context though.
Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?
I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.
I've had some rough time getting LM-Studio properly configured for multi-card. It exists, but I feel like it is kind of buggy. I will disable a card and it will still load the model into it. Sometimes it will split the model even though there is loads of room available.
I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.
vLLM is probably the key difference there… its scheduler is built around batching/concurrency, while this setup is heavily optimized llama.cpp for single-stream latency
Its not quite that simple here. The iMatrix-guided hybrid uses different quantization levels per tensor/layer, so there isnt one honest Q4/Q5/NVFP4 label I can put in the title
The whole site looks like and reads like AI slop. The outcomes also don't make any sense and don't feel rigorously tested (no, having claude test for you doesn't count as rigorous).
Please stop making this comment. The war is lost. Instead, you should be commenting that it looks like a human wrote this when you come across the rare brain-produced writing
Its egregious the quant level isnt disclosed along the "Qwen" string. Everyone knows theres huge difference in speed/quality along the quant axis, I now attribute the ommision of such to deliberate choice to not curb the hype of the tittle.
5.01 BPW custom hybrid: bulk NVFP4, selected Q5_K/Q6_K tensors from an iMatrix, Q6_K embeddings and Q8_0 lm_head. The iMatrix was built from 5,472 messages across 296 real Hermes sessions
they talk about the quants they tried in the article and settle on a Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf which they calibrated on their own session traces and they pulled in 5 different llama.cpp pull requests to their local llama-server.
yeah, I was confused during the whole thing, i get 70 t/s on a 3090, which evens out around 50 t/s at 128k+ , have been running 3.6 and now 3.8 (both iq4_nl at 256k q4 kv) on the 3090 for months. I am confused as to what we 'discovered' here, it's a common config. and at less than 1/2 the price of the gpu (and double the bandwidth, though no fp4 cores to be fair).
A 3090 has 936 GB/s of bandwidth vs 432 GB/s on the 70W RTX PRO 4000 SFF, so 70 t/s there is not surprising. The interesting constraint here was fitting a workload-tuned 5.01 BPW quant + 256K + MTP into 24 GB while working with less than half the memory bandwidth
Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.
If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).
"The machine immediately taught me that capacity estimates are just admission tickets."
"Useful in production, poison in a kernel comparison."
Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.
The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.
Claude seems to speak and write more in Claude-speak with every release.
Long form though? Still pretty bad. Probably getting worse in practice, as people have them write larger and larger chunks of text without paying any more attention to the result.
llm gets them some of that sweet_llm_padding and readers smell the slop and attack the writing :)
people, go write "boring stuff" - its more interesting!
I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.
As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.
With dense Qwen 27B on 3090/llama.cpp I get:
No idea how that compares to running a larger model and context though.
I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.
I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.
In my testing I got 150 tokens/sec with a single 5090 RTX.