I feel like these types of benchmarks mostly test what the mainstream tells you is possible and or works. You rarely get to hear the different implementations of inference engines, different types of setups (e.g. hybrid ones or older hardware) and their pros and cons. Also it seems these were tested with very minimal token usage.
Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.
If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.
Locally the MOE based models get a lot more out of the hardware than a dense model. Qwen 3.8-27B is the current golden child if you have >24GB VRAM but because its a dense model it is pretty slow to run and requires a lot of hardware. A lot of the guides are very much focussed on getting the dense models to run but in practice the MOE models with few active parameters are just faster and easier to run without too much loss of intelligence, they behave a lot more like their peak size than the active parameter size.
RAM is going to be the big bottleneck for MOE (at least for now) as it will determine the quantisations you could run.
On a 3080 10GB and 32GB of RAM I can run the qwen-next coder model and that is a 127B model and it gets 40-50 Token/s with 65k context. That is a lot more usable than a Quantised Qwen3.8-27B in my experience both because its faster and more capable.
I am looking forward to Qwen4 soon, I am hoping we get some more 27-35B MOE models, those are going to be really good for local hardware.
> in practice the MOE models with few active parameters are just faster and easier to run without too much loss of intelligence
Is this true for coding tasks? Genuinely asking because I don't yet have the hardware to be able to run/compare. I thought I remembered the sentiment that the 27B dense was better than the 35B MoE
The dense is better in its result, but is out of reach for most hardware compared to the MoE models, and the MoE models produce output tokens much much faster.
Thanks for sharing! I would love for some go to source for this type of insider info from the people who actually have tried. (The best I've found so far are some guys on Twitter/X like Alex Fateev (https://x.com/superalesha) and AJ (https://x.com/ItsmeAjayKV))
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.
Try to use your IGPU as well if you have enough RAM, almost everyone has dual GPUs in their computers.
Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.
Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.
The A3B part of that model means that each "expert" in the mixture of experts (MOE) model use 3B parameters. At 4-bit quant, you can fit 2 parameters per byte. That gives you 1.5GB per expert (plus a bit for overhead), which makes it usable on an 8GB card.
It's all about trade-offs: how much of the model you want to load into VRAM; how much VRAM you want to reserve for context; how much quantization you/the model can tolerate; how many tokens/second you can tolerate; whether you want to use the VRAM for anything else (graphics, Text-to-Speech, etc.); etc.
A3B refers to 3 billion parameters active during a forward pass, not the size of an individual expert.
The 3 billion includes all active experts, and then other shared weights, attention components, embeddings.
In the qwen 3.5 35b a3b series, each expert is ~3 million parameters. There are 40 layers which each have 1 shared expert and 8 routed experts. Roughly 1.1B of the 3B active are experts.
Yes, this is all good knowledge in theory for testing local ai. There are some more technical stuffs to consider that are happening in the background that is well outside my expertise to explain. I have been testing every MoE model I can fit on my system every year to find out which is the best for me.
Last year, it would have been impossible to run a 35B A3B model on old Pascual gpu and make it somehow usable. But for some reason now it just works even with a better finetuned version. So clearly there is more to it that just having it theoretically fit the system. Look for example how Strata works on certain models and systems compared to its llamacpp counterpart.
It sounds great and I hope local LLM/AI will be more powerful and available even on old hardware.
I tried llama.cpp with offloading "all what could be" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-pasting between web chat and code editor.
Is it even worth for trying something on my GPU? What do you think?
And yes, I use Claude Code (Opus 5.5) and Codex in my job but for personal stuff I would prefer trying free solutions (it is the reason I tried local LLM). I do not want to buy subs, at leats for now.
Models and inference engines are getting better and more optimized everyday so it is definitely worth experimenting with your GPU. You should probably check on newer finetuned and or REAP models Huggingface that could fit on your system.
I believe in your case the model quality matters a lot. To my (trust me bro) experience any MOE model, finetuned or not, with params < 19B even with Q4-quant always give lackluster results. So any Qwen2.5 wouldn't suffice to even simple coding tasks. For your case maybe you can bump it up to a Gemma4 E2B/E4B or Qwen3.5-9B. Also make sure that your current build of llamacpp actually uses your GPU with CUDA.
Yes, I was using CUDA toolkit on my Debian. Anyway, thanks for reply. I think I could try something but definitely it looks like it will not have better quality, even for writing simple code function by function :P
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
Ideally you want to find something that allows a context window for at least 65k for agent work. Over 100k is better but 32k is about the bare minimum for anything in a harness/agent like Pi. You might have to drop to a lower quantisation to make that possible but context is the key for longer lots of thinking questions like this.
If you drop the thinking level to low or even off the context will go further at the cost of the models intelligence. That might also help you get some answer but since such a question will read so many files into that context its the wrong approach. This is one of the bigger limitations of local LLMs, you can't do work that spans across a lot of files because you are almost always context limited. There is a limit of 1M even with the frontier so even that can't review an entire project. Focussing in on individual files or small groups is more achievable.
So just to be specific the -c needs to be >65536 and depending on the size of what is in the folder might very well not be enough.
Your setup is definitely a monster compared to mine so it should give much better results. I also believe you run dense models with decent results with that setup.
Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw):
would drop it from fp16 to 8 bits. You don't want to go below 8 bits if you can possibly avoid it and if you really do need to squeeze just drop the v value to q4_0 but leave the key alone. Unlike model quantisation this really makes things worse quickly.
By running vllm at work I learned that you will usually outperform benchmarks which you will find on the internet. Mostly by running proper parameter tuning. So you get impressive numbers. Still in reality those numbers do not tell the whole truth. We found that real usage is more 60k-140k prompt with a few k of output for realistic agentic coding workloads. Most of the time that will give you half to a third of decode throughput vs the benchmark. It also becomes much more about cache aware routing and kv cache management (how to offload and restore). Because prefilling 60-140k will simply take too long for interactive work.
Note that this assumes a single chat (single cached context). As soon as you want to use it for several tasks in parallel or in frequent alternation, hardware requirements increase or performance drops.
around this time, i was thinking about running a local LLM for the company i work for. i'm trying to figure out what kind of hardware i'd need and what level of performance i could expect. i tried a few things on your website but how much do the results reflect real world usage?
in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
1. It is difficult to figure out if dropping $10k+ on hardware will make a meaningful difference for running local models.
2. Downloading each model and each quantization of that model to see the speed seems rude to free services like hugging face, so now I can see things like "with my hardware, going to a Q8 only results in 3 more tokens per second"
My one complaint is picking my hardware (9070XT) and Qwen3.6 27B FP16 results in the message:
> This combination won't run. Qwen 3.6/3.8 27B at FP16 needs 54.6 GB — the AMD RX 9070 XT 16GB has 14.9 GB usable. No quantization of this model fits; pick a smaller model or more memory.
While it is true that it won't fit in your GPU memory, it demonstrably does run on my hardware. Llama.cpp supports putting some layers on the GPU and running others on the CPU.
Also, that message says "No quantization of this model fits" but switching to a different quantization of that model (for example, Q8_0) makes the message go away?
It is nice to have simulated confirmation of the M4 Max 128gb limitations on Gemma4 31B vs Gemma4 26B A4B at Q4:
Time to first token (TTFT) on 5,687 tokens, like pasting a short code block into the prompt, takes 36 seconds vs 4.7 seconds. It also breaks down the seconds for each step like KV and prefilling.
Gemma4 31B TTFT on the M5 Max 128gb with same setup is showing as 19 seconds.
A random number I checked feels kinda optimistic, or there's something wrong with my setup; Gemma 26B-A4B on a 16GB 5060Ti is reported to get 21-88tok/s with Q4_K_M, meanwhile I can just barely eke out 70tok/s with Q3 (which fully fits in vram), and 40tok/s with Q4 (with recently-merged MoE cache on llama-cpp). I guess the average fits, but the peak feels implausible.
No clue about details, but Sebastian Raschka's site reports its kv cache usage at 210kB per token, which is like 6x more than this site claims ("0.273 GB per 8k"). Also, the site claims qwen 27b to use 2x more memory per token than gemma, which is the opposite from what I know.
just go rent one for a few minutes and try it out ... it'll cost you well under $0.50 if you're just trying to get a feel for if it's right for you. There's a lot of things like context window that this doesn't capture.
Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.
My take as well. The dual-3090 and dual-spark configs have an outsized voice in this conversation. I'd add pipeline parallelism vs tensor parallelism and in the case of TP setting your lanes and speeds per card.
Would love to have multi-gpu support eventually but it's a balance between whats popular and what actually has good quality benchmarks available. Multi-gpu numbers can also vary between setups
yeah opencode just runs with it with lmstudio as llm engine.
It won't win benchmarks of course, but 20 tps generation is okay and letting it chug away on my local kubernetes cluster to troubleshoot is pretty nice. First start is pretty slow though, parsing is around 100-200 tps. Which is crap for long questions or harnesses like opencode which start around with thousands of tokens. Once cache is warm, pretty enjoyable.
I think it could have way more use cases where it's just 'good enough'. Email categorization, voice commands for the home. 1 year ago this was still a very miserable experience, but qwen 9B + better cache quantization and more tricks in the latest llm engines makes it viable.
Will it do difficult architectural refactors by just giving a one-shot prompt like opus allows, nope, not gonna happen.
Yeah simple, but menial refactors where IDEs don't have clear cut refactors yet. Or languages like CUE which has awesome tooling, but not so great refactors in its lsp yet.
It would be great if numbers weren't off, would include optimised versions (ie. speculative decoding), context size, include allowed parallel inference (single agents use parallel inference a lot) etc.
I don't think in general people care about long list of models, they just care about the best one.
Also dual gpu is common setup, there should be ie. 2x 4090.
Model list is a bit limited, and whole harness affects the speed, so the libs used in harness and their versions might be useful to show for each individual model setup.
I’ll look into adding harnesses. The benchmark sources usually don’t contain harness so the problem is limited data. What models would you like to see?
He's exagerating, I hope. The only bottleneck a harness attachs is pre-context fill and whatever tooling is installed. A proper harness reduces these to negligable startup costs.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
Exagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with.
This is a great idea but I'm far less concerned about output token speed and way more about the ability to parallelize prefill - for agentic coding that's pretty much way more important than output speed.
Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.
If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.
RAM is going to be the big bottleneck for MOE (at least for now) as it will determine the quantisations you could run.
On a 3080 10GB and 32GB of RAM I can run the qwen-next coder model and that is a 127B model and it gets 40-50 Token/s with 65k context. That is a lot more usable than a Quantised Qwen3.8-27B in my experience both because its faster and more capable.
I am looking forward to Qwen4 soon, I am hoping we get some more 27-35B MOE models, those are going to be really good for local hardware.
Is this true for coding tasks? Genuinely asking because I don't yet have the hardware to be able to run/compare. I thought I remembered the sentiment that the 27B dense was better than the 35B MoE
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.
Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.
Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.
I might write more about this setup soon.
It's all about trade-offs: how much of the model you want to load into VRAM; how much VRAM you want to reserve for context; how much quantization you/the model can tolerate; how many tokens/second you can tolerate; whether you want to use the VRAM for anything else (graphics, Text-to-Speech, etc.); etc.
The 3 billion includes all active experts, and then other shared weights, attention components, embeddings.
In the qwen 3.5 35b a3b series, each expert is ~3 million parameters. There are 40 layers which each have 1 shared expert and 8 routed experts. Roughly 1.1B of the 3B active are experts.
Last year, it would have been impossible to run a 35B A3B model on old Pascual gpu and make it somehow usable. But for some reason now it just works even with a better finetuned version. So clearly there is more to it that just having it theoretically fit the system. Look for example how Strata works on certain models and systems compared to its llamacpp counterpart.
I tried llama.cpp with offloading "all what could be" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-pasting between web chat and code editor.
Is it even worth for trying something on my GPU? What do you think?
And yes, I use Claude Code (Opus 5.5) and Codex in my job but for personal stuff I would prefer trying free solutions (it is the reason I tried local LLM). I do not want to buy subs, at leats for now.
I believe in your case the model quality matters a lot. To my (trust me bro) experience any MOE model, finetuned or not, with params < 19B even with Q4-quant always give lackluster results. So any Qwen2.5 wouldn't suffice to even simple coding tasks. For your case maybe you can bump it up to a Gemma4 E2B/E4B or Qwen3.5-9B. Also make sure that your current build of llamacpp actually uses your GPU with CUDA.
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
llama-server \ -m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \ -ngl 20 \ -c 16384 \ --jinja \ --alias tiel-coder
using it in opencode
{ "$schema": "https://opencode.ai/config.json", "provider": { "llamacpp": { "npm": "@ai-sdk/openai-compatible", "name": "Local llama.cpp", "options": { "baseURL": "http://127.0.0.1:8080/v1" }, "models": { "tiel-coder": { "name": "Tiel-Coder 35B A3B", "limit": { "context": 16384, "output": 4096 } } } } }, "model": "llamacpp/tiel-coder" }
If you drop the thinking level to low or even off the context will go further at the cost of the models intelligence. That might also help you get some answer but since such a question will read so many files into that context its the wrong approach. This is one of the bigger limitations of local LLMs, you can't do work that spans across a lot of files because you are almost always context limited. There is a limit of 1M even with the frontier so even that can't review an entire project. Focussing in on individual files or small groups is more achievable.
So just to be specific the -c needs to be >65536 and depending on the size of what is in the folder might very well not be enough.
Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw):
[*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2
[qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning = on n-predict = 32768 reasoning-format = deepseek reasoning-budget = 16384 reasoning-budget-message = "Proceed to final answer" reasoning-preserve = true swa-full = true np = 2 batch-size = 4096 ubatch-size = 2048 cache-reuse = 256 ngl = auto ctx-checkpoints = 33 ctx-size = 262144 no-mmproj = true no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 2
I am also certain that these flags could be improved somehow depending on your system.
I would go ahead and give it at least 64k but the more the better really!
I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.
would drop it from fp16 to 8 bits. You don't want to go below 8 bits if you can possibly avoid it and if you really do need to squeeze just drop the v value to q4_0 but leave the key alone. Unlike model quantisation this really makes things worse quickly.
in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
good job.
> This combination won't run. Qwen 3.6/3.8 27B at FP16 needs 54.6 GB — the AMD RX 9070 XT 16GB has 14.9 GB usable. No quantization of this model fits; pick a smaller model or more memory.
While it is true that it won't fit in your GPU memory, it demonstrably does run on my hardware. Llama.cpp supports putting some layers on the GPU and running others on the CPU.
Also, that message says "No quantization of this model fits" but switching to a different quantization of that model (for example, Q8_0) makes the message go away?
Time to first token (TTFT) on 5,687 tokens, like pasting a short code block into the prompt, takes 36 seconds vs 4.7 seconds. It also breaks down the seconds for each step like KV and prefilling.
Gemma4 31B TTFT on the M5 Max 128gb with same setup is showing as 19 seconds.
No clue about details, but Sebastian Raschka's site reports its kv cache usage at 210kB per token, which is like 6x more than this site claims ("0.273 GB per 8k"). Also, the site claims qwen 27b to use 2x more memory per token than gemma, which is the opposite from what I know.
Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.
- multi GPU configuration (hetero and homo)
- PCIe gen chooser (3.0/4.0/5.0)
- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)
- is PCIe bifurcated or CPU native (threadripper pro for example)
It won't win benchmarks of course, but 20 tps generation is okay and letting it chug away on my local kubernetes cluster to troubleshoot is pretty nice. First start is pretty slow though, parsing is around 100-200 tps. Which is crap for long questions or harnesses like opencode which start around with thousands of tokens. Once cache is warm, pretty enjoyable.
I think it could have way more use cases where it's just 'good enough'. Email categorization, voice commands for the home. 1 year ago this was still a very miserable experience, but qwen 9B + better cache quantization and more tricks in the latest llm engines makes it viable.
Will it do difficult architectural refactors by just giving a one-shot prompt like opus allows, nope, not gonna happen.
Yeah simple, but menial refactors where IDEs don't have clear cut refactors yet. Or languages like CUE which has awesome tooling, but not so great refactors in its lsp yet.
Asus ProArt P14, Surface Ultra, Yoga Pro 9n, etc.
I am hoping the cheaper N1 models will also come out soon, but might be several months.
Also would be good to see a few more quant and model options.
I don't think in general people care about long list of models, they just care about the best one.
Also dual gpu is common setup, there should be ie. 2x 4090.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
https://github.com/ggml-org/llama.cpp/issues/27330
I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.
It would be neat to also include NVFP4 variants for the Blackwell hardware.