> We'll assume a 32B dense model, as they've have gotten quite good for production use and a B200 can comfortably serve them. This could be a Gemma, Qwen, DeepSeek, whatever.
That seems like a very consequential point to include halfway through the post. They aren't wrong that Qwen 3.6 26B or Gemma 4 31B are quite good, depending on the use case, but if we're doing napkin math, I'd want some more headroom in the assumptions.
They really ought to have Qwen parameterize their post's calculations and add sliders so a reader could play around with the values.
Edit: And since they especially mentioned DeekSeek (or whatever), as far as I know, none of their current generation of models is a dense model, and even the smallest of the mixture of experts (MoE) models is 284B parameters (13B activated). That will completely incinerate their napkin.
But in reality, 32B dense is very similar* to 32B activated on MoE in terms of inference costs. And I highly suspect eg Opus is around that level of active params.
A 284ba13b model at scale, is almost certainly cheaper to serve than a 32b dense model.
*as you can shard the model across multiple GPUs at scale. but in reality you have some loss of efficiency from GPU coordination and expert routing
That's good information. I couldn't possibly even start to run even DeepSeek Flash on my system, but also if you're assuming multiple GPUs, that is going to affect the napkin math.
>This largely depends on whether you own or rent your hardware. At $40,000 per B200, your lifetime cost per user is 40_000/num_users.
In the 100% duty cycle case (worst for cost), that's 6k$ per user. Realistically, serving 300 users per GPU you'll spend a lifetime cost of about $133 per user, plus the datacenter/upkeep bill.
If you rent the GPU, the cost is more straightforward. At an hourly rate of $43, your hourly cost per user is 4/num_users. For num_users=300 you get an hourly rate of about $0.013 per user, or $9.36 per month.
This leads me to believe you can buy a GPU but leave it at a data center?
Do people do this? I don't understand. Or are you equating upkeep bill to electricity on premises?
The biggest advantage is you can have economies of scale for 24/7/365 physical support of thousands of racks without you yourself having the scale to require thousands of racks.
Similarly you don’t need to pay premium office space prices when data centers can be placed in the middle of nowhere. You don’t need to have backup generators and redundant network connections or worry about improving your buildings AC to handle the heat load as you scale. Or worse setting all that up and then realize you now need 10 racks of equipment and nothing is scaled for that.
It makes a lot of sense for major hospitals to have small on site data centers, but most businesses operate very differently.
That seems like a very consequential point to include halfway through the post. They aren't wrong that Qwen 3.6 26B or Gemma 4 31B are quite good, depending on the use case, but if we're doing napkin math, I'd want some more headroom in the assumptions.
They really ought to have Qwen parameterize their post's calculations and add sliders so a reader could play around with the values.
Edit: And since they especially mentioned DeekSeek (or whatever), as far as I know, none of their current generation of models is a dense model, and even the smallest of the mixture of experts (MoE) models is 284B parameters (13B activated). That will completely incinerate their napkin.
But in reality, 32B dense is very similar* to 32B activated on MoE in terms of inference costs. And I highly suspect eg Opus is around that level of active params.
A 284ba13b model at scale, is almost certainly cheaper to serve than a 32b dense model.
*as you can shard the model across multiple GPUs at scale. but in reality you have some loss of efficiency from GPU coordination and expert routing
This leads me to believe you can buy a GPU but leave it at a data center?
Do people do this? I don't understand. Or are you equating upkeep bill to electricity on premises?
Network throughout?
Similarly you don’t need to pay premium office space prices when data centers can be placed in the middle of nowhere. You don’t need to have backup generators and redundant network connections or worry about improving your buildings AC to handle the heat load as you scale. Or worse setting all that up and then realize you now need 10 racks of equipment and nothing is scaled for that.
It makes a lot of sense for major hospitals to have small on site data centers, but most businesses operate very differently.
What is the operational cost and when does it become more expensive than the upfront capex?
The B200 tops out at 1000W and idles around 140W. It averages around 600W. https://www.lightly.ai/blog/nvidia-b200-vs-h100 U.S. average electricity cost is $.14 per kWh in March. https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...
600/1000 *.14 =$0.084 per hour. $2.01 per day. $60.30 per month. With 300 users, $.20 per user per month. Seems fairly cheap for the electricity.
Does anyone know how to estimate colo/data center rent costs? Where did I screw up my estimates?
what kind of math is this? why isn't it B = 562 / 2 = 281?