Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.
Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.
I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).
I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".
Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
> Those two combined means that unless LLMs reach the state of 'good enough'
Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.
Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.
I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.
> I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens.
The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.
iPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people.
However, the compute situation just skew it even further when it comes to LLMs.
Phones are communication devices. So I would always expect connectivity to be central to their use. But that's a different issue than the "inefficiency" of edge computation and whether powerful edge devices can sell (they obviously can).
A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text
> Most people are already used to rely on the internet on basically everything
FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.
And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".
I don't understand how anyone with any degree of technological or scientific literacy or knowledge of advances in the past 100-200 years can say something won't happen in the future.
I am absolutely certain it will - the only uncertainty is how long, but given the rate of progress, I'm guessing it won't be far away.
Unfortunately you are wrong. Time marches on and progress in silicon does too. You may not like the timeline, but it will happen.
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).
2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.
3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.
4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).
5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).
5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
it’s equivalent take to laugh at first transformers 9 years ago, because they were shit and hardware requirements were immense.
IMHO it’s a matter of time until we (consumers) will get the hardware (maybe coupled maybe even more novel techniques). Though I expect it will take another 10 years or more.
"It will never be possible to have a computer in your home. You will always have to buy computer time and access via dumb terminals. Technology will never again advance."
Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...
There are certain physical limits. Calculations need to be done. Either less calculations are necessary for the intelligence, or u accept less intelligence. But there is a limit in what u can do with specific hardware.
By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited.
No, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
That's status right now, and that's because we just got models than can speed up the mitigation or find new CVEs. Plenty of reports of how many of them get patched now, most recently with Chrome.
8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.
1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today.
Theoretically around 104.8 TFLOPS.
In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we currently have. As if we were trying to make a current AAA game to run smoothly on the 1080 GTX of your example.
Here you have shown yourself that progress slows down and doesnt speed up. 8.9/0.35 = ~25x more performance in 10 years from 2006 to 2016. 104.8/8.9 = ~12x more performance in 10 years from 2016 to 2026. Growth has dropped 50%.
That isn't deceleration, you've just chosen a very selective way to compare. If you use time as a denominator, which is kind of intrinsic when talking about rates of acceleration, you get a very different result. If you graphed .3, 8.9, and 104.4 on the y axis, with years on the x axis, it would be pretty clear that there was in increase in the rate of progress.
We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add "only" 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.
It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
(2) Are you saying that you think we're at the limits of computing in general, or that specific technology?
We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily.
Leaving aside the discussion on LLMs intelligence vs human intelligence, on a purely energy consumption level we are definitely and without any possible questioning nowhere near that indeed.
We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.
This is definitely being done with private models by HFT/quant firms, data processing agencies/orgs (large intelligence agencies, _every_ data analytics org, etc).
Yes, right now it would be obsolete in six months, but I also must add that this never stopped crypto miners from making new ASICs.
However, with how useful Kimi is right now - at some point if someone makes a dedicated hardware board with "good enough" model for daily tasks - that would be a very sought after commodity.
They're important everywhere of course, but especially on mobile. If AI researchers figure out how to offload knowledge and expertise from reasoning weights, then a core reasoning ASIC linked to the knowledge would totally rock.
Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
Especially if you’re Apple and your expertise and reputation is in making privacy conscious devices and cloud services that are exceptionally user friendly.
Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.”
Skips the ugly horse race and massive risk while focusing on what they do better than anyone.
And they’re the only ones who have cultivated the trust with normal people to be able to say, “We now have the only AI service that can safely reference your personal info without giving it all to strangers.”
They could possibly end up the second biggest winners in this insane hype cycle… without having had to compete on making a sota model.
Apple has a real knack for not being first mover but being “best” mover. They didn’t invent tablets or mp3 players or thin laptops or ARM laptops, but once they entered those markets they absolutely became the definitional product for the entire category.
A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.
> Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]
Unified memory has existed for decades in the PC space, Apple didnt invent it.
And the test you linked to has nothing to do with unified memory, its a web browser benchmark (almost entirely constrained by single threaded CPU performance that Apple better than competitors at).
The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.
Agreed. We've just seen DeepSeek post-train their ~300 billion parameter flash model to outperform their 1.6 trillion parameter pro model, in the space of a few months. There would seem to still be quite a few opportunities on the table to bring big model smarts down to the smaller models
What might the “parameters/layers to gates” ratio look like? My naive and uninformed guess would be that 1B+ parameter would also need a 1B+ gate FPGA, but according to google they typically range from tens of thousands to several million (which would still be a fraction of a billion).
Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README.
I am glad it inspired more people to explore area of on-device AI further!
This looks useful, you can increase the RAM cache so if you have a Mac with 24-32GB it should speed up a lot and still run models that wouldn’t normally fit. I’m going to run some tests…
The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?
There are many search APIs available, I like Kagi's.
Microsoft and Amazon both provide web snapshot services that purport to give you a kind of agent-first internet archive. You can approximate something like that using common crawl, but it's a huge amount of data. Downloading the internet is impossible or a bad idea for almost everyone.
Achieving high-efficiency model compression to run large models locally on consumer hardware like Macs and iPhones mirrors the architectural goals of decentralized identity.
I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
> I wonder, is there a way to make the RAM usage tunable?
In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.
That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.
I do wonder if there are some experts that are more likely to be hit. So if the normal optimised setup runs in 12GiB an you have 4GiB extra to spare, you could say “promote the most used X experts to this stable (old gen in GC parlance) region and don’t swap it out.
Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.
It's tunable, --cache-gb N on the CLI. In my sweep the speed barely moved between a 1GB and 6GB cache (43% vs 70% hit rate, same tok/s) because right now the bottleneck is GPU dispatch, not the SSD. so more RAM doesnt buy much yet. once the kernel work lands it should start to matter, so on 32GB I would just set 8 and let it age well. Also the hit rates themselves answer the "can you even know which experts stay hot" question, reuse across tokens is very real.
this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.
half an hour to process 10k tokens on an M5 seems... not great
On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM.
The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:
1. the organization of the data being sent to the GPU to optimize throughput;
2. the speed at which the GPU can read that data from its VRAM;
3. how many triangles it can process in parallel by using individual compute units.
I suspect that given parallel improvements for neural networks, we'll see similar improvements:
1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;
2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;
3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.
If you've got X GB of weights in slow access memory (be it RAM vs VRAM or SSD vs RAM) and Y GB of fast memory then no matter what, if you want to use them you'll need to transfer X-Y GB and will be bound by memory throughput. You can try to reduce number of activated weights, but how much can be gained that way is speculative so far.
Domestic electricity is free nowadays, certainly for most of the year, as solar plus battery covers your usage for a tiny percentage of the cost of your house.
Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue even with very heavy read workloads.
Hm, I asked 5.6 Sol to calculate/research and it came up with way less optimistic lifespan than Gemini (that I've asked last year).
It claims that each individual page read induces read disturb across whole block.
And references https://arxiv.org/pdf/2501.02517 that tested Samsung 3D TLC and found ~518K sequential page reads in a block to be ECC threshold (although it's unclear how they got 518K number -- e.g. is it single worst chip they've tried? authors brings up 160 chip sample size later on).
With 7704 pages in a block that's only ~70 sequential block reads till data is lost and to retain data controller would have to refresh block fair bit earlier.. basically it gives modern 3D TLC SSD lifespan measured in months (1TB drive 24/7 sequential reads at 5GB/s).
Am I the only one that has no flash lifetime anxiety? I still have drives from more than a decade ago that keep on chugging fine.
I remember the time spinning rust was the only option and reliable they weren't. In 30 years of computing I have had more than ten hdds and zero ssds die.
> Swiftlet was built in collaboration with Claude Code.
Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude), or sort of reuse some of its parts?
Is it like someone saying "built in collaboration with VS Code" or ".. in collaboration with <xyz> autocomplete plugin"?
Or merely a disclaimer about vibe-coding or AI written tool?
no Anthropic involvement, I just used Claude Code heavily while building this and putting that in the README felt more honest than not mentioning it. Now that I think about it may be it shuld be "built with claude code" instead of "built in collaboration...". Changed it.
When you have Claude Code indepedently author commits and PRs and merge them in, it'll always credit itself as an author. I assume it showing up in the README is a byproduct of the same logic.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.
I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).
I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".
That's how the AI boom came around to start with, before CUDA it was an academic topic.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.
Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.
The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.
However, the compute situation just skew it even further when it comes to LLMs.
FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.
And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".
This is progress in the same way that a man climbing a tree is making progress toward reaching the moon.
This project is essentially the MoE-of-the day, with some platform-related optimizations.
> Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
That won't happen. Projects like this just give the illusion that that will be possible.
I don't understand how anyone with any degree of technological or scientific literacy or knowledge of advances in the past 100-200 years can say something won't happen in the future.
I am absolutely certain it will - the only uncertainty is how long, but given the rate of progress, I'm guessing it won't be far away.
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).
2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.
3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.
4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).
5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).
5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
That is, an essential step?
IMHO it’s a matter of time until we (consumers) will get the hardware (maybe coupled maybe even more novel techniques). Though I expect it will take another 10 years or more.
Do reads wear out nvme? Iirc only writes do
That time won’t be as much fun as you think
We'll be fine.
Oh well, maybe the last one to leave the internet can turn the lights off.
1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS.
In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
The suggestion is that a 1T model could be made to run on cheap consumer hardware of the future.
We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add "only" 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.
It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
Aren't we already approaching theoretical physical limits? We're at 2nm
(2) Are you saying that you think we're at the limits of computing in general, or that specific technology?
We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily.
But that is...not a low hanging fruit.
Nowhere?
GTX 1080 in 2016: 8 GB of GDDR5X, with 320 GB/s.
RTX 5090 in 2026: 32 GB of GDDR7, with 1.792 TB/s.
This is fun, what's next?!
PCI 8.0 is breaking 1TB/s, GDDR7 is 1TB/s.
With just the _current_ timeline, things are looking like they'll compress once we get over this initial lump.
This is definitely being done with private models by HFT/quant firms, data processing agencies/orgs (large intelligence agencies, _every_ data analytics org, etc).
At the rate models are improving, it would be obsolete in six months.
Not too wild an idea!
Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.”
Skips the ugly horse race and massive risk while focusing on what they do better than anyone.
And they’re the only ones who have cultivated the trust with normal people to be able to say, “We now have the only AI service that can safely reference your personal info without giving it all to strangers.”
They could possibly end up the second biggest winners in this insane hype cycle… without having had to compete on making a sota model.
We don’t talk about AVP.
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.
[1] https://youtu.be/x26A28DoT-w?t=605
Unified memory has existed for decades in the PC space, Apple didnt invent it.
And the test you linked to has nothing to do with unified memory, its a web browser benchmark (almost entirely constrained by single threaded CPU performance that Apple better than competitors at).
That’s a six core processor. It’s an A18 Pro in the Neo, same chip as on the Iphone 16 Pro
Nvidia leads in bandwidth and specialized architecture, but local inference takes off when it's usably fast at much lower cost and power consumption.
Who needs memory when your model is set in silicon ?
Microsoft and Amazon both provide web snapshot services that purport to give you a kind of agent-first internet archive. You can approximate something like that using common crawl, but it's a huge amount of data. Downloading the internet is impossible or a bad idea for almost everyone.
In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.
Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.
I feel like I've seen a similar statement on a lot of these streaming weight projects.
400b model on an iPhone: https://x.com/anemll/status/2035901335984611412
Hello Claude!
half an hour to process 10k tokens on an M5 seems... not great
People like this person are laying the foundations.
The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:
1. the organization of the data being sent to the GPU to optimize throughput;
2. the speed at which the GPU can read that data from its VRAM;
3. how many triangles it can process in parallel by using individual compute units.
I suspect that given parallel improvements for neural networks, we'll see similar improvements:
1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;
2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;
3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.
Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
It claims that each individual page read induces read disturb across whole block. And references https://arxiv.org/pdf/2501.02517 that tested Samsung 3D TLC and found ~518K sequential page reads in a block to be ECC threshold (although it's unclear how they got 518K number -- e.g. is it single worst chip they've tried? authors brings up 160 chip sample size later on).
With 7704 pages in a block that's only ~70 sequential block reads till data is lost and to retain data controller would have to refresh block fair bit earlier.. basically it gives modern 3D TLC SSD lifespan measured in months (1TB drive 24/7 sequential reads at 5GB/s).
I remember the time spinning rust was the only option and reliable they weren't. In 30 years of computing I have had more than ten hdds and zero ssds die.
> Swiftlet was built in collaboration with Claude Code.
Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude), or sort of reuse some of its parts?
Is it like someone saying "built in collaboration with VS Code" or ".. in collaboration with <xyz> autocomplete plugin"?
Or merely a disclaimer about vibe-coding or AI written tool?