I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription.
There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than sufficient.
A company called Taalas is working on something like that. Not Opus4.6 quality, but I'm sure they're targeting larger models. Currently they're using a LLama 8B model. It runs at ~17k tokens per second, and you can test it at https://chatjimmy.ai/.
I’m not sure if this is what you meant, but at 17k t/s, you start to compete with the speed of network calls. You could approach the point of generating an HTML/js/css page faster than some websites can be returned over the network.
"Design me a 3d printable rocket engine for a hobby rocket project. Verify it's design in a full simulation. Iterate until it works reliably in simulation based on a verified printable design on a consumer laser sintering device (or substitute contract manufacture for under 1000 dollars)."
This is a hobby version of a project, but you can imagine commercial versions of the same prompt for new databases, genomics studies, material analysis, operating systems etc.
I can't put a 10 page narrative on how my specific motor should work into a hacker news post ;) you can also imagine the above where the goal is to have the ai exceed the performance of stock motors.
You almost certainly do not want an LLM to do that. Leap71 actually has computational models generating functional rocket engines that way. You could absolutely wrap a tool like that in a shell and handle control with an LLM and not need anywhere near the tokens.
Thats the thing - these models see and predict tokens. For any real engineering you get more bang for your buck using math.
Verifying how the model works against the real world is the difficult, dangerous bit.
There might be some interesting side effects from making simulation software, which is currently either an expensive niche or quirky university project (SPICE always has that feel).
I’m not convinced at all that the model won’t just get stuck in a loop where it doesn’t understand how to fix the broken rocket. I see similar failure modes in far simpler projects strictly confined to coding. This feels closer to “make me a profitable business, make no mistakes” than to a simple coding project.
Right now - there's some heavily subsidized subscriptions that are more or less cheating. For instance, Github CoPilot at $39/month gives you claude opus 4.6. They're going to close that off, but right now it's like a freebie for those doing API agentic harnesses.
That said, if you are doing always on agents and you spend $3k-$4k on a GB10 or, $5+ k on Apple Silicon as your sunk cost, you will probably come out ahead.
I've got 5 agents running a purely experimental social experiment. AThey operate in an evennia mud (a familiar sounding city called "gothmud). I've built a channel, idle prompts, sleep schedule. I feed in real world news, weather. There's a character up in a clock tower that reads evennia's audit logs every 20 minutes to surveil the city, and a cast of people wandering around, investigating things, having coffee, repairing robots. This is all hitting qwen3.6-35-A3B on the Asus GB10, which cost me $3k.
Over the last 30 days, I've hit 394M input tokens, 1.6B output tokens. I would have spent between $1600 to $1700 if I was using openrouter. Not calculated - I also have comfyui running in the spare space, and the agents "take photos" of the rooms they're in, selfies, workshop photos, etc.
How much did I spend on electricity? I don't have a meter on my box. My total electric bill for the last 30 days was $220, so I know it's less than that. My rate to compare is 11.7/kwh, but it's closer to 15c/Kwh total. The Asus GX10 has a 240W power supply, and it's probably only pulling 180. I estimate $15-$20/month. But worst case red-lining. 240 Watts, 720 hours = 172KWH , and at $0.20, I come to $35
Here's the kicker thought - that github copilot subscription I mentioned? I have another agent running on that, reading all my other agent logs, managing my obsidian notes, doing research, sending briefings. And all by itself, it used almost the same amount of claude-opus tokens for that $39/month subscription. I was actually a bit shocked when I pulled a recent report and saw that. I'm working to migrate functionality away from copilot subscription to the local model. A lot of the initial setup might have needed it, but not the ongoing review style work it does.
A few things. I replied to someone else above, but I feed lessons learned from my social ant farm agents back into more productive agents.
Memory recall:
Lots of systems out there to give agents memory. I've used a bunch and written a couple. Storing memories is easy, but getting an agent to recall them, no matter how much you mention it in your AGENT/CLAUDE.md is a bit of an uphill battle. I've even watched claude make useful project memories and never refer to them again.
In my agent ant farm - agents go "to sleep" at night. They get nudged to head home, once there they get prompted to make notes about their day, about other characters. Then we do a compact with custom instructions. After compact/sleep cycle, if they enter a room with one of the characters in their notes, that gets loaded back into context automatically.
That all boils down to hooks in Pi like before_agent_turn. You can intercept a prompt, check it against code/flat files, and smartly inject more information into context. You can have a long running main session with compacts that discard procedural bits and offload the rest to memory.
Time Awareness:
Agents have no concept of time. You can send them a message at 5am, then at 10pm, and it's been 2 turns for them. For coding, this is fine. But for assistant level stuff, adding a message like "It's 3PM. It has been 3 hours since the last interaction with the user" goes a long way. Without me saying something like "new topic", it knows now that time has passed, i'm probably onto something new. If I left something hanging, it will remind me about it, or maybe go check on things that should have happened during the day.
Inner Thoughts/Idle nudges:
I can have an extension run every 5 seconds, check a a schedule, check activity level of the main session and fire off nudges on the main session. These look like the user sent it, but I generally prefix it with [inner thought]. For my social bot, I tested this along the lines of "[inner thought] it's been 3 hours since you last talked with user, why not reach out, let him know what's new, maybe send a selfie or a photo of where you are". For my assistant bot, it's an 8am, 3pm, and 7pm nudge along the lines of "[inner thought] put together an activity report of work things that has changed since the last report". This all runs in the main context, they get the thought, have historical context, can run skill to check on vault updates, open beads, anything observed from ingesting other agent sessions, and sends me a summary. It take into account my idle factor. If I'm heavily engaged in conversation at 3PM, the report might get delayed 15 minutes or an hour, or skipped altogether.
Awesome project and thanks for sharing. I've been trying to do similar things with much, much more meager hardware and your observations align with what I've discovered. Autonomy is hard, memory and "will" is hard to get going. Time is not a concept to LLMs in anything resembling a human manner. I'm trying a more emergent approach but the urge (and occasional need) to nudge is strong. If you're interested in seeing what I've been doing my Github is in my profile.
A little of A, a little of B. I have a lot of fun building it out, it's surpassed Factorio in addiction, and I've been able to flesh out some patterns that I roll back into more productive agent harness bits.
For A:
The learning is in building agent harnesses that aren't just cron jobs reading a file like HEARTBEAT.md. I have some serious tools for my own use. One main assistant/coordinator agent, one SRE/coder agent (with sub-agents of its own).
I originally just started last year with the AI assistant (Jane from enderverse). Along the way building scheduled systems, hand offs to other agents, etc. As I ran into problems, I'd be rewriting and refactoring. So I spent some time making a low-stake hatbot with history and routines. Instead a from-scratch golang harness, I built it around pi and extensions. Time of day prompt splices (extensions can inject into or modify prompts on the fly, wake up reminders. Things that you do in the main session vs spinning up an ephemeral session. Self improvement daydreaming (modify your own skills and AGENTS.md) A lot of that went back into rebuilding Jane to something more useful for me.
For B:
The "dynamic dollhouse" as you put it was seeing where I could take that living chatbot next. There's a lot of projects pointing agents at slack, discord, message boards. I figured why not a mud with rooms, weather, and props. Lots of interesting challenges. How to keep bots from nesting in their own room, how to keep them from yes-anding each other all day long. How to slow down 3 bots talking at each other so a human can get a word in edge-wise.
Different levels. There's plain old NPCs that have dice roll random responses. There's LLM driven NPCs that only remember the last 5-10 messages. And the main ones are bot agents. Full agent harness, moving around the environment. Long lived context windows. One character (a nurse at the hospital) gets into arguments with an NPC receptionist that treats her as another patient. Complains about it to other characters, they remember and the word spreads.
The agents get prompted to write down notes, the head home for sleep (and session compaction). Next time they enter a room with that person after compact, their notes get loaded automatically. This kind of behavior can feed back into the more productivity based agents.
For open models, usually not well. You get 5+ providers competing on cost, all with cheaper electricity and better hardware utilization than your local setup
The TL;DR though is that a 10-15b param model baked into an ASIC with the latest fab tech would take around 62W of power draw when active. At ~10k+ t/s though it likely would only be active for short bursts of time. It'd fit perfectly fine within the thermal envelope of a laptop.
The approach makes a lot of sense. Once you get to those speeds, latency of the network becomes one of the bigger bottlenecks, so local has a real advantage over a subscription.
This assume Claude's price doesn't change. Which isn't a great assumption considering inference providers are moving to usage based billing. Also the VC money isn't going to last indefinitely. Current inference providers are being subsidized with VC money at this point.
Ok heres the thing you will nevwr be able to truly do this due to logic.
Logically five people pooling their resources beats one guy.
therefore datacenters will always win because they get higher time utilization.
so forget it.
I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware.
you will get better hardware though, cutting edge will always be cloud
Laptops/desktops are cheaper per flop than any datacenter hardware by a good order of magnitude.
The problem is that expectations rise in datacenters, hardware/power/security/availability guarantees cost real money. Then the operator providing these guarantees expects some margin.
You can see this most clearly with "developer desktops", a gcp instance costs about 10x a hetzner instance which costs between 5 and 10x the same hardware sitting in the back of an office somewhere. While all of these premiums matter for 24/7 systems under active development, they don't really matter for ephemeral small scale workloads.
They just mean this part: "where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription."
Upgrading local hardware will remain the more expensive alternative to the subscription regardless what the relative cost of running the models themselves are. If the local hardware to do so becomes affordable then the subscription will be even more affordable, not expensive.
At least for these kinds of mega tasks. For more micro task we will always end up with unutilized local compute we already purchased which will be "free" since we already paid for non-AI reasons (e.g. a gaming GPU while not gaming).
Twenty years ago, I don't think any of us were excited about a future internet where we couldn't trust whether what we were seeing or reading was genuine. I hope one day we'll be able to look back on this era as an aberration, like that scene in Mad Men where the Drapers fling their picnic rubbish onto the grass and drive away.
It seems to me the era of being able to trust pictures was an aberration. Before the camera images created in any form might depict something that really happened, an exaggeration, or a total fabrication. The camera represented a technological leap that made capturing reality significantly more easy than faking it, though faking it was never actually all that hard. Now technology has progressed again and we're back where we started. Any image might be real, edited, or totally fabricated, and we can no longer fool ourselves about "photographic evidence." Trust is and always will be about credibility of the claimant. Additional evidence is itself only as trustworthy as its providence. An attempt to destroy the ability to create images that resemble photographs is doomed to fail and wrongheaded to begin with. The only reason such an idea would occur to someone is they were born in an aberrant era where the culture had ingrained in them the semi-grounded belief that certain types of images are representative of reality. That wasn't the case historically and won't be again.
Twenty years ago my teachers were telling me not to use Wikipedia because you can't trust anything on the internet. You should never date someone you met through an app or website because they are 100% murderers. "The internet is for porn". Things have a way of improving over time, and people always overestimate societal risks of new tech in the beginning.
Young girls suicides, Brexit&Trump (post-truth politics, general democratic decline), demographic catastrophe, obesity crisis are often partially attributed to social media.
Not saying that tech is inherently evil, these could have been prevented, but to me it seems we have underestimated the social risks and failed to regulate accordingly.
For any new change we underestimate certain risk and overestimate others. This leads to a lot of focus and discussion on the things that don't matter much while the things that do matter become apparent. It's natural, guessing everything about the future correctly is never going to happen, but I don't really think that info tells you things are getting better or getting worse overall - it just tells you what things we focused on instead of should have focused on.
Even in healthcare alone, I would question whether the impact was net positive. You have to balance the "access to information" with how people need to deal with false information, health charlatans, ads shoving then supplements they don't need, people's inability to discern what to do based on even true information that they get.
What are the positives? Before, you had to go to a doctor to do stuff, now you do too. Maybe people are more likely to get tested. Maybe scheduling appointments is a bit better. What am I missing?
debating this isn't particularly useful of either person's time.
There are plenty of documented cases of reddit posts encouraging deeper investigations or legally authorized tele-health providers (where you can communicate directly with a licensed medical professional).
The number of people using telehealth services is significantly less than the number of people that have bad health outcomes from internet usage. If it were otherwise, ChatGPT, Reddit, and telehealth companies would be managed very differently.
Demographic decay was already going on in the 90s in some countries (Italy, South Korea, Japan), and the obesity crisis in the US too. For Brexit & Trump it could have been a contributing factor, but Italy had Berlusconi in the 90s just with TV and France almost elected Le Pen (father).
That said, while I am generally skeptical of these effects, fake news are a real problem that social media has exacerbated.
You should reconsider the scope. Nobody is losing sleep over Airbnb pics.
We’re in an era now where every image and video (and for that matter audio) is potentially fake; where knowing what’s real and true is no longer possible.
> I think the inability to see the freedom AI gives people is one of the saddest things I've seen.
No one’s failing to see the good things, hypothetical or not. Most of us are aware just fine, we just don’t all agree that the negative trade-offs are worth it.
I enjoy the technology too, but the tradeoffs are pretty grim. It takes stepping outside of my bubble to see it in full force, but AI misinformation is already rampant.
I think it may turn out postive; That the less we are able to take images and video at face value the better.
Motivated actors have been able to doctor, fake, or spin media content since time immemorial. But peoples default mode was to trust what they saw. Now that fake imagery is ubiquitous, maybe we'll all get a bit more skeptical.
Sure, but LLMS and image generators are not the death of "consensus reality". Healthy democracies will still have investigative journalism, public debate, trustworthy institutions, etc.
> will still have investigative journalism, public debate, trustworthy institutions, etc.
All of which are under serious threat from social media, buyouts by billionaires, and simple smear campaigns. Not dependent on AI, but effects which will be magnified by AI.
being skeptical and determining the truth takes a lot of work. I fear that we may just refuse to wade through all the lies and just accept a enforced willful ignorance.
You don't remember the discussion around Narrative Science (https://en.wikipedia.org/wiki/Narrative_Science) then. They were a university spin-out that could write plausible-sounding baseball news articles (and later finance) from the stats. Their software enabled local news websites to publish articles about every game, which was seen as a boon to sports fans and a key driver for web traffic. There was a lot of criticism about how it wasn't 'real' though.
For as long as we've had computers people have tried to make them sound human. It's not a new thing that people are concerned about knowing if they're talking to (or reading) a robot imitating a person.
> could write plausible-sounding baseball news articles (and later finance) from the stats
Back in the day, baseball commentators sometimes did this for live games they couldn't see based on very limited information they were being passed. One such commentator was .. Ronald Reagan.
Literally the first thing I wrote after OpenAI's chat completions API came out was a Python script that took in a JSON description of a football (soccer) game from an API and used gpt-3.5-turbo to generate an article about it.
I don't know what else you'd call the widespread and enthusiastic adoption of a technology that is designed to exploit people's trust in the veracity of images by mimicking reality as seamlessly as possible. I think it's both aberrant and abhorrent for the tech industry to be actively developing something that's permanently polluting our information environment.
I think the massive scale up in making bogus images is actually a net good in the end. For decades, even before computers, images were edited on the idea people trust an image. Anything from removing/adding figures to a photo, changing something in the image to modify the meaning, or cropping something out to hide part of it. People had a default trust of images because 99% of the images they saw weren't worth modifying, so they didn't think about it.
Now images have gone back to being like a book. People are beginning to assume anyone can write anything, make up quotes, etc. We still get to keep the ones we just like or the ones we know were really that way (e.g. your wedding vows/wedding photos) but we've dropped this silly notion image=fact just because it's an image. It's not all good that faking an image has gotten consistently easier over the last 100 years, but it's also not bad that it's fallen apart enough people are seeing through it.
Are there any examples where it has happened in tech? Maybe internet Pop-ups are closest, that are now automatically blocked. But seems unlikely that we would not use image generation. Just not trust any image by default.
There's always existed misinformation in text and in images. It's been possible to manipulate photographs for as long as photography has been around. It's becoming easier for sure, but it's not really a qualitative change. Trusting anything you saw on the internet twenty years ago would have been as ridiculous as it is today.
This is seriously underplaying it. It's become trivial to generate and inundate the internet with fake content (either for laughs, for internet points, or for more nefarious purposes). Manipulating photos required a lot of skill to make something plausible. We're reaching a point (if we're not there yet) where most content produced on the internet is fake.
I am pretty excited. The factuality of important events has been distorted for most of history. Moving to a low information trust society is something that I think will be positive.
I don’t see it leading anywhere but a flat earth. When no one can be trusted whoever can tell you want to hear will be who people listen to and snake oil salesmen will reign supreme. Even if he was CIA, Cronkite’s world was closer to the truth than Alex Jones’.
One of my professors at Uni a year ago was arguing that due to genAI we would have a shift of trust into established institutions/people, so government (I'm not American so I don't know if this is possible after many recent scandals for example), Universities, people with authority/knowledge, close family members that are trust worthy, knowledgeable and/or work in before mentioned institutions. So basically we would revert to pre-web times where we had to trust some entities whenever we liked it or not.
I personally worry that what that would mean is we are left with little to no institutions to trust, besides Universities and family members, I don't think I would be able to trust governments and corporations, but I guess before internet people also weren't blindly trusting those.
I doubt the new trust bearers will be anything like governments and universities because trust in them has been severely eroded. Sadly, they will be select Youtubers and Internet influencers.
This is kinda cute because it glosses over the lack of critical thinking skills, lack of research skills, and willingness to believe magic bullets, which would make most of us believe nothing of substance and yet fall for anyone with a silver tongue. Heck, we’re already dangerously close to that without LLMs.
Low trust societies are poorer because everyone has to spend much more effort on verifying everything. People give up business opportunities because they can't trust their partners. It becomes more nepotistic because people trust family over strangers.
Low information trust societies get destroyed by pandemics of both physical viruses (due to anti-vax and medical distrust; we can see this happening again with Ebola) and destructive memetic lies (see 20th century fascism).
Ignore the naysayers, they are just jealous. You got it totally tight, not everyone get's it like we do. We are facing alot of backlash for our beliefs these days.
Listen, I'm hosting this Telegram channel for people like us, where we can exchange free information without media bias, share the real facts and plan coordinated activities against these poisoning mainstream scumbags.
I also have a 20% coupon code for Wamp® Wolf-Testosteron for you, Wamp® really helped me stay awake and alert in these dire times.
I saw '1-bit' and my mind first went to 1-bit dithered B&W image generation, not 1-bit model weights....
and so now I'm wondering how cool /fast / compressed a diffusion image generator could be if the images it was trained on / space it worked in was limited to 1 bit (Floyd-Steinberg / Atkinson / your favorite algo here) dithered images.
Training would surely be pretty quick and probably fit onto one modern GPU.
IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on.
Okay, maybe this allows running a local model on something that has a reasonably powerful GPU and limited memory, like an iPhone, but is that really a common requirement?
It's useful progress. Decent-fidelity local-scale inference means that you can create a product that generates throwaway images frequently without worrying about cost. Thus far every product I've seen that generates images is metered, which severely limits the value. I don't know if this is actually at the "decent fidelity" point yet.
We are in an era of extreme demand for GPU and limited supply. Every inference we push to the edge frees cloud resources for other tasks. Every efficiency gain increases what we can achieve with existing resources. If images can be rendered with half as much compute, we need half as many GPUs.
I think the value of it is currently more academic than useful in the real world. Everything at the frontier is still only marginally Good Enough (in image generation, most of it is shit even from the best models), so things far behind the frontier in terms of capability (as a tiny 1-bit model necessarily must be) are unusable.
But, getting remarkably higher density of capability per unit of compute is a big thing. It means the frontier can get better and cheaper to operate and less resource hungry, and it means what can be accomplished at the edge, on personal laptops or phones, becomes a broader spectrum of tasks.
And, for privacy, there are a lot of things that should run on-device and not everyone has big dedicated GPUs.
It solves part of the download issue if they actually delivers a 1-bit whole package (currently their download is around 3.5GiB, still not ideal since FLUX.2 [klein] 4B you can get a package including text encoder ~6 GiB).
For speed, no. Draw Things runs on iPhone just fine and generally faster than their implementation on the same model (FLUX.2 [klein] 4B).
> Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway.
Not the bottom end - most people are on laptops or mobile devices that are much lower GPU power than this.
Probably the bottom end an individual would want to consider using due to slow generation time.
Sure, you could theoretically take a model compressed in this manner and deploy it on an old netbook and run the calculations on the CPU, but each image would probably take an hour…
I was thinking more about the 0-3 year old midrange x86 laptops and phones, they have unified memory GPUs that are easily worth using (vs CPU), support narrow FP datatypes but don't have a ton of memory bandwidth.
Genuine question: doesn't it blow your mind that there exists a 1 Gigabyte file/program that can generate any image you can think of just from a rough description of it?
Their 1-bit quantized Diffusion Transformer is just under 1 GB. You also need the text-encoder (4-bit quantized) and VAE (unquantized) for inference and their combined weight is ~3.42 GB.
Yeah, it's pretty incredible. And I guess that's mostly what's behind the question: whether this is more of an impressive research/technique demonstrator, or a real product advancement solving a need.
> doesn't it blow your mind that there exists a 1 Gigabyte file/program that can generate any image you can think of just from a rough description of it?
I can make this into a 5-lines Python program. I’m not saying the images will match the description, but that isn’t part of your spec ;)
It’s like asking how did Memoji generation on iPhone solved a real problem?
It does not need to directly solve any particular problem to be overall good for consumers, by putting pressure to all those subscription based solutions… at least it’s private and does not require you to provide all your data…
Yes, size and performance are not only problems for local LLMs, they are problems for frontier LLM companies like OpenAI and Anthropic. The latter still lose a ton of money on inference and advances in efficient, performant models helps their bottom line.
Yes its a huge deal because these are starting to get bound by memory bandwidth not compute. therefore one bit wirfhts stream way faster leading to substantially better results. At least thats what Id guess!
Not quite as I understand it. The ternary approach bonsai uses leverages a FP16 scaling factor that each value in the ternary maps to. You're still using 16 bit multiplication, it's just that the weights are far more compressed.
fair, i think i was referring more to 1.58 bit architecture in general since the original paper (Figure 3) shows that we eliminate FP16 multiplication and addition just for INT8 addition. I need to dive deeper into bonsai overall if it differs
> To our knowledge, Bonsai Image 4B is the first image model in its parameter class to run directly on an iPhone.
This is wrong. But they worded it carefully to be not entirely wrong.
FLUX.2 [klein] 4B (the same parameter class, basically the same model) runs on iPhone through Draw Things app, with 8-bit or 6-bit quantization (hence not "directly", I guess, but that is the technicality that sounds fishy enough).
Couldn't try it because the demo app is iOS only and the web version just crashes my browser. The small model is impressive but if you front load a 1.8GB text encoder model, the savings aren't quite as useful.
I extracted the code from the web demo to add to make a web image generation node to my in browser ai workflow tool, and it’s pretty sweet. Waiting for xenova to add to transformersjs 4.3 and I’ll release as well. Couldn’t wait though to test.
can you describe your "in browser ai workflow tool"? I may or may not be working on something similar and am very interested in what others are building in the space.
I believe it's the way the HN algorithm works. In order to give new and obscure posts a shot, it will add them to peoples feeds in their front page and see how they measure. Otherwise new posts wouldn't get seen and the flywheel would never get started.
So everyone acts as a sort of beta tester for obscure posts.
On weekends, yes. During the week, that’s also true if they arrive within a short time frame, e.g., three minutes. Almost no one looks at “New”. That is the real issue.
It’s about how quickly they get those points. It doesn’t have to be bots. Sending a post to friends with reputable human profiles, and asking for a vote kinda works of most social networks. Some social networks claim they have protection against this but I wouldn’t bet they catch everything.
The white paper says "mean-active memory pressure down to 1.95 GB for 1-bit Bonsai Image 4B and 2.38 GB for
Ternary Bonsai Image 4B". Storage is on the linked page, and is about half that.
That is very low, looks like it should run in base MacMini M4 with 16GB RAM. I understand it is not released yet? What sort of harness is necessary for this type of model? (I have only used coding agents through GH Copilot in VS Code, the JetBrains AI tool and Pi, this last one was sort of a pain to setup…)
Stuff like this is great - more promises of things that can run on phones please!
Sadly right now the expensive developer subscription means the few folks willing to hold a forever subscription make something that barely works then move on… or make something with so many ads it is an app. For example Google’s “Model Garden” app has no ads but still has major UX issues and isn’t suitable for daily use, even though the models are amazing.
Raising awareness of how capable today’s phone hardware is will make normal people demand to run what they choose on their phones. It’d be a much stronger way back to general purpose computing than via all legislation that has been tried so far..
I run a moderately popular image comparison benchmark site called GenAI Image Showdown [1]. You can click “View All Models” and filter the list down to just locally runnable options (Flux, Qwen, Hunyuan, etc.).
Except the two (GPT-Image-2 and Nano Banana Pro), anything displayed here can run on the 16 GiB MacBook (including the FLUX.2 [dev]): https://tests.drawthings.ai/generate
If it were illegal to generate a sunset, people would absolutely seek it out to generate a sunset, if just as an act of civil disobedience. Look at how people reacted to being told sharing a number is illegal[1]. Why would they act differently about an illegal computation?
Just a side note, that this website is classified by Apple as an Adult website. I have Limit Adult Websites set in Content & Privacy Restrictions switched on.
Led me to wonder what happens if a domain gets a new owner, and they want to petition Apple to remove the block.
what trade off would one need to clear to justify the hardware and the work to get this running locally as part of a broader system? It’s a lot of work setting up and maintaining a production harness/system on a local device. I don’t personally repeatedly generate images at a scale where using a lab’s app somehow burns all my tokens. I like the ideas of local ai but I don’t see widespread adoption of it happening in commercial or customer situations anytime soon no matter how little/good enough they get. Even Uber- token burn whiplash but I doubt their answer will be “run some of it local”. IT nightmare, I’d imagine.
This is why I don't think the big AI companies and nvidia will dominate the market. AIs will just run locally, on whatever hardware you have. Perhaps that's why they worked on this yet-to-be-defined partnership with ARM.
Can't speak for browser demos, but I just got the ternary model working on my M5 generating images. The 1 bit didn't work, as it has a known bug with XCode 24.5 and I wasn't in the mood for installing 24.4 alongside.
The online demos require WebGPU so Firefox on mobilr and privacy enhanced browsers will break. WebGPU support on Linux and other open source systems is also trash, you can force it to work in Chrome but it won't be happy.
There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than sufficient.
This is a hobby version of a project, but you can imagine commercial versions of the same prompt for new databases, genomics studies, material analysis, operating systems etc.
Thats the thing - these models see and predict tokens. For any real engineering you get more bang for your buck using math.
There might be some interesting side effects from making simulation software, which is currently either an expensive niche or quirky university project (SPICE always has that feel).
I would very easily find ways to hit that level of token usage if it was cheaper/faster.
That said, if you are doing always on agents and you spend $3k-$4k on a GB10 or, $5+ k on Apple Silicon as your sunk cost, you will probably come out ahead.
I've got 5 agents running a purely experimental social experiment. AThey operate in an evennia mud (a familiar sounding city called "gothmud). I've built a channel, idle prompts, sleep schedule. I feed in real world news, weather. There's a character up in a clock tower that reads evennia's audit logs every 20 minutes to surveil the city, and a cast of people wandering around, investigating things, having coffee, repairing robots. This is all hitting qwen3.6-35-A3B on the Asus GB10, which cost me $3k.
Over the last 30 days, I've hit 394M input tokens, 1.6B output tokens. I would have spent between $1600 to $1700 if I was using openrouter. Not calculated - I also have comfyui running in the spare space, and the agents "take photos" of the rooms they're in, selfies, workshop photos, etc.
How much did I spend on electricity? I don't have a meter on my box. My total electric bill for the last 30 days was $220, so I know it's less than that. My rate to compare is 11.7/kwh, but it's closer to 15c/Kwh total. The Asus GX10 has a 240W power supply, and it's probably only pulling 180. I estimate $15-$20/month. But worst case red-lining. 240 Watts, 720 hours = 172KWH , and at $0.20, I come to $35
Here's the kicker thought - that github copilot subscription I mentioned? I have another agent running on that, reading all my other agent logs, managing my obsidian notes, doing research, sending briefings. And all by itself, it used almost the same amount of claude-opus tokens for that $39/month subscription. I was actually a bit shocked when I pulled a recent report and saw that. I'm working to migrate functionality away from copilot subscription to the local model. A lot of the initial setup might have needed it, but not the ongoing review style work it does.
Memory recall:
Lots of systems out there to give agents memory. I've used a bunch and written a couple. Storing memories is easy, but getting an agent to recall them, no matter how much you mention it in your AGENT/CLAUDE.md is a bit of an uphill battle. I've even watched claude make useful project memories and never refer to them again.
In my agent ant farm - agents go "to sleep" at night. They get nudged to head home, once there they get prompted to make notes about their day, about other characters. Then we do a compact with custom instructions. After compact/sleep cycle, if they enter a room with one of the characters in their notes, that gets loaded back into context automatically.
That all boils down to hooks in Pi like before_agent_turn. You can intercept a prompt, check it against code/flat files, and smartly inject more information into context. You can have a long running main session with compacts that discard procedural bits and offload the rest to memory.
Time Awareness:
Agents have no concept of time. You can send them a message at 5am, then at 10pm, and it's been 2 turns for them. For coding, this is fine. But for assistant level stuff, adding a message like "It's 3PM. It has been 3 hours since the last interaction with the user" goes a long way. Without me saying something like "new topic", it knows now that time has passed, i'm probably onto something new. If I left something hanging, it will remind me about it, or maybe go check on things that should have happened during the day.
Inner Thoughts/Idle nudges:
I can have an extension run every 5 seconds, check a a schedule, check activity level of the main session and fire off nudges on the main session. These look like the user sent it, but I generally prefix it with [inner thought]. For my social bot, I tested this along the lines of "[inner thought] it's been 3 hours since you last talked with user, why not reach out, let him know what's new, maybe send a selfie or a photo of where you are". For my assistant bot, it's an 8am, 3pm, and 7pm nudge along the lines of "[inner thought] put together an activity report of work things that has changed since the last report". This all runs in the main context, they get the thought, have historical context, can run skill to check on vault updates, open beads, anything observed from ingesting other agent sessions, and sends me a summary. It take into account my idle factor. If I'm heavily engaged in conversation at 3PM, the report might get delayed 15 minutes or an hour, or skipped altogether.
What is the experiment? What are you hoping to learn from all this?
Or do you just mean you've made a dynamic dollhouse that you think is cool? The Sims on your own terms?
For A:
The learning is in building agent harnesses that aren't just cron jobs reading a file like HEARTBEAT.md. I have some serious tools for my own use. One main assistant/coordinator agent, one SRE/coder agent (with sub-agents of its own).
I originally just started last year with the AI assistant (Jane from enderverse). Along the way building scheduled systems, hand offs to other agents, etc. As I ran into problems, I'd be rewriting and refactoring. So I spent some time making a low-stake hatbot with history and routines. Instead a from-scratch golang harness, I built it around pi and extensions. Time of day prompt splices (extensions can inject into or modify prompts on the fly, wake up reminders. Things that you do in the main session vs spinning up an ephemeral session. Self improvement daydreaming (modify your own skills and AGENTS.md) A lot of that went back into rebuilding Jane to something more useful for me.
For B:
The "dynamic dollhouse" as you put it was seeing where I could take that living chatbot next. There's a lot of projects pointing agents at slack, discord, message boards. I figured why not a mud with rooms, weather, and props. Lots of interesting challenges. How to keep bots from nesting in their own room, how to keep them from yes-anding each other all day long. How to slow down 3 bots talking at each other so a human can get a word in edge-wise.
Different levels. There's plain old NPCs that have dice roll random responses. There's LLM driven NPCs that only remember the last 5-10 messages. And the main ones are bot agents. Full agent harness, moving around the environment. Long lived context windows. One character (a nurse at the hospital) gets into arguments with an NPC receptionist that treats her as another patient. Complains about it to other characters, they remember and the word spreads.
The agents get prompted to write down notes, the head home for sleep (and session compaction). Next time they enter a room with that person after compact, their notes get loaded automatically. This kind of behavior can feed back into the more productivity based agents.
The TL;DR though is that a 10-15b param model baked into an ASIC with the latest fab tech would take around 62W of power draw when active. At ~10k+ t/s though it likely would only be active for short bursts of time. It'd fit perfectly fine within the thermal envelope of a laptop.
The approach makes a lot of sense. Once you get to those speeds, latency of the network becomes one of the bigger bottlenecks, so local has a real advantage over a subscription.
This isn't true.
Anthropic is making an operating profit including the loss making subsidised subscriptions (but excluding training).
Your normal inference provider is doing great. Do the math on a H100 rental and you can see the margins.
Logically five people pooling their resources beats one guy.
therefore datacenters will always win because they get higher time utilization.
so forget it.
I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware.
you will get better hardware though, cutting edge will always be cloud
The problem is that expectations rise in datacenters, hardware/power/security/availability guarantees cost real money. Then the operator providing these guarantees expects some margin.
You can see this most clearly with "developer desktops", a gcp instance costs about 10x a hetzner instance which costs between 5 and 10x the same hardware sitting in the back of an office somewhere. While all of these premiums matter for 24/7 systems under active development, they don't really matter for ephemeral small scale workloads.
At 10x you have to be at hours per day and 5x you’re at 4h.
HBM has way higher bandwidth and its not all about flops.
Also the FP4 flops (inference) are so mind bogglingly high on these things.
Lastly what you fail to consider is the chip to chip bandwidth which is critical.
the people running these know that networking is just as critical.
all reduce etc.
they wouldnt pay if they could get something better value.
Don't think anyone was refuting that?
And of course when you pool resources you have access to more resources.
Upgrading local hardware will remain the more expensive alternative to the subscription regardless what the relative cost of running the models themselves are. If the local hardware to do so becomes affordable then the subscription will be even more affordable, not expensive.
At least for these kinds of mega tasks. For more micro task we will always end up with unutilized local compute we already purchased which will be "free" since we already paid for non-AI reasons (e.g. a gaming GPU while not gaming).
Which explains why you're using a dumb terminal to access compute services?
Not saying that tech is inherently evil, these could have been prevented, but to me it seems we have underestimated the social risks and failed to regulate accordingly.
If the internet saves 1,000 lives per day, and hurts 1, then the roi is there.
What are the positives? Before, you had to go to a doctor to do stuff, now you do too. Maybe people are more likely to get tested. Maybe scheduling appointments is a bit better. What am I missing?
There are plenty of documented cases of reddit posts encouraging deeper investigations or legally authorized tele-health providers (where you can communicate directly with a licensed medical professional).
The number of people using telehealth services is significantly less than the number of people that have bad health outcomes from internet usage. If it were otherwise, ChatGPT, Reddit, and telehealth companies would be managed very differently.
98point6 - 50k+ reviews
https://apps.apple.com/us/app/98point6-by-transcarent/id1157...
doctor on demand has millions of users
https://apps.apple.com/us/app/doctor-on-demand/id591981144
Its seems to be hidden from view. Global health is getting worse, not better
Covid pushed back the longevity gains, but there hasn't be a decline or flatten since the internet grew in popularity (2000 - 2019).
https://www.who.int/data/gho/data/themes/mortality-and-globa...
That said, while I am generally skeptical of these effects, fake news are a real problem that social media has exacerbated.
I think the inability to see the freedom AI gives people is one of the saddest things I've seen.
I remember when the internet was young people would complain about how it was becoming read-only.
Now we have a tool to let people express themselves and people complain the fact there are fake pics on AirBNB means the collapse of society. Please!
We’re in an era now where every image and video (and for that matter audio) is potentially fake; where knowing what’s real and true is no longer possible.
No one’s failing to see the good things, hypothetical or not. Most of us are aware just fine, we just don’t all agree that the negative trade-offs are worth it.
Motivated actors have been able to doctor, fake, or spin media content since time immemorial. But peoples default mode was to trust what they saw. Now that fake imagery is ubiquitous, maybe we'll all get a bit more skeptical.
All of which are under serious threat from social media, buyouts by billionaires, and simple smear campaigns. Not dependent on AI, but effects which will be magnified by AI.
Slate published this about it in 2012: https://slate.com/technology/2012/03/narrative-science-robot...
For as long as we've had computers people have tried to make them sound human. It's not a new thing that people are concerned about knowing if they're talking to (or reading) a robot imitating a person.
Back in the day, baseball commentators sometimes did this for live games they couldn't see based on very limited information they were being passed. One such commentator was .. Ronald Reagan.
I was surprised how well it worked, even then.
Here's a local story published after I made my comment, about tour operators using AI images to misrepresent destinations in the area: https://www.abc.net.au/news/2026-06-01/ai-videos-spark-conce...
Increasing the availability of fake image generators directly enables more harms like these.
Now images have gone back to being like a book. People are beginning to assume anyone can write anything, make up quotes, etc. We still get to keep the ones we just like or the ones we know were really that way (e.g. your wedding vows/wedding photos) but we've dropped this silly notion image=fact just because it's an image. It's not all good that faking an image has gotten consistently easier over the last 100 years, but it's also not bad that it's fallen apart enough people are seeing through it.
This is seriously underplaying it. It's become trivial to generate and inundate the internet with fake content (either for laughs, for internet points, or for more nefarious purposes). Manipulating photos required a lot of skill to make something plausible. We're reaching a point (if we're not there yet) where most content produced on the internet is fake.
I understand the point of distorted facts, but what I’m not sure how things are improved by basically having no trust in any facts?
I personally worry that what that would mean is we are left with little to no institutions to trust, besides Universities and family members, I don't think I would be able to trust governments and corporations, but I guess before internet people also weren't blindly trusting those.
Low information trust societies get destroyed by pandemics of both physical viruses (due to anti-vax and medical distrust; we can see this happening again with Ebola) and destructive memetic lies (see 20th century fascism).
Listen, I'm hosting this Telegram channel for people like us, where we can exchange free information without media bias, share the real facts and plan coordinated activities against these poisoning mainstream scumbags.
I also have a 20% coupon code for Wamp® Wolf-Testosteron for you, Wamp® really helped me stay awake and alert in these dire times.
It's true: Wamp, it really whips the Llama's ass!
and so now I'm wondering how cool /fast / compressed a diffusion image generator could be if the images it was trained on / space it worked in was limited to 1 bit (Floyd-Steinberg / Atkinson / your favorite algo here) dithered images.
Training would surely be pretty quick and probably fit onto one modern GPU.
IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on.
Okay, maybe this allows running a local model on something that has a reasonably powerful GPU and limited memory, like an iPhone, but is that really a common requirement?
But, getting remarkably higher density of capability per unit of compute is a big thing. It means the frontier can get better and cheaper to operate and less resource hungry, and it means what can be accomplished at the edge, on personal laptops or phones, becomes a broader spectrum of tasks.
And, for privacy, there are a lot of things that should run on-device and not everyone has big dedicated GPUs.
For speed, no. Draw Things runs on iPhone just fine and generally faster than their implementation on the same model (FLUX.2 [klein] 4B).
Not the bottom end - most people are on laptops or mobile devices that are much lower GPU power than this.
Sure, you could theoretically take a model compressed in this manner and deploy it on an old netbook and run the calculations on the CPU, but each image would probably take an hour…
Their 1-bit quantized Diffusion Transformer is just under 1 GB. You also need the text-encoder (4-bit quantized) and VAE (unquantized) for inference and their combined weight is ~3.42 GB.
TBF, even at that size it's no less mind blowing.
I can make this into a 5-lines Python program. I’m not saying the images will match the description, but that isn’t part of your spec ;)
It does not need to directly solve any particular problem to be overall good for consumers, by putting pressure to all those subscription based solutions… at least it’s private and does not require you to provide all your data…
https://arxiv.org/pdf/2402.17764
This is wrong. But they worded it carefully to be not entirely wrong.
FLUX.2 [klein] 4B (the same parameter class, basically the same model) runs on iPhone through Draw Things app, with 8-bit or 6-bit quantization (hence not "directly", I guess, but that is the technicality that sounds fishy enough).
Isn't SD XL 3.5B? And the refiner model is even larger. Those can run on an iPhone 13 Pro.
I do wonder how these compare to existing image generation models. I've tried https://github.com/alichherawalla/off-grid-mobile-ai for a while but I find the image generation models rather lacking.
So everyone acts as a sort of beta tester for obscure posts.
Sadly right now the expensive developer subscription means the few folks willing to hold a forever subscription make something that barely works then move on… or make something with so many ads it is an app. For example Google’s “Model Garden” app has no ads but still has major UX issues and isn’t suitable for daily use, even though the models are amazing.
Raising awareness of how capable today’s phone hardware is will make normal people demand to run what they choose on their phones. It’d be a much stronger way back to general purpose computing than via all legislation that has been tried so far..
https://genai-showdown.specr.net
I can think of a lot of positives. The negatives amount to a convoluted argument about the limits of free speech.
Prisoner 2: I made a picture of a nice sunset over the ocean
[1]https://en.wikipedia.org/wiki/AACS_encryption_key_controvers...
Website Not Allowed “prismml.com” is a restricted website.
Led me to wonder what happens if a domain gets a new owner, and they want to petition Apple to remove the block.
I took few minutes to try to make it work on ROCm (AMD's alternative to CUDA), landed in python dependency hell.
What do you mean? They are the ones introducing the matmul extensions to Vulkan, which makes compute like this possible
Here's a generation in your honor: https://peterc.org/img/johndoe.png
NVIDIA Card Firefox wayland
Is it compatible with Ollama, ComfyUI or are those providers unneeded, compatible with low-end hardware?
Also, where does "./setup.sh/ drop the components in Linux?
Thank you, Sol
having trouble loading the webgl browser demo on my phone but no biggy