Very very cool! Impressed that it can be done with only fine tuning
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
You jest, but that actually sounds like a really good research direction. Imagine what you could do if merging two codebases was as easy as merging with neural git.
Love it. In fact the demo already does something like that. It trains 2 models, the neural net that saves the data, and a classifier that trains on correcting the data - and it has a huge impact, you can go from 30% fidelity to 90-100% just with that
That's very cool project.
I read this blog post a few days ago about one way to look at memory encoding in models https://alexwang.ai/posts/associative-memory/ that you might find relevant
Just to point out your site text appears too small on mobile (no mobile view) and it doesn't have a direct link to the repo. If either is by choice then that's fine of course.
> Commit code, it trains. Checkout, it reconstructs from memory.
I see a "Commit" button, but I don't see any checkout, I guess that is what happens automatically? Although that panel says "Reconstruction", not "Checkout"... You might want to do a pass after the AI created the initial version, and make it "humanly consistent" or similar, as right now it seems a bit incoherent.
> Run the JS code before and after a commit
How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
> How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
Probably by clicking the button that says run. It is not a weird way to say what they did.
I don't see such a button at all, all I see is text/image pills, samples and a "Commit" button, which the text correctly mentioned, but the other things the text mentions, don't seem to actually exist?
It is weird, a human wouldn't write such text on that page, or if they did, make the text reference actual UI elements, not what they talked about while building it.
There's two sections, an Image one and a Text one. The image one doesn't have a Run button, maybe that's where you are getting confused? The text tab does have it, so you can Run the code shown in the box. The button is at the top right of the element with the code
To share a joke. Whereas by your logic then George Carlin was not a comedian, but a policy wonk who just happened to use the stage and mic to solicit feedback.
Actually, to be fairer to GP - their _git with vibed storage_ project is not a joke, it’s *art*. And the world needs more art.
Right, a joke that is better because it's inconsistent and contains instructions to do stuff the UI doesn't facilitate? I guess I might be overly strict, but I like my tech jokes to make sense, at least internally consistent, and at least seemingly made by humans.
> And the world needs more art.
The world needs more human art, not just "any art created by any manner", that's just slop.
It explains what the thing does, and then there’s also a web demo, where you can run the code and see the reconstruction (which is what would happen when it’s checked out) and run that. You can alter the key cli args it tells you to.
I don’t see any instruction here that it’s telling you to do that you can’t in the demo.
I think you might understand it better if you took a moment to.
The useful thing I actually want to do with this: Zero-latency ssh.
1. Train a smaller (<10M-50M) model. Lower accuracy is fine; I want latency <10ms.
2. Then create a "reactive" version of `ssh` that gives you the perception of zero latency over shitty networks and high latency ssh connections across the world. Basically, the local screen should respond instantly to my keystrokes (including how arrow, tab, etc. esc keys respond in various TUI apps), and then correct itself when the server actually responds.
You should check out mosh! Aside from adding roaming support to ssh, it has instant echo (of course it wouldn't be able to predict a game of snake, but for typing commands and editing files it's pretty good)
One of the first very cool demos I've seen of ChatGPT was in a post where the author prompted AI "pretend to be a terminal". Then went on to execute commands like `cd .fbi`, `ls files`, `cat files/confidential.txt`. Very fun demo. Looking forward to exploring similar idea with this terminal
I think this was right at the beginning, in the GPT-3.5 era still.
I always wanted to put it behind an open SSH server as a "fake" honeypot of sorts, see what attackers and script kiddies do with it. But I guess by now someone's tried that already..
>I used Qwen2.5-Coder-0.5B as the starting point. I needed a small model to get close to realtime performance. Being a coding model, it had also already seen plenty of CLI usage.
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
Very impressive, and a lot more work than it looks from the outside (the DAgger loop especially). I built something similar in June, vibeSH, a shell that hallucinates the whole computer, but from the other direction: no training, just a frontier model and a long prompt, so maybe the comparison is useful.
The granularity is different though. You model keypresses and predict which rows change, I batch keystrokes and have the model repaint the whole screen each round trip. Non interactive cli programs work very good, but interactive cli programs like vim, less and top do work, but it feels like SSH over a 1996 modem, you type a burst, wait, and the screen catches up. Yours is going to be a lot more responsive at that.
The big model buys consistency, for as long as the session fits in the context window. Files come back the way you wrote them, python output is usually right instead of guessed, nested paths stay where they are. The price is an API call per round trip and a cost that grows with the session (and it's not 208 MB and doesnt run offline).
Mine offers an @ai prefix that talks to the director instead of the machine, so mid-session you can say "@ai make this a haunted Solaris machine from 1996" and the filesystem, the motd and the error messages all change to match, and the directives stick.
Your hardest problem and mine are the same one I think. For you it's a 360M model losing track of state, for me it's the context window filling up, after which the box gets amnesia and old files have quietly rewritten themselves. I decided early not to fight that completely and made it canon (the box has bad memory), your DAgger setup is an actual fix for it, which I don't have.
i agree absolutely with you. and this kind of reinforce my idea that, like this predicted terminal could not be considered a real terminal, predicted intelligence could not be considered real intelligence
Thats an very interesting way to think about this, never thought to invert it that way.
I allways found it interesting to look at this issue in the way the imitation game was modeled. For me I concluded, that as long as we can replicate the intution we associate with intelligence, there is not much room left to debate about "real intelligence".
The chinese room experiment I find in that sense a valuable extension, cuz the turing model more or less is a very conversational "voided of action" kind of test. The room model adds the act of translating without having the knowledge of translation.
But your take invalidates my stances here clearly. There is a possible scenario, in which we can replicate the intuition for operating a real terminal but still keeping the act of the terminal itself a "voided" sort of action by itself, cuz well, its not a real terminal, reading real files and so on...
Thank you for this, I will have to chew on that :)
thanks for the great comment! You answer is great too, I will probably think about it for the next 4 nights or more, triggering my wife's comment "why are you so weird and distant? are you cheating on me?"
Like with many things, it depends entirely on how you define "real terminal"; that is, what a "real terminal" must do before it satisfies your definition. If it's "takes input and deterministically executes its instructions, producing output along with side effects," a well-trained model with argmax sampling could well meet the bar. if given the appropriate external interfaces.
generally i expect if i invoke `touch file` to see my file update date as now, on the filesystem. so i want to have a mapping between physical world (in this case, memory on my machine) and intelligence representation. If the LLM terminal, upon invoking `touch file` does not update the physical file, it is not a real terminal. you can argue that i will use the terminal itself to invoke `ls -la` and check the updated file, but verifying from an external scope is mandatory imho otherwise it's obvious you cannot logically demonstrate that. hope i was clear
I remember a couple of these from a few years ago. Long gone now of course but a fun demo of AI hallucinating the internet endlessly, generated pages from URLs that had more URLs in them to navigate to.
I wonder how much a small model could do if it had a tool calling API for a filesystem, an embedded busybox, and a single output token for 'emit what busybox would do here'. Then presumably the model parameters could learn everything that diverges from what busybox can handle.
I remember doing something like this in the ancient era of GPT 3 or 4. I just told it that it was a terminal responding to commands and it was surprisingly consistent.
That seems as likely as anything, but that just moves the problem to counting, doesn't it? The model gets the duration, but I was under the impression LLMs struggled with the arithmetic that would make that useful.
Idk about Qwen 2.5 Coder that this is based on, but modern models don't struggle with arithmetic. I see Qwen 3.8 do long-hand math in thinking traces (ie. it outputs 17+35+12=64 in the thinking trace) so I would imagine this is something they train into the model specifically. Sometimes it creates and runs a simple python script to check its work.
It's still strange to me that an ability to perform arithmetic (and not just memorize a giant table of results) could emerge from training on language.
In principle a large enough neural network can provably approximate any continuous function. Since arithmetic can be represented with a concrete grammar, it's not surprising to me that an LLM could contain an abstract representation if trained correctly.
Quite possibly. Because the models are "eating" the harness --they don't need it to tell them to do multiple steps with planning, implement, review, or to spawn sub tasks, etc. anymore.
I also assume that in some way, in order to produce correct code without having to do a lot of but fixes, the larger models have gained implicit to execution engines or at keast predictors in their weights.
There are also several projects that have trained VMs into layers. Or embedded them in various ways.
There are two major potential advantages if you can really make it work well. One of them is potentially huge speed improvements by iterating inside the accelerator without deciding and running external programs. The other is being self-contained, eliminating the need for most of the harness complexity.
Another reason I think this is likely to become a real thing is that it seems with hardware improvements with flash and within a few years fully in-memory compute, we are going to get much larger, faster and smarter models. So they will be able to more easily absorb the complexity of emulating systems with more fidelity, as well as operate more quickly.
You can also imagine that as models get larger and faster, it is a natural next step in training. The top models now automatically run the software the generate and test it thoroughly. They do not need to be told to do this. By feeding enough of those parts of the sessions back into a large enough model to train on, the system gains the ability to predict the outcome without having to run the code. So it's the core part of a training pipeline already built into recent models.
There also might be some convergence with continuous learning. Because altering weights based on some token context seems like it could be similar to keeping a program resident. And maybe the continuous learning part provides the "compilation" step.
The idea we may have fast continuous learning soon really just worries the shit out of me. I was watching some other clips about models on chips producing 12k tokens/s, it wasn't very good yet, but man once you start getting kind of good at those speeds you can start to do some crazy things.
Cerebras can do like 4000 tokens per second on gpt oss 120b with it's new chip and around 1000 tokens per second on frontier models using its new chip.
Were you talking about Talaas 8b model? They will probably do something like Bonsai or Qwen 27 B next I hope. But it makes might be an old model.
I regretfully suspect this actually is the future of UI. You just need smart local models running fast enough to keep the framerate high, but that’s inevitable in the long term, isn’t it?
We had decades to work on making software more performant, but instead most users' space-age 2nm chips spend most of their time running Javascript because it was faster to make software that way.
I think it's hard to understand how much better your experience could be if all your software was 10x faster and using 0.1x the resources, since you can't test both experiences side by side
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
I just wonder if it points at something more fundamental. We could have always written the code to use less memory, for example embedded code, but unless constrained we don't.
Why would you do that instead of having the model do the task you're using the UI for? It could generate UI widgets on-the-fly as needed to take richer inputs.
It might be fine for one-off applications, but it's incredibly inefficient for something that has to be done over and over.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
Even assuming that the hardware and models catch up, wouldn't there be an issue with the model not being completely deterministic? You don't really want any sort of randomness in your terminal commands.
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.
Thank you for linking that. I distinctly remember this moment as an important point in time, played around with emulating a terminal for hours, couldn’t believe how realistic it was, it even remembered when I changed a file earlier etc. Another point was when people on twitter with early access were like “oh my god it can write code too! I asked it to design a website and it just works!”, programming wasn’t even what LLMs were about, and now it’s one of the main use cases. Another revelation was when it could kinda play chess. A single model that can one shot a huge array of different problems, still mind blown how well all of this works to this day.
I tried it and is not working remotely as close as the demo gif. It keeps telling me that "test1" folder that I want to create already exists (not true), then letting me do "mkdier test2" (sic) but then the folder is not there when ls.
I mean all its expected, this has no state, follow no rules and is making up things from the current context, but is not as "magical" as I thought when watching the demo.
I fry my brain everyday trying to keep pace, but honestly, stop it guys. Give the guy a break.
This is the most magical time in my adult life. The 70s, 80s, and 90s, look like a painting compared to the current pace of things. I am literally tired at the end of each day. Dog tired.
The base model already have a good grasp of language, so that man pages look like coherent sentences. It also knows python and similar language well, which means that things like 1+5 or print(["hi" for _ in range(5)]) work.
I did try to train from scratch, but it was much weaker (also added a lot of aux losses).
Terminal world models have many applications, but empirical observation of successful pretraining or fine-tuning can be significant for encapsulating from rather expensive Bash traces, such as ones obtained from systems with scarce accelerators, etc.
No, the model is simulating the entire TUI, not just terminal commands. It also actually is that slow (because it's generating a diff for the entire screen each time you make a keypress).
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
* web demo: https://nicobrenner.github.io/lossy-git/demo.html
* cli version: https://nicobrenner.github.io/lossy-git/
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
Just to point out your site text appears too small on mobile (no mobile view) and it doesn't have a direct link to the repo. If either is by choice then that's fine of course.
The link to the repo is at the top of the article: https://github.com/nicobrenner/lossy-git
Love it :)
I see a "Commit" button, but I don't see any checkout, I guess that is what happens automatically? Although that panel says "Reconstruction", not "Checkout"... You might want to do a pass after the AI created the initial version, and make it "humanly consistent" or similar, as right now it seems a bit incoherent.
> Run the JS code before and after a commit
How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
Probably by clicking the button that says run. It is not a weird way to say what they did.
I don't see such a button at all, all I see is text/image pills, samples and a "Commit" button, which the text correctly mentioned, but the other things the text mentions, don't seem to actually exist?
It is weird, a human wouldn't write such text on that page, or if they did, make the text reference actual UI elements, not what they talked about while building it.
Actually, to be fairer to GP - their _git with vibed storage_ project is not a joke, it’s *art*. And the world needs more art.
> And the world needs more art.
The world needs more human art, not just "any art created by any manner", that's just slop.
I don’t see any instruction here that it’s telling you to do that you can’t in the demo.
I think you might understand it better if you took a moment to.
So interestingly enough Anthropic has been using models of this kind for their alignment evaluations: https://alignment.anthropic.com/2026/reward-seeker/#:~:text=...
1. Train a smaller (<10M-50M) model. Lower accuracy is fine; I want latency <10ms.
2. Then create a "reactive" version of `ssh` that gives you the perception of zero latency over shitty networks and high latency ssh connections across the world. Basically, the local screen should respond instantly to my keystrokes (including how arrow, tab, etc. esc keys respond in various TUI apps), and then correct itself when the server actually responds.
https://mosh.org/
I always wanted to put it behind an open SSH server as a "fake" honeypot of sorts, see what attackers and script kiddies do with it. But I guess by now someone's tried that already..
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
—Practice, man... Practice...
Please elaborate.
"Building A Virtual Machine inside ChatGPT"
- https://www.engraved.blog/building-a-virtual-machine-inside/
It's mentioned in the "Related Work" section too.
EDIT (comparison)
Very impressive, and a lot more work than it looks from the outside (the DAgger loop especially). I built something similar in June, vibeSH, a shell that hallucinates the whole computer, but from the other direction: no training, just a frontier model and a long prompt, so maybe the comparison is useful.
The granularity is different though. You model keypresses and predict which rows change, I batch keystrokes and have the model repaint the whole screen each round trip. Non interactive cli programs work very good, but interactive cli programs like vim, less and top do work, but it feels like SSH over a 1996 modem, you type a burst, wait, and the screen catches up. Yours is going to be a lot more responsive at that.
The big model buys consistency, for as long as the session fits in the context window. Files come back the way you wrote them, python output is usually right instead of guessed, nested paths stay where they are. The price is an API call per round trip and a cost that grows with the session (and it's not 208 MB and doesnt run offline).
Mine offers an @ai prefix that talks to the director instead of the machine, so mid-session you can say "@ai make this a haunted Solaris machine from 1996" and the filesystem, the motd and the error messages all change to match, and the directives stick.
Your hardest problem and mine are the same one I think. For you it's a 360M model losing track of state, for me it's the context window filling up, after which the box gets amnesia and old files have quietly rewritten themselves. I decided early not to fight that completely and made it canon (the box has bad memory), your DAgger setup is an actual fix for it, which I don't have.
>What's the difference between a model that predicts intelligent output, and intelligence itself?
It doesn't directly attack the question, but gives a neatly constrained thought experiment which you can see running.
I allways found it interesting to look at this issue in the way the imitation game was modeled. For me I concluded, that as long as we can replicate the intution we associate with intelligence, there is not much room left to debate about "real intelligence".
The chinese room experiment I find in that sense a valuable extension, cuz the turing model more or less is a very conversational "voided of action" kind of test. The room model adds the act of translating without having the knowledge of translation.
But your take invalidates my stances here clearly. There is a possible scenario, in which we can replicate the intuition for operating a real terminal but still keeping the act of the terminal itself a "voided" sort of action by itself, cuz well, its not a real terminal, reading real files and so on...
Thank you for this, I will have to chew on that :)
https://www.youtube.com/watch?v=z3pV6FHvcgM
Edit: Maybe this was it—https://youtube.com/watch?v=7NfyZhV1dKM
Maybe instead of simulating a terminal you could use it to predict what you'd type next?
So simulate _you_, not the terminal.
(You could run lynx in that terminal to train/test it without images)
https://github.com/amazon-science/Cyber-Zero
the paper is quite interesting - https://arxiv.org/html/2508.00910v2
High-interaction honeypots are about extracting as much information from the attacker as possible.
Instead of just observing, you could give the model intelligence objectives and have it dynamically steer interactions to achieve them.
Deliberately leading an attacker down paths that reveal their capabilities, tradecraft, or infrastructure.
You can do this today…but it requires effort to develop the scenarios. With this, it’s just a disposable goal.
https://www.microsoft.com/en-us/research/articles/whamm-real...
On thin that would be useful is an entirely new permission system for files to allow for users and agents acting for users.
I did it for web apps last year, basically a web server that feeds all requests into an LLM.
https://www.ohad.com/blog/voidware/
Time is an input too. If nothing is pressed, the model gets a WAIT token with a duration instead of a key, which is how snake keeps moving on its own.
Of course there may be an electronic civilization out there that would say the same things about our mushy brains... "They are made out of meat!"
I also assume that in some way, in order to produce correct code without having to do a lot of but fixes, the larger models have gained implicit to execution engines or at keast predictors in their weights.
There are also several projects that have trained VMs into layers. Or embedded them in various ways.
There are two major potential advantages if you can really make it work well. One of them is potentially huge speed improvements by iterating inside the accelerator without deciding and running external programs. The other is being self-contained, eliminating the need for most of the harness complexity.
Another reason I think this is likely to become a real thing is that it seems with hardware improvements with flash and within a few years fully in-memory compute, we are going to get much larger, faster and smarter models. So they will be able to more easily absorb the complexity of emulating systems with more fidelity, as well as operate more quickly.
You can also imagine that as models get larger and faster, it is a natural next step in training. The top models now automatically run the software the generate and test it thoroughly. They do not need to be told to do this. By feeding enough of those parts of the sessions back into a large enough model to train on, the system gains the ability to predict the outcome without having to run the code. So it's the core part of a training pipeline already built into recent models.
There also might be some convergence with continuous learning. Because altering weights based on some token context seems like it could be similar to keeping a program resident. And maybe the continuous learning part provides the "compilation" step.
Were you talking about Talaas 8b model? They will probably do something like Bonsai or Qwen 27 B next I hope. But it makes might be an old model.
If I had to complain about any site/app it'd be datadog... each tab uses like 1gb in firefox.
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
It’s cheaper to hire developers, and quicker to ship software. The user experience be damned.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
Edit: oh, I saw you wrote a related comment after I posted this. So you already know that direct LLMs for UIs are all but inevitable...
Maybe I do, if by "randomness" you also mean the ability to handle unexpected usecases. Imagine running something like:
$ ps "which processes are slowing my PC down??"
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.
The tmux thing had me confused for a moment until i realized it was emulating tmux. Super fun. Is the model window width aware?
``` BOS SCREEN_START 32 96 ```
I trained it on three different resolutions, but it should kind of work with in between ones too.
This is the most magical time in my adult life. The 70s, 80s, and 90s, look like a painting compared to the current pace of things. I am literally tired at the end of each day. Dog tired.
Great job OP. I mean. This is nuts.
I'm a noob in this, could you enlighten me: what role does the base model play which you started with?
The base model already have a good grasp of language, so that man pages look like coherent sentences. It also knows python and similar language well, which means that things like 1+5 or print(["hi" for _ in range(5)]) work.
I did try to train from scratch, but it was much weaker (also added a lot of aux losses).