Luna is unusable garbage. When I see people in threads talking about how useful Luna is for all of their tasks and it's so cheap and fast, it reads to me like people relating anecdotes about how tasty the garbage scraps they're eating out of the dumpster behind McD's are. Sol Max is the threshold of "just barely usable", and even then it's certainly not what the people are hyping it up to be.
> But GPT also offers a $100 plan with five times the usage limits. And how long would that last me?
Sol usage is basically unlimited on the $100 plan if you're human-in-the-loop, reviewing the code, and not doing some useless Gas Town crime against humanity.
I’ve been slowly converting my org to use opus 5.5 as default over sonnet and the difference is night and day. More usage, because it does shit better the first time. Less cost, because it does shit better the first time. It’s literally been the thing that’s kicked our org in the pants to actually adopt agentic methodology at scale.
I have found in my own benchmarks that a simple prompt which has a single constraint is reliably ignored by Luna, even on Max, even with another Luna agent reviewing with instructions to check the output against the constraint, at which point they OK it anyways. There is not a single task of any scale I can trust it not to fuck up.
It just gets worse as your codebase grows with more code and more files especially. It's like a human developer who is under intense time pressure. If the codebase is large, it might miss an existing function and implement a second one... or it might miss the fact that a database table already exists to solve its current problem and create another one. The solution is to keep the codebase as simple and small as possible and name everything as intuitively as possible with proper separation of concerns. Same principles which were useful before AI are still useful with AI.
I don’t understand how it will ever overcome something like not replicating an existing function. It can’t possibly have full context on a massive codebase at all times so additional prompt runs checking for existing functionality would be necessary.
Even scarier is something like a legacy change made for a specific client, like “our biggest client needs checkboxes as the UI, do not switch it to a dropdown.” was in a commit message 2 years ago, AI will just overwrite that without thinking.
Most of these issues are vastly improved or mitigated by adding clear instructions in AGENTS.md.
I follow a specific process with agents, and it works reasonably well:
1) describe my intended feature or change and then say “please recapitulate my intention and write an implementation plan”
2) read the plan carefully and refine it, make any corrections
3) tell the agent to go, but I check in frequently as it’s going. I watch the stream of thought and make course corrections or pull it out of rabbit holes
4) carefully review the result. Checking that tests actually fail, removing unnecessary changes
5) add instructions in AGENTS.md if I’ve found a new behavior that needs redirection
I've been doing software engineering for 20 years and approximately half of my total work output has been in the last 2 years. Once you get it set up and maintain it regularly it's easily three to ten times faster
I used cursor for a project built on a framework with a huge API that I hadn't studied exhaustively. I used a similar workflow with grok for the planning and composer for the implementation. You have to be vigilant, ask questions about the plan, inspect every line of output, and ask for do-overs. At the end I estimated that I would have needed 5-6 months without the coding agent instead of six weeks. I didn't develop as much knowledge of the API as I would have without the coding agent, but I doubt that my employer would have knowingly accepted a longer schedule in exchange for my education. It's not just junior engineers who are going to miss out with coding agents.
Two to three days of up front planning results in a 100x plus productivity gain. Honestly at this point writing code is no longer a blocker for me, it’s more how many agents can I keep running at the same time, how much time can I spend testing and verifying products, etc…
Unlike the OP, this makes using an agent actually sound useful.
My summary of the article: I use a really cheap LLM, and suffer like everyone else using GPT 3.1. This is definitely a throwback to the days not so long ago when LLMs were completely incompetent.
How people are getting anything done at all with less than 5-6 claude max and 4-5 codex pro max plans and codex-lb and better ccflare juggling weekly limits at all times, I will never know.
You are right and here is my hot take into why working with models is a f pain.
Harness... yes I have one https://github.com/alainnothere/privibe but is not what mine has that the other doesn't but what it doesn't have.
I suspect the whole reason for how bad models accessed through subscriptions tied to a harness of the provider (looking at you copilot) is that microsoft in the case of copilot "builds into the harness and hence into the model" an intention; and I don't have proofs (which qualifies me to start a popular podcast) but I suspect the harness keeps pushing the model to fix an issue as quickly as possible so the service can appear as "it just bam fixed the issue".
The "sitius altius and fortius" of the technobro is what you see in the tv ad "copilot check my git and fix a bug", how stupid do you think someone has to be to believe it works like that? my instructions to model say that I HATE urgency to make changes, we are researching and until we figure out causes, and understand all the mechanism involved AND THEN AND ONLY THEN we can check what's the actual intention, what business wants... then we can talk about a fix.
Grab a harness that injects the minimum possible and let the agent be and you will see how much difference there is in the quality of the model response.
> Inability to change the response style
In recent months, I’ve been noticing a new pattern in sentence structure more and more often. Here are some examples:
There’s a difference between "I read the text" and "I understood the meaning of the text".
We too often mistake "I’m used to this" for "I like this".
The path from "I’m afraid of making a mistake" to "I’m ready to try" can take years.
It’s worth taking the time to setup a style guide for your agent to remove llm-isms like this. Somewhat paradoxically, you can even have the agent help write it. Just include a couple of snippets of “bad” phrasing from previous chats, add some commentary about why it’s bad, and what to replace it with, and put that in your system prompt. I could provide examples but the whole point is that you can tailor it to exactly how you want it. This has the added bonus of being a quick sanity check for model quality. If you start noticing llm-ism appear, then likely the model is not following your instruction’s and the thread should be discarded/you need to switch to a better model
Oh there’s probably some system prompt built in to the harness adding those then. I had to patch the js of the harness I’m using to remove stuff like that from the system prompt to regain full control of the style
Or maybe just luna too dumb idk. I’ve even had good success with a style guide on Bonsai 2 27B though so I doubt luna is dumber than that
some people are not ready to read and accept this but I will say it anyway: stop reading code and trust that it does what you asked to do, focus on high level architecture and human description of what the system is doing. The only exception to this is if what you are doing carries a significant amount of risk that can't be mitigated with a follow up PR, for example launching a rocket or controlling an airplane.
Idk how you are running out of budget so fast. My usage is basically unrestricted tokenmaxxing ("/goal", flagship models at xhigh reasoning, etc) and my monthly burn is single digit thousands of dollars (maybe ~$350/day). Granted, my employer foots the bill so I don't care at all. But the return-per-spend is easily worthwhile and I'm not worried about the cost at these ratios.
Nerves of steel? Not even close.
I think 90% of the article is the author complaining about (predictably) bad small-model behavior. The large models are much better!
> But GPT also offers a $100 plan with five times the usage limits. And how long would that last me?
Sol usage is basically unlimited on the $100 plan if you're human-in-the-loop, reviewing the code, and not doing some useless Gas Town crime against humanity.
What are some uses for which you found Luna to be unusable garbage?
From my side, I use Harbor to score tasks and Luna certainly performs when the prompt is either well or overspecified.
If it's used for that then using anything more powerful is just inefficient usage of your tokens.
Luna medium effort is very slightly better than its output and completes tasks in 5-10x less time.
Even scarier is something like a legacy change made for a specific client, like “our biggest client needs checkboxes as the UI, do not switch it to a dropdown.” was in a commit message 2 years ago, AI will just overwrite that without thinking.
I follow a specific process with agents, and it works reasonably well: 1) describe my intended feature or change and then say “please recapitulate my intention and write an implementation plan” 2) read the plan carefully and refine it, make any corrections 3) tell the agent to go, but I check in frequently as it’s going. I watch the stream of thought and make course corrections or pull it out of rabbit holes 4) carefully review the result. Checking that tests actually fail, removing unnecessary changes 5) add instructions in AGENTS.md if I’ve found a new behavior that needs redirection
My summary of the article: I use a really cheap LLM, and suffer like everyone else using GPT 3.1. This is definitely a throwback to the days not so long ago when LLMs were completely incompetent.
I suspect the whole reason for how bad models accessed through subscriptions tied to a harness of the provider (looking at you copilot) is that microsoft in the case of copilot "builds into the harness and hence into the model" an intention; and I don't have proofs (which qualifies me to start a popular podcast) but I suspect the harness keeps pushing the model to fix an issue as quickly as possible so the service can appear as "it just bam fixed the issue".
The "sitius altius and fortius" of the technobro is what you see in the tv ad "copilot check my git and fix a bug", how stupid do you think someone has to be to believe it works like that? my instructions to model say that I HATE urgency to make changes, we are researching and until we figure out causes, and understand all the mechanism involved AND THEN AND ONLY THEN we can check what's the actual intention, what business wants... then we can talk about a fix.
Grab a harness that injects the minimum possible and let the agent be and you will see how much difference there is in the quality of the model response.
It’s worth taking the time to setup a style guide for your agent to remove llm-isms like this. Somewhat paradoxically, you can even have the agent help write it. Just include a couple of snippets of “bad” phrasing from previous chats, add some commentary about why it’s bad, and what to replace it with, and put that in your system prompt. I could provide examples but the whole point is that you can tailor it to exactly how you want it. This has the added bonus of being a quick sanity check for model quality. If you start noticing llm-ism appear, then likely the model is not following your instruction’s and the thread should be discarded/you need to switch to a better model
> Okay, let’s move the rule to the top. In the next reply, right in the same chat, I get text with quotation marks again.
Or maybe just luna too dumb idk. I’ve even had good success with a style guide on Bonsai 2 27B though so I doubt luna is dumber than that
Nerves of steel? Not even close.
I think 90% of the article is the author complaining about (predictably) bad small-model behavior. The large models are much better!