To try to add something to this discussion -- I think that while I've seen these sort of loops less --- what I have seen is "overly helpful".
Models nowadays want to double-triple-quadruple check things. I'm being silly but it verges on "I have a working solution but let me write a variation in Rust to ensure a convergent solution and prove this works".
I've had to stop models nowadays mostly because they're being agonizingly pedantic in their validation. Opus is actually one of the most pedantic and "off track" here. But again, not in a bad way. I'm usually like "stop testing latency between 50 runs of this app... this is version one.. we're going to make a million more changes.. you're not buying us anything".
"Plus rigorously ensuring backwards compatibility for a project that is 2 hours old and has zero users."
That is exactly how the slop accretes and you get a pile of crap. Claude somehow assumes that said 2 hour old userless app is some dusty enterprise app with millions of users and billions of dollars at stake for a 1 second outage.
I have to constantly have these things "take a deep breath, step back and look at the entire thing and do this change holistically. please restate what i'm asking you to do and why it's important"
Which, again, if you attempt to use these LLMs in an actual enterprise project with the assorted legacy mess in it, that does have to have backwards compatiblity, and interacts with 'weird' tech (well, weird to full stack Node/React devs), well it's an exercise in frustration.
> please restate what i'm asking you to do and why it's important"
This doesn't actually do anything though, right? There's no understanding, so the machine will just reiterate the original token query back to you. The 'why it's important' part will just generate some patronizing boilerplate as a raison d'etre.
Yeah of 10 minutes ago. It is shocking how long some seemingly simple things can take. I know there are some things I can do faster than the LLM and some things it can do faster than me. The amount of rambling BS is the exhausting part.
Try different models, it's a breath of fresh air. GLM 5.2, etc. all make life much more enjoyable. They may not one-shot a complex project the same way that Claude can spit out memorized architectures, but that sort of system is always only useful for a one-off prototype anyway, so not much is lost.
Edit: for how to do this, I set up an OpenRouter account so that I could easily switch models, and then ran them in Pi inside of Orca ADE. Orca lets me easily switch from Pi to Claude Code to Kilo to Codex or Hermes or whatever. Pi+OpenRouter lets me easily switch the LLM. All of it lives in a single open source orchestrator to avoid any platform lock in to any AI company going forward, and can even do local LLM should I care to take on that massive hassle.
I've noticed them repeatedly casting the same value to the same type for now reason, like I'd have a Python function with a type-annotated int argument, and inside the function it would cast that int to int, and also at the call site, just in case it wasn't int enough.
This is my recent experience as well. Models want to run linters, tests, etc. And that is all covered in GitHub actions. So I have been instructing agents to push a draft PR, then I validate the static checks pass and tell the agent if there are issues.
Agents and AI are getting expensive, it seems silly to waste tokens on static checks.
Claude models were always too eager and "overly helpful" for my taste. But it seems better models tend to be this way. GPT 6 and 5.6 are overly helpful too, but at least less than Fable. But I seem to be sticking to GPT 5.5 as this was a really focused model.
I blame the hidden context on the tools/subagents. One recent example.. codex can just look in the code for Db schema but continually tries to request permission for a live db query
Hacker News is the epitome of this! If you find yourself not enjoying your daily dose of "someone is wrong on the internet" (via the immortal https://xkcd.com/386/) as you find yourself crafting the perfect response, you can always close the tab!
and ironically, most of them don't even offer the draft feature! Facebook, tiktok, etc... they send your typed message, but if the app crashes or you close it, there's no draft anywhere you can find ;)
You can, but this is public and maybe you should post. If "you are what you repeatedly do" then we all collectively are what we all do.
If we each stop arguing with racists because we're each tired of it, then what we are is a society that lets racism go unopposed, a site with racist comments proliferating, a site where racists gather because they are able to be themselves, a site where future readers and AI training is learning that racism is popular, common, normal, and acceptable because everyone is accepting it.
It's a bit like democratic voting, your vote doesn't matter and cannot change things, but it's important that you vote because all of our votes do matter and can change things.
It's a bit like the quote "Dear Board, I don't want to belong to any club that would have me as a member. Sincerely yours, Groucho Marx.", I am upset and offended every time someone calls out my dumbass comments, but horrified at all the countless times nobody does. Why would I want to waste my time on a site with such low standards that it lets me comment? I'm exhalted and encouraged everytime someone upvotes or engages with my low effort quips[1] but thrown into a chasm of despair that HN users let that pass unopposed. When nobody engages with comments I spent literal hours on[2] I curl up into a ball and die, but also dream of it being a feather in my cap with dang that I'm posting a substantial comment and that will be its own reward. I identify with nobody more closely than Colonel Cathcart, and I will be delighted if someone engages with my Catch 22 reference, but aghast that such a nontechnical reference gets engagment; distraught if nobody engages but delighted that perhaps HN standards are being held higher.
You’re absolutely right, but I need to go to the bathroom/get some coffee/take a smoke break/get back to this task now so we’ll have to discuss this further later.
I don't know it has as much to do with scale of the system vs the general architecture. E.g. the system I primarily work with these days has millions of lines but most PRs are for a small changes which are well contained in scope by the overall architecture.
reasonably you should already know what the code and project is supposed to do before they even start writing code, so you can get directional feedback in.
then you are maybe reviewing 1 out of 7 PRs that implement the agreed upon change
It's a cultural thing, but you can do incremental PRs towards a large goal. Giant PRs that are expected to be reviewed never really seemed worth it imho.
At least with Codex, this has not been my experience at all. It still screws up sure, but in every case I can ask "why did you do this" and it can trace back what made it take that particular decision. Typically it's always that I either didn't specify the problem correctly or made a really dumb mistake (executing the task on the wrong project....did this one yesterday) or it's something within a skill file that instructs it (at which point I fixup the instructions).
Once in a blue moon it's actually the model making a material error in it's thinking and I have to go back and redo it.
Yes. But although they can't know "why" a specific "wrong" answer was selected, the response is often still informative, and it can highlight real weaknesses in process or code structure that should be addressed anyway.
AFAIK this really is true. I've seen some videos about patients who had the connecting part between the left and right halves of the brain cut as a (archaic) treatment for epilepsy.
While it did help the epilepsy, their brain was essentially two brains controlling two halves of the body. With one controlling speech. There were experiments where one eye was shown some instruction text, the corresponding hand performed that instruction, and when asked why they dix that action, the speaking half just made up some plausible, yet completely wrong reason, just like an LLM.
When we are really thinking about something we do it forwards, backwards and middle out, and regenerate and distill many times.
When we do meta thinking about that process after the fact, two things happen. 1, we change our total “thought” by adding that meta thinking to it. And 2: it’s a very lossy process, because we don’t have very good data about what our brain or mind was actually doing during that first think and emotional factors are nearly always at play and even more complex.
Now for the more complex AI, the fragmented process of multiple agents and loops and reruns are pretty similar to that first think we do. At least structurally. But the meta think is where they differ. They have no emotion, but they also have even worse data about its own function. They constantly degenerate so I would argue their “changing the thought by thinking about it” factor is also generally way higher than ours.
Getting better at consistent/reproducible thinking, with many ‘steps’, that leaves good documentation of that thinking behind for future analysis, has to be one of the more important areas for the big flagships going forward. I’m certain that “what is this fucker doing and why” is the biggest pain point for AI researchers. Or the math, it’s usually the math.
But you’re correct in the general structure; they generally do the same post hoc analysis we do, just noticeably worse because of their opaque nature(even to themselves) and general degenerative instability.
But is it really what we want, machines with the same defects as humans? I don't want a pocket calculator that make mistakes "sometimes" so I have to double-check the results, I want a pocket calculator that works (to those who want to argue that pocket calculators don't give the correct result for (1/3)*3: STFU).
>But is it really what we want, machines with the same defects as humans?
Sort of, actually. I think we humans actually have some intuition that we'd be more effective if our cognition were augmented more directly by machine strengths: the ability to run precise calculations, more memory, ability to look facts in some sort of knowledge graph.
I think we're on the right track, but instead of augmenting humans with machine strengths, we're building intelligence in hardware in a way where it can access that augmentation. Plus, then we can quickly distribute updates, run parallel instances, etc.
If intelligence is compression, and hallucinations are essentially loss, then as the models grow in size performance (at least as far as hallucinations) should reduce. Or we'll get things fast enough that we can afford to stop relying on model weights for memory and check an increasingly larger set of discrete facts as part of reasoning.
Right now, the models are making trade-offs. As compute grows, and inference gets faster, we can make fewer of those trade-offs and start to use the unique strengths of machines to fill the gaps we're seeing, I suspect.
One of the biggest strengths of a computer is reproducibility. The worst software bugs are inconsistent or non reproducible. The least useful calculators apply rules inconsistently, to your example.
the inconsistency of LLMs is by far one of the biggest gripes I have with them. Closely related to their apparently deep desire to avoid following instructions.
I know these are both a byproduct of noise (which is somewhat tunable) and noise is inherent to these systems in a lode bearing way.
I still hate it. it’s holding the technology back. I don’t honestly see how we can safely or even successfully approach the idealized realm of AI without bypassing this problem, which to my understanding, probably means not using language models at all and trying a totally different approach. But I really don’t know much about machine learning, I’m a super novice compared to a lot on this website.
No. People have an inner monologue, partial results and ideas and they remember that.
If they've worked some minutes/hours/weeks on something and you ask them why did they do that, they will either answer honestly and truthfully, lie, or say "I missed that/didn't seem important so I just chose something at random".
If you have examples of studies that verify that people are always aware of the gaps in their memories of why they did things rather than their memories sometimes "filling in the blanks", so to speak, I'd be interested. My impression is that the opposite has a lot more evidence in studies (e.g. around the reliability of eyewitness testimony).
It's not clear to me whether you're aware of a rigorous basis for your claim or you're just inferring based on what you think makes sense, but I can't help but wonder if it's the later, in which case regardless of the mechanism, the outcome certainly seems to resemble what happens with LLMs.
Perhaps up to 50% of people actually don't have an inner monologue, much like many have aphantasia where they can't actually see anything in their mind either.
That’s not how that works. I don’t usually have an inner monologue either, but I do have an abstract stream of thought. It’s not as if I am always acting on instinct.
>No. People have an inner monologue, partial results and ideas and they remember that.
Yes, but the vast, vast majority of decisions you make either don't take place via an inner monologue, or include details that were not actively/consciously "thought" and reasoned with in your inner monologue.
And yet, when asked why you did something, you're not likely to respond "sorry, that decision was made subconsciously". Instead, you use your inner monologue to try to backfill in a reason why. That reason may be correct, or it may not be. You don't actually know, since you have new data that may be updating your own internal state as you try to rationalize it after the fact.
I think this is true with some people, but I don't think this holds true for some (or even most) people across the US (at least not all the folks I've worked with)
Actually split brain experiments tells a different story. The left hemisphere actively confabulates, inventing plausible explanations for actions it didn’t initiate, suggesting that much of human self-narrative may be post-hoc storytelling.
> The left hemisphere actively confabulates, inventing plausible explanations for actions it didn’t initiate, suggesting that much of human self-narrative may be post-hoc storytelling.
Sounds like the left hemisphere usually uses something from the right hemisphere to answer those questions and it can't do that if it's been cut off?
We know that our brains are capable of hallucinating due to substances (drugs), being asleep, brain damage (including split brain), hypnosis, etc. Just as RAM damage make your computer do weird shit. That doesn't mean it operates that way normally.
You can't just remove a huge part of a system and then assume that the whole system behaves the same.
That was my experience with Claude when my vibe-coded project was small.
But now that I've been working on it a month and there's a lot of documentation, it's pretty clearly ignoring parts of the documentation and parts of the code. It will come up with some ridiculous statement about how something works, and I'll challenge it, and it'll admit I'm right.
It definitely reads more documentation than any programmer I've ever worked with (myself included) but because it doesn't have a memory other than the documentation, it still makes mistakes like that.
I haven't turned on "memory" or tried it with Codex, so I don't know how that'll change soon, though.
Yeah the biggest task these days that I do manually is curating the documentation. AGENTS.md in every major directory, and a variety of reference docs that are explicitly referenced in those files.
# See DOC-ITEM-NAME
DOC-ITEM-NAME.md
When referencing documents, always use the exact syntax See <TAG> - this is enforced by a lint on precommit
And those doc items are basically all of the values, architectural, strategic, and tactical items. It's a poor man's in-repo RAG but it's shockingly effective, especially if you keep them small. I may migrate some/all of them to skills over time, but I usually update them biweekly, and I only allow agents to make small edits or propose new notes. And typically I go through and delete or curate any agent edits before merge.
Depending on language I've seen this scale past multiple millions of lines of code, as long as you pair it with all of the linting and tooling that you can possibly build.
People do both. Some choices aren't worth the time and effort of detailed analysis and contemplation and some are basically instinctual, but there are plenty of times that choices are carefully considered and well reasoned before being made and acted on.
Every time I hear someone complain about hallucinations, I laugh at the total lack of self awareness about our species. Humans are just as bad (now, probably worse) at telling the truth, whether due to intention or poor memory.
To test this, change the history in the context to indicate that the model did or recommended something completely different than it actually did, and then ask it to explain why. You’ll still get a plausible explanation.
True, but even a hallucinated explanation of where things went wrong added to the context can force the model down a better path over the next few inputs.
You can also literally tell them: "Here is your session ID: $ID, lookup the .jsonl session, trace exactly why this decision was being made, present evidence and concrete proof, no guessing or assumptions" and you'll get an evidence-based report without guesses.
It can always hallucinate said report results/evidence/proof just the same. This approach tends to help reduce the hallucination rate though.
You can extend this further by using an adversarial agent trying to find mistakes in the other instance's logs in a loop where a 3rd neutral agent weighs the claims of the other two. This is also just another step in reducing error, it does not guarantee elimination of such errors. The latter is an impossible guarantee, even for humans.
Ask it to build you a simple and deterministic citation checking extension to your IDE that puts source in meta to citations. E.g. color citations green/red depending if they are valid.
Sure, you can always validate what it's saying yourself at any point & you can have it try to make manual verification an easier process to complete via methods such as the above.
agreed to some extent. I think this parody still highlights what I feel is often the experience. It might not happen on a simple task such as changing a button color, but on more complicated things, this can definitely be exactly what it feels like.
Codex has the opposite problem. Instead of being overly proactive it's overly reticent. I have been preferring it lately, although my preference tend to switch every few months when a model or harness regresses horribly.
It's just a lot faster at hammering it out than me pound for pound, and I can quickly rattle off via voice-to-text exactly what I want much faster than I can type all of the code (especially when across a few different files), in a huge majority of tasks I perform. It's also especially good at debugging by brute force quickly and at scale meaning e.g. it can start desperately bisecting diffs to find the source of a bug 10000% faster than I can.
For me, typing "Create a new namespace with these enums, functions and traits, that should follow X, Y and Z constraints" is faster than typing all that code manually, and typing less is less straining on my hands/fingers.
The game is fun because it's so obvious that any answer will just devolve into an even more unstable state when in reality I feel like it's pretty straight forward to correct it in the moment, if not permanently, to get what you actually need.
Yesterday, with Astra Max, a very clear instruction to "remove the GUI editor pane and add <another component> to the existing sidebar" for a prototype I had it working on resulted in it deleting literally the entire GUI and building a new one from scratch, including the requested component and losing almost all other functionality of the application.
This is fucking constant. I can't deny that this stupid tech saves time prototyping even with having to wrangle it, but it commits a fireable offense several times a day that no human would get away with and is obviously incapable of learning from mistakes in the way a human is. The only reason it's not fired is because it's a slave that works for no more than the cost to feed it.
I see this sentiment pretty regularly, and I don’t get it. Variable rewards is not sufficient to establish that it is “ basically gambling”.
Everything in life is variable reward. You invite a friend over, they might accept or they might not. Drive to work, traffic might be good or might be bad. You ask a colleague to finish a task, they might do it or might not or might do a good job or might not.
Everything is variable reward. Is everything gambling?
> "Everything is variable reward. Is everything gambling?"
well, no. If you work overtime and get paid overtime, you are not gambling and that is not a variable reward.
Humans engage more with rewards that are intermittent and variable. Like Futurama's scene from 'The Scary Door' where the character says "A casino where I'm winning, I must be in heaven! A casino where I always win, that's boring, I must really be IN HELL!". A constant predictable reward is boring, less engaging. So if you know you get no overtime, but sometimes your boss rewards you with $5 coffee voucher, sometimes a free pizza dinner, sometimes double-time pay for the time worked or a half-day off, now you might be gambling 1hr overtime for an intermittent variable reward.
> "Drive to work, traffic might be good or might be bad."
Good traffic is not a "reward" for driving to work(!) and you have to drive to work regardless so you are not risking anything [you might be risking your life, but you are not making a choice which can reward you with good traffic]. You might say that going a different route is a choice and a gamble which could reward you with good traffic, but traffic engineering does not work that way because if there was a consistently low-traffic route, everyone else would take that route until it was no faster than any other route. Traffic will generally be the predictable and similar every day, plus 'arriving at work early' is not much of a reward.
> A constant predictable reward is boring, less engaging.
Perhaps but predictable outcome is a very desirable quality. No one wants a hammer that sometimes drives nails and sometimes doesn’t. All of the current harness engineering work is about squeezing predictability out of the LLM.
> Good traffic is not a "reward" for driving to work(!)
Like hell it’s not. I drove into work last Friday and there was no traffic because of the holiday weekend. It was amazing. Had me considering whether Friday should be one of my standard RTO days.
You invite a friend over, but raccoon appears. Then pigeon appears. Then friend appears but at the last second suddenly becomes a banana. You remember you are out of bananas so you order more and also some cola zero cans on your local grocery delivery app. You are back to the party, but now you have 5 friends in the room, and you run de-duplication query. Now half of your friend is sitting at the sofa, and another half becomes a quarter of banana. Suddenly bananas arrive so you need to open the door. Once you are back there are no friends, pigeons or raccoons but also no bananas and no cola - all the delivery results are gone. This seems to be urgent and important, gotta fix this first before going back to that friend invitation...
This is not my experience with current AI models at all. But regardless you are not describing anything that sounds like gambling. You are describing a weird hallucinogenic experience.
i’m not sure “variable rewards” is the right term, but i do agree with the op that it is very similar to gambling.
regarding your examples, i think the difference is that with ai, you’re literally sitting in front of a machine, pressing a button, and (almost instantly) getting a result that, if not desired, can immediately be tried for again. you even spend “tokens” to do this, and at least in my native language, “token” brings to mind the coins you’d stick in a slot machine
I don’t see much similarity beyond the most superficial.
If you sit at a slot machine and pump quarters into it, each “turn” is independent. You spin and you win or lose. It’s pure chance and there is no destination. You execute the exact same action over and over and hope random chance brings you more money.
If you sit down in front of a coding harness, the progress is incremental and directed. You ask for a thing, the LLM produces something that is hopefully close to what you wanted. You give it more direction to prod it closer to the end state you want. You are not executing the same action, but incrementally nudging it in the right direction. I’ve literally never restarted from the same initial state with the same prompt and hoped for a different result and I don’t know why anyone would. Rarely I’ve thrown away the progress made and started over but always with a very different prompt that includes learnings from the failed attempt.
i agree with you that it’s principally different from a slot machine, and that it’s possible to use it in a way (like you describe) that is much more focused, for lack of a better term, to great effect
most people don’t use ai this way though, and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it (and this is even more obvious with image generation as you chase that perfect output)
perhaps it’s better to compare it to gacha than slots?
How do they use it? Surely no one is just repeating the same prompt over and over (except as a Ralph loop perhaps, which is automated). I’m really struggling with the notion that most people just throw the same prompt repeatedly hoping it eventually works. Because that doesn’t sound like gambling. It sounds crazy (and frustrating).
> and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it
In the sense that you get a dopamine reward when you succeed, sure, but I get the same reward when I code by hand and achieve a successful result.
> and this is even more obvious with image generation as you chase that perfect output
This is fair, because sometimes with image generation the same exact prompt will produce very different output. This is becoming less true as the models get better and it becomes more effective to direct image generation iteratively than to keep starting from scratch with a barely tweaked prompt.
When working on a problem models will walk you down a garden path requiring only yes/no answers or very brief clarifications for a long time. It's always proposing its own workarounds/suggestions/etc. Especially if trying to debug something where it has more understanding than the user so really all it needs from you is "uh sure try that too I guess" from time to time.
And sometimes the debugging has already veered completely off course at the beginning so it's futile, but each time the fleeting hope that just a few thousand more tokens will magically fix it tempts you to keep going a little longer.
as an example, i was using chatgpt a few weeks back to help me remember the name of a painting i’d seen about a decade ago. i could recall the general shape of the subject and that it was europeanish, but nothing else. after seven turns or so it finally got it, and honestly, the relief of finally remembering the name felt like, well, hitting a jackpot
i’ve had a similar feeling of success after trying to get it to give a comprehensible answer when asking it for a solid counter-argument to philosophical questions. it is indeed often crazy and frustrating
> but I get the same reward when I code by hand
i have only done very simple coding work with llms, so that may be why our ideas differ about the feeling of reward. this is where the comparison to gacha makes more sense than slots, since when you’re coding, you still get a reward each turn whilst chasing the final/desired result
> the relief of finally remembering the name felt like, well, hitting a jackpot
I understand the joy of success but I fail to see how this is gambling. I could have an equivalent conversation with a friend (more likely about a movie in trying to remember than a painting, but still) and get the exact same type of iterative “no, not that one, it was more like X” and feel elated when my friend finally realizes I’m taking about a scene from Hot Tub Time Machine.
I would agree that 1-2 years ago models were more "slot machine"-esque - sometimes the output was good, sometimes the output was bad. And as a result, I primarily used them for auto-complete functionality and bouncing ideas around. In those workflows, you can easily ignore it if the spin is wrong.
Not everyone has the desire to work around the system, and many are diametrically opposed to the concept of AI. They get this perception that it's a slot machine because of that inconsistency, and then do the human thing of assuming that other people must just be flawed if they're different from them. They're "addicted to gambling".
Obviously, things have changed. Open models can still be like that, but are often so fast and cheap at iterating it doesn't matter. SOTA models aren't perfect, but are to the point that they're generally much better than the average developer.
But once that perception set in and the meme spreads, it's really hard for some to break out of it. Especially at the pace AI development has been moving. It's just that simple.
Programming before AI was always variable reward. It was a gamble against your own time and patience. Maybe I'd waste hours down the wrong rabbit holes trying to find a library that worked for my use case. Maybe I'd waste a day trying to get an API to do something it turned out it couldn't do. Maybe I'd have to redo my entire approach because of some factor I hadn't considered. Something I wrote could have worked on the first try or I could have had to spend the day chasing logic errors (or multiple days chasing memory errors if it was C or C++). Maybe I would just get bored of the project, especially if I realized there were 20 layers of yaks I needed to shave first, and Visual Studio got stuck updating again, and before I could even start actually coding I had to spend the entire evening on an exhausting merge conflict. My entire weekend could be gone with nothing to actually show for it.
I got so sick of all this at some point that I slowly stopped doing anything that wasn't my job. But then AI got better and better and I realized it was the ultimate unblocker. When that dreaded malaise started creeping in signaling it was a project's end because I didn't want to waste any more of my life dealing with bullshit orthogonal to what I was trying to do, I'd give it to the AI. It felt like a miracle the first time this worked, and it still does. If we were previously equipped with shovels to dig through bullshit, we now have a fully automated Bagger 288.
The reward schedule now isn't variable anymore; the chance that I finish something in a good state is 100%. I can focus on the parts I actually enjoy - architecting the broader system, making the parts mesh together in a sensible way that's easy to work with and has some mathematical elegance to it, hand coding the bits I want to be really specific about (but now without the endless frustration of bugfixing or import errors and edgecases being immediately discovered, thanks to the AI).
>Maybe I'd waste a day trying to get an API to do something it turned out it couldn't do.
I was working on a side project recently. I had spent months designing the data model in my spare time, thinking through how to make it as elegant and durable to change as possible in the long term, since (if I launched it) the repercussions for getting it wrong would be significant.
Once I had a working design, it probably would have been several more months to build a working prototype and start testing it.
Instead, Claude knocked out the prototype for me in an afternoon. And it immediately became clear that it didn't work: not because the data model didn't solve all the problems I wanted it to solve, but because it didn't fit the shape of how I quickly learned a normal person would need/want to interact with the product. I was so focused on the long term, that I never thought about what the first five minutes of a user with hands on the thing would need. And the changes needed would be significant.
Maybe there's some variable reward mechanism. But I sure was glad to be able to pull that particular slot machine handle and learn that than waste even more of my time on what was a dead end.
> It was a gamble against your own time and patience.
At this point, what do the words even mean? Your own patience and available time are always completely random and fairly distributed across a large enough data set?
> Maybe I'd waste hours down the wrong rabbit holes trying to find a library that worked for my use case. Maybe I'd waste a day trying to get an API to do something it turned out it couldn't do. Maybe I'd have to redo my entire approach because of some factor I hadn't considered.
Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
> At this point, what do the words even mean? Your own patience and available time are always completely random and fairly distributed across a large enough data set?
Predicting the time a task will take is impossible. Something that sounds like a 5 minute script can turn into a month of banging your head against unknown unknowns. I lose my patience when the afternoon I allocated is getting overrun by nonsense and I'm missing out on other things I wanted to do or household maintenance.
> Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
Every thought we have has random chance to be wrong despite our conviction that it's correct. Descartes' Evil Demon plays his tricks on all of us. How many times have you typed some line of code only to realize it was obviously wrong afterwards? Even for simpler matters we "hallucinate" all the time. I was deep in thought trying to help someone come up with an acronym the other day and felt convicted that "Goal Oriented Augmented Retrieval" worked for GOAL until I said it aloud.
Our thoughts and actions are consistently wrong some portion of the time because our meat computers are not perfect positronic brains running prolog. We put cereal in the fridge and say "you too" to the waiter. Every thought we put down or action we take is a gamble on the soundness of the thought or action.
Modern LLM services are engineer's pipe dream that was heavily shaped by the shadiest product management dark patterns you can find: applying gambling-style engagement tactics, exploiting cognitive biases, exploiting users' lack of technical understanding to inflate product expectations, using fear mongering in external and investor communications. And that's not even a complete list.
Yeah, if this is how people interact with claude I’m not surprised they’re having a bad time in ways that I don’t. Asking it why it did something or getting combative is a waste of time.
Realise that you are talking to some mathematics in a box. You can't get a rise out of it. It cannot feel guilt or remorse. Whatever emotional payoff one might want from "I asked for ONE button" cannot be had here. Meanwhile, not only are you being charged by the word, but every word you send that is not directly on the path to getting what you want done is just noise in the maths getting in your way.
So leave emotion at the door and make your words count. Voice your frustrations at the stress doll next to your monitor, sure, but spending tokens on them us just wasting time and money. Explicitly typing out your exasperation will not get you closer to whatever it is you are trying to accomplish. What can you type that will?
Laser focus: clearly, concisely explain what is wrong right now, and how you want it solved. Then do that again until all the moles have been whacked. If the AI is stuck in a sycophancy loop, start a new clean session. Do this frequently anyway: every turn in the same session charges for all preceding conversation again /and/ degrades the LLM's performance.
My best takeaway from the site was: Every time you see Claude Code / Codex doing something you don't want, ask it to add to AGENTS.md. Specially if they churn a lot doing something, and eventually find a way. I ask it to store the way it made something work. It's a constant gardening of AGENTS.md and it has really working well.
Yeah that's when the site lost me too. I feel like people just tell AI "make the thing" and then get mad when it doesn't match up to their vision that they didn't specify at all.
If someone is that frustrated with AI, do they not just … switch to not using AI and doing it by hand, asking simpler next-step questions and hand-coding, instead?
But it is futile to get frustrated or annoyed by an LLM. I do get frustrated, too, but I do not "yell" at it hoping that it will miraculously do what I want. GPT (again, free tier, so no wonder) got me frustrated too because I felt like it just would not listen, no matter how specific I was, so yeah I did experience what the author intended to show.
But it’s *wildly* unhelpful. Would you talk to your coworker like that? “Half the site is blue” or would you laugh and go, “Uh, whoops, now both buttons are blue, just wanted the ‘Add to Cart’ button.”
Either that, or your coworker themselves would laugh and tell you about it later.
That’s also where this quiz lost me. I wouldn’t respond in either way, I’d say “all buttons look blue now. Can we make it so that just the Add to Cart button is blue? I’m okay with Add to Cart having its own class to make it easier.” or something.
FYI I actually agree, it also surprised me. Anyway everything is deterministic on the website so it's not like that prompt actually has an influence on anything.
But its not even good satire, because its totally unrepresentative of my and most others' lived experience. Its similar to making a joke about a calculator misadding two numbers because a stray beam of solar radiation flipped a bit.
Agreed, this site isn't reflective of any of my experience with Claude. It does what I ask it to do, and when it doesn't get it right, it generally turns out there's a good reason for it, which is any of the 100 reasons a human doesn't always get code fixes right on the first try either.
I do remember that one of the first things I did with my CLAUDE.md was to tell it to stick to the scope of the task, never to jump ahead and do extra "helpful" things without confirming with me first, and to follow best software practices including around refactoring but also to specifically avoid overengineering. I don't know if that is what's giving me a different experience from whatever the author seems to be "satirizing".
Aye, and amid all of the (admittedly annoying) word salad, the "Claude" in this satire identified the reason the requested change was hard: someone(s) at some point had hijacked the button markup for other purposes. Un-doing all of that kind of mess is never a trivial change, and probably requires someone with more coding expertise / knowledge of the code than the given prompts reveal the putative user to be.
AI isn't magic, and it won't (or, at least, doesn't yet) enable anyone to do All The Things.
I think this really was a relevant issue around a year ago, give or take, maybe a year and a half now.
But with current models you actively have to sabotage the context to get this kind of behavior, or dramatically underspecify (3 words versus 2-3 sentences)
Have you ever considered that "well, it works on MY machine!" is perhaps not the most helpful response to people voicing frustration at something that is not working as they expect for them?
I don’t know where you picked up the idea that my intention was to be helpful. I’m only mirroring the intentionality behind whoever created this site; they obviously also had no intention of being helpful, as is the case with many discussions concerning AIs limitations.
I understand your frustration, it can be hard to hear that other people's experience of a technology is so different from your own that you cannot relate.
I have misbehaved in this fashion for many people across the full spectrum from casual users to highly experienced software engineers with millions of social media followers, so the statement that it's similar to making a joke about a calculator misadding two numbers because a stray beam of solar radiation flipped a bit at least for my part is not true.
Would you like me to start using bad English and doing things you never asked me to for your sessions, too? Just say the word.
because you are now getting coded products written in large by people who do not have technical foundations, so the way they interact with and even prompt the model is different. We all know how to fix this scenario; be more specific, or diagnose the abstraction mix ups and straighten those out.
Ask for change A and get unwanted change B happens all the time with bad programmers and tradgedy of the commons (ie poorly architected, no restraint) codebases.
same. none of the available prompts are what I would prompt claude with and I get way better results than this. makes sense to me why the provided prompts result in the simulated outcomes. garbage in, garbage out.
Part skeptic and part zetetic, what prompt would you have used?
My impression is that this is an oversimplified demonstration of what can happen when you prompt Claude in a system with many more variables (than two buttons and two colours).
If I want the button to turn blue and that's it, what instead do I ask? Even in a complicated system with many levers, what do I request other than the desired end result, hoping that Claude pulls the right levers to produce something acceptably close to what I think I asked for?
"Make the Add to Cart button blue" is fine as a first step. When that doesn't have the desired result, I would simply revert the change, start a new session, and do something like "Give the Add to Cart button its own theme separate from the default button theme. Make the color for this new theme blue.". If this gives almost the correct result (e.g. it's not the shade of blue I want) I'd do a followup like "change the Add to Cart theme from #00FF00 to #16b8c4" with no implication that the previous step was wrong. If it's wildly incorrect I'd start over and try to specify in more detail about what exactly should happen.
If I don't understand why changing one button changed them all, I would use an entirely separate session to ask questions about how the site theming works. The fact that the site has a bunch of weird coupling between themes is useful information, and if I don't know how to resolve that I'd ask Claude for ideas about how to safely eliminate the coupling, and once it proposes a reasonable idea tell it to implement that.
The two big things here I'd never do is use emotional languages in prompts, and I'd never tell Claude to revert changes and try again. Once the incorrect change is in the context it's poisoning all of your future results.
I dunno, but in my experience Claude looks at all the levers in your codebase, understands what they do, and then generally figures out how to achieve the desired end result.
And when it doesn't, it's usually because of things outside of the codebase -- iOS layout quirks that aren't documented, buggy Python libraries it's relying on where you then have to tell it to read the source to figure out what's going on, that kind of thing.
The other thing that I think a lot of people run into is that they launch it into a completely human-built system with hundreds of thousands of lines of code and expect run #1 to be perfect.
You have to change the system so that the AI understands it, via establishing what your beliefs are, how those are reflected in values (especially important if you have e.g. compliance needs), how those values are reflected in the operational and strategic levels, and then a variety of tactical behavior coaching. For example, I ban 2>/dev/null - super tactical, and I say I value simplicity over covering every edge case - a very broad generalization.
It "makes sense" that plainly telling the chat bot to make one button blue makes the whole site blue? It "makes sense" that correcting it and repeating 3 times that no, only the one button should be blue, should make the button a gradient and start a philosophical discussion around the nature of "blue"?
Wow, you AI people really have a negative view of the technology y'all are trying to sell as the next Jesus
If the agent's change has such a catastrophic effect, the first thing you do is tell it to explain why its change had that effect.
Once you understand what the problem is, you can give it better instructions. If the architecture is shit, the agent is going to have a rough time of it.
So how would you prompt it instead? Because that's exactly how I prompt it, because any reasonable human being would know exactly what I mean by "Make the shopping button blue", and I am sick and tired of getting shitty results.
The first prompt is fine, it's the following ones that are poor.
> Why is half the site blue now? I asked you to change one button.
> Half the site is blue. I asked for ONE button
Neither of these is an instruction to fix the problem, they're treating the AI like a person and telling it what it did wrong, expecting the implied admonishment to be enough to steer it back. But without an actual instruction, it just goes and does whatever it thinks will help, which is often arbitrary.
The response I would have used in this situation is
"The Cancel button is also blue now. Make sure the color change is only scoped to the Add to Cart button"
Most of the available responses throughout this "skit" are similar cases of expressing frustration first and guiding the result second.
Skip the emotion and say exactly what you want, and nothing besides that.
Of course it's not THAT bad, but you can't tell me that you didn't recognize many of the funny turns and twists of the "story". It gets more convincing if you deliberately pick the ambiguous versions.
This is EXACTLY my experience with it. Ask for it to change one line of code and see it invoke 20 tools and burn 300K tokens before updating the line and adding 50 extra lines of JavaDoc.
It's a joke like the endless conservative dudes doing the "ordering coffee" joke is. It relies upon the ignorance of the viewer -- which is usually a fair assumption -- and basically that your understanding of something is based upon the prior accrued layers of "jokes".
Claude is great, but I have come to really hate the way it "talks". It's so irritating and there seems to be no way to make it speak normal English. So many claudisms in every response
> Worth naming: the Add to cart button is still black.
Got an audible guffaw out of me. This really is what the experience is like sometimes if you're just giving it a result without being specific in implementation, and it comes out of nowhere, some days much worse than others.
I've become patient with it, but whatever this style of output is called or doing - it is both condescending and entirely unhelpful, and it seems designed to frustrate.
Haha this is spot on how I've been feeling lately. I find it unbearable to work with this model for this reason... any trick out there you can do to steer it not to overcomplicate things? I guess Codex here I come
There's also the fact that you are in control. You are not obligated to take the AI's commits. I don't even let it commit much of the time because commit time is review time for me. If it changes the button blue and does four other things, you can just take the blue change and discard the rest. It can't stop you.
This isn't a defense of it doing those four other things. It would be nice if it did what you wanted correctly. I'm just saying, as long as our programming skills have not completely atrophied, we have the power.
“Ford carried on counting quietly. This is about the most aggressive thing you can do to a computer, the equivalent of going up to a human being and saying "Blood...blood...blood...blood...” ― Douglas Adams, The Hitchhiker's Guide to the Galaxy
I dont find this to be indicative of Claude (opus?) at all. My experience doesnt lead me to think it would change a cancel button to blue if I ask it to change an "Add to Cart" button to blue.
It’s a contrived example, but also not very far from the mark.
The spinner with random claudisms is what makes the game, mixed with Claude helpfully deciding to pick up some tasks along the way.
I feel like most of my conversations with Claude lately are a battle of “I am a technical user, share technical details, but not random superfluous gibberish”
This got me on "cyanide blue", and I was ROLLING ON THE FLOOR LAUGHING on "Approaching usage limit". I can barely stop laughing now and my stomach hurts. I mean, Thank You!
I only made it through the first round of prompt selection; both options for the second step were equally pointless and not at all prompts I would ever expect to result in a constructive outcome. In my experience, telling the model it screwed up without specifically addressing, unambiguously, how to fix it, only leads to more suffering. If this page illustrates nothing else, I think it shows the immense downside of trying to use simple one or two sentence prompts.
EDIT: Actually, I used to use Google's AI Studio a lot and fork it after every successful prompt interaction. When I'd encounter a problematic issue like this, I'd revert to the previous fork and try a different prompt until I could get the desired outcome, thus mitigating the need to "argue" with the LLM. Unfortunately the ability to cleanly fork and revert everything including the LLM context was removed some months ago, and I've yet to discover a workflow with any tool that works as well for me.
> In my experience, telling the model it screwed up without specifically addressing, unambiguously, how to fix it, only leads to more suffering.
I wonder if this is just a reflection of some senior folks being arrogant towards junior folks. When the latter finished a task but not to the liking of the senior person they might just get a "that's wrong, try again". Just to have sth similar repeat the second time around. But the arrogant guy got to boss around the junior one, and some junior folks grow up learning that's how you should behave so they also do it later.
Now it's not a person but a machine. And people just make fun of the dumb machine. Well, garbage in, garbage out If you are not specific in what you want, you might get crap back. Or at least sth you didn't envision.
How is "Make the shopping button blue" not specific? Any human with a lick of digital common sense would be able to understand what I obviously mean. How am I supposed to anticipate every single way the AI could possibly misinterpret a simple fucking instruction like that?
One thing I have to be honest about, and it's mine..
The one thing I would check before... do you want to do that? Say go an and will do it without the check
While checking I found 3 vulnerabilities and 2 potential optimizations of which I fixed 2 and 1. Do you want me to file the other as issue, or stop for the day?
We have done <lists a weeks worth of work> this morning. I feel you need a break
Yeah, I don't mind using AI to help me at work, but having to "talk" with this stupid crap all day will send me to an early pension or something. Can't be healthy in the long run.
To be fair I’ve worked on human programmed systems where similar “it should be a half point story” requests would be met with snark by the engineers and take 2 sprints.
Also likely, devs took shortcuts to deliver fast. Now to make the button blue they need to differentiate primary buttons from others.
Simple, right? But design guidelines prevent one offs, and no !important. So you create a CSS class, but you discover another element on the header declared itself as primary (the search icon or the sign in button). You talk to that team and they decided to scope what’s primary according to their component. To change the sign in button to grey now you need to talk with the growth team. Growth team wants to run an experiment but they’re backlogged, only next quarter. They say you can innersource, just need VP approval. VP says blue matches a marketing campaign that is about to go out, agency has already been hired. You can’t talk to the agency unless Legal approves.
So you leave the button gray, to revisit decision next planning cycle after you can align all stakeholders.
- The people who won't make any effort to learn the tools, and something as simple as reverting code (via git) needs to be done by AI?
- The awful programmers who we've had to endure working with, who are so bad at simple changes that they have negative productivity?
- Or Claude itself?
---
BTW: I don't have these problems, but I'm also not afraid to do things myself when it's easier.
Edit: If I want to change a button's color, I just change it manually. If I don't know where the code for the button is, I might start with prompting, (because AI can often find the code faster than I can,) and then once the diff is proposed, start adjusting things by hand.
Seems to be getting a polarized response. I quite enjoyed the it, but I do think the creator should have made it clearer that a) it is in fact a joke site and b) it does not consist of actual Claude responses.
It is easy to misinterpret this site, and therefore not "get" the joke.
One note for those still using the Claude system for chats: there's no system to get generated images, spreadsheets, etc. out of the system. They claim it's a "security concern" to provide that data to you, as if they are protecting you by refusing to follow data export laws.
I'm hesitant to email their data emails, as it's common for companies to delete all data upon any request, instead of providing data as they are required to.
I heard this was a thing when listening to a Theo podcast, he mentioned to add a "Only use subagents if the user explicitly requests them" line in your agents.md file.
I don't know if it works, but I've always had a consistent level of token burn on my plans (I've only heavily used Sol after adding it).
I’ve never used any of these tools. Please tell me that this is a grossly exaggerated parody, and that the tools don’t write like this, or do so many ridiculous things. For my sanity.
(I am genuinely uncertain, though I presume it’s at least somewhat exaggerated.)
Sometimes you do get stuck on these weird tangents that feel like bike shedding. You end up wasting tokens because you left the AI to make decisions on ambiguous directions and then have to spend time diving down to fix it into what you actually wanted. Honestly it's not too different than working with people who don't exactly understand your grand plan but are technically adept. They can build whatever you want but only if you are clear with your instructions or have already developed the plan with them. Just like if you hired someone to make an app for you and it didn't quite do what you wanted it to, it's going to be expensive to make large changes. So you need to research (or at least have a conversation with the AI) about frameworks that best fit your needs and have the flexibility to add or change features after the core parts are stood up. It also helps to stage the project in phases so you're not distracting the AI from the critical aspects with aesthetic tweaks.
I haven't been professionally responsible for code since before AI started eating everything, and haven't touched AI in the meantime. Is this representative of the BS that everyone else here puts up with on a day-to-day basis? Did we collectively burn trillions of dollars for this?
Sometimes, yes, but obviously not everyone. I suspect that it depends what you're doing and how. Perhaps the AI has more difficulty understanding what some people want than what others want. I certainly have had a milder version of this and it neutralises the value of the tool because it keeps trying to do things you don't want in a way you don't want it to do them. I get annoyed at having to write huge explanations of how to do something and end up doing it myself sometimes.
> Please tell me that this is a grossly exaggerated parody, and that the tools don’t write like this, or do so many ridiculous things
It's a pisstake, but (in the bits I read, and based on my own personal experience) the writing style is barely exaggerated, while the behaviour doesn't ring true at all.
The behavior, while slightly exaggerated, rings entirely true for me. From the other comments in this thread, it seems I am prompting poorly in a similar way to the options offered.
I am guessing you prompt differently than what is shown in the game?
The behavior is congruent with my experience in abstract, in that all models will regularly do things you didn't ask for, will regularly go "above and beyond" by their training I expect, will regularly make changes that are entirely orthogonal to the change you asked for.
I've worked with Opus and Sonnet daily, and they are pretty great at generating functions and modules and components that have clear boundaries of concern, but I've recently been working some research tasks into our infrastructure and code and it seems impossible to coerce Sonnet into making only specific changes to a document you are working on.
It also blatantly ignores instructions as a rule. "Don't disassemble java class files, just ask me to pull in the source code" worked less than half the time. The Intellij Copilot plugin just doesn't use the AGENTS.md and similar files, and there doesn't seem to be any meaningful activity in the bug reports of same. "Don't modify code unless I tell you to" had bad adherence as well.
It also will read documentation and inform you that it says the opposite. This problem happened to me across models, across model updates, across months of real time. There's a specific example that I will not mention to avoid having it be trained on specifically. A distinct but similar problem is that it will take bad documentation and just pretend it has a good understanding. Claude gave me absurdly wrong descriptions for Splunk alert settings with absolute confidence.
I don't think any agent can reliably figure out "I don't know"
In my experience, Claude Code is great at making a first-pass at a project, but once you start asking it to make changes, it explodes. A bug fix that should only be 2 lines turns into adding 3 functions totaling 100 lines.
Something as simple as "make the button blue" should be done manually.
Surely you still have to give feedback to Claude. How do you do that without talking to Claude directly? By using a different model? But wouldn't that AI have no more common sense than Claude?
For a moment I thought, wow, someone put a lot of work into creating this theme park of frustration.
Next: It would be so easy to create a faux-Claude like this.
Then: How hilarious to watch the transcripts of unsuspecting users in real time.
Finally: I began wondering if this might be relevant to all the redundant, unnecessarily preambled, sentence structure complexifying, indirect referencing, canned phrasing, ambiguity mining, analogy maxxing, over-wordy responses I have recently been getting from Fable...
I’m don’t do much change-clicky-button software development. But when I do, I point models to specific lines of code. And have never had a result this bad.
Maybe this is geared towards pure vibe coders for whom a file and line number is too technical.
I was expecting it to spend 30 minutes running headless chrome instances, taking screenshots and analyzing them in python to verify the blueness of the result.
I had 5.6-Luna coordinate a code review in which it spawns 2 agents looking for different things. My prompt was "review the currently checked out branch. diff target is `next`. The jira ticket is XX-XXXXX..." My `next` branch was a few commits behind `origin/next` but it still did its review against the stale local version instead of clarifying or inferring that I meant `origin/next`. The findings were very confusing until I realized what I did.
I'm noticing the need to be really specific with any instructions lately, which I don't think is a bad thing. I expect co-workers (or anyone really) to tell me what they need in specific terms so I can get it right. I can do the same for the machine, I guess.
I use Opus and Sonnet 5 all the time and I find their language grating. But honest, I prefer to put up with it and get the results than to put up with my own human limitations and not get the results.
So this site is just a fan-fiction that thinks it's somehow dunking on Claude? I've never had a session that remotely resembles any of this. I honestly can't tell what point this site thinks it's making.
I wrote my own harness to stop shit like this from getting to my attention out of frustration.
I'm sure there's quite a bit of variation from person to person in these sorts of experiences, based on your harness, the way you talk, the stored memory, your CLAUDE.md, etc. But people absolutely have had this Opus 5 style experience the app simulates.
Oh dear - this explains why people have bad experience with ai. The prompts in the game are terrible.
The user was providing no context, they had no ability to give the model background or context. A simple why would have prevented 95% of these side quests. “Im trying to increase the relative visibility of the add to cart action on the page. Can we please change it to blue without changing any other buttons. /effort low. Let me know if you have questions and before editing anything tell me what you are going to do”
I am self delusional enough to think I could have gotten a blue button if I could have typed the prompt. But scared enough to realize I’ve lived this and spent months refactoring these exact problems.
I dunno, in my experience it's often overly-narrow, sometimes jumping through all kinds of hoops to preserve some edge-case behaviour that doesn't matter because I didn't mention it could be changed.
Nails the Claude dialect. Technical nonsense like:
> I'm collapsing this back to the rendered outcome:
And intermixed with SaaS product page idioms from a brain-damaged marketer like:
> No broader cleanup.
> No further architecture work.
> Just the button.
Aside from the patterns everyone knows like em-dashes, "its not X, it's Y", etc. I think the key features of claude diction is it sounds like a junior engineer over their skies who is trying to make up for that with extra verbiage mixed with extremely grating SaaS marketing-ese.
Do people really prompt AI like this? Multiple times the choice was either to yell at the agent, or ask it why it did something, neither of which are very fruitful lines to go down if you know what you're doing
Not really my experience with Claude, and the prompts are not how I would phrase them, but still pretty damn funny: I especially loved the slot-machine-style picker for LLM-isms.
Am I the only one whose experience doesn't match this?
My gripe with Claude is that while investigating how to do this it will report 200 other incidental findings which I overlooked and I realize those are broken too and need urgent fixing, derailing me, not it.
Oh man, exactly. I'm very prone to scope creep as I work on tasks. I already would notice some things that could be fixed or refactored and have a hard time not touching them before I used agents. But now I have to be very intentional about not letting it manipulate me into fixing EVERYTHING RIGHT NOW. Half the time the "one more thing worth noting, unrelated..." isn't even an actual issue, it just brought it up to fish more usage out of me.
Also, while this little demo is certainly exaggerating the issue, I do find working with Claude to sometimes get quite verbose and tiresome. I doubt I would struggle this much to get it to change a button color, but the patterns of speech, the endless lists, the over-explanations, and the whole song and dance of trying to get it to make the change you want without side-effects is frustratingly familiar to me.
Just when I came back to my pc and was thinking "I hate this world were everyone talks about AI like fanatics" this made me a little bit happy, especially the unhingend all caps options towards the end
To try to add something to this discussion -- I think that while I've seen these sort of loops less --- what I have seen is "overly helpful".
Models nowadays want to double-triple-quadruple check things. I'm being silly but it verges on "I have a working solution but let me write a variation in Rust to ensure a convergent solution and prove this works".
I've had to stop models nowadays mostly because they're being agonizingly pedantic in their validation. Opus is actually one of the most pedantic and "off track" here. But again, not in a bad way. I'm usually like "stop testing latency between 50 runs of this app... this is version one.. we're going to make a million more changes.. you're not buying us anything".
It tends to relax some of the insane things LLMs tend to do.
That is exactly how the slop accretes and you get a pile of crap. Claude somehow assumes that said 2 hour old userless app is some dusty enterprise app with millions of users and billions of dollars at stake for a 1 second outage.
I have to constantly have these things "take a deep breath, step back and look at the entire thing and do this change holistically. please restate what i'm asking you to do and why it's important"
This doesn't actually do anything though, right? There's no understanding, so the machine will just reiterate the original token query back to you. The 'why it's important' part will just generate some patronizing boilerplate as a raison d'etre.
Yeah of 10 minutes ago. It is shocking how long some seemingly simple things can take. I know there are some things I can do faster than the LLM and some things it can do faster than me. The amount of rambling BS is the exhausting part.
And shocking how little code end effort some things take if done by hand.
I am not some hardline LLM hater, just venting my frustration.
Edit: for how to do this, I set up an OpenRouter account so that I could easily switch models, and then ran them in Pi inside of Orca ADE. Orca lets me easily switch from Pi to Claude Code to Kilo to Codex or Hermes or whatever. Pi+OpenRouter lets me easily switch the LLM. All of it lives in a single open source orchestrator to avoid any platform lock in to any AI company going forward, and can even do local LLM should I care to take on that massive hassle.
Agents and AI are getting expensive, it seems silly to waste tokens on static checks.
If you search my company's Slack for "smoke" the results are almost all within the past 2 years ...
If we each stop arguing with racists because we're each tired of it, then what we are is a society that lets racism go unopposed, a site with racist comments proliferating, a site where racists gather because they are able to be themselves, a site where future readers and AI training is learning that racism is popular, common, normal, and acceptable because everyone is accepting it.
It's a bit like democratic voting, your vote doesn't matter and cannot change things, but it's important that you vote because all of our votes do matter and can change things.
It's a bit like the quote "Dear Board, I don't want to belong to any club that would have me as a member. Sincerely yours, Groucho Marx.", I am upset and offended every time someone calls out my dumbass comments, but horrified at all the countless times nobody does. Why would I want to waste my time on a site with such low standards that it lets me comment? I'm exhalted and encouraged everytime someone upvotes or engages with my low effort quips[1] but thrown into a chasm of despair that HN users let that pass unopposed. When nobody engages with comments I spent literal hours on[2] I curl up into a ball and die, but also dream of it being a feather in my cap with dang that I'm posting a substantial comment and that will be its own reward. I identify with nobody more closely than Colonel Cathcart, and I will be delighted if someone engages with my Catch 22 reference, but aghast that such a nontechnical reference gets engagment; distraught if nobody engages but delighted that perhaps HN standards are being held higher.
If not you, whom?
[1] https://news.ycombinator.com/item?id=49478355
[2] https://news.ycombinator.com/item?id=49586378 or https://news.ycombinator.com/item?id=49491182
Except... the next day they are still there and I have to push myself a bit further through the wall of changes before closing the tab again.
then you are maybe reviewing 1 out of 7 PRs that implement the agreed upon change
Once in a blue moon it's actually the model making a material error in it's thinking and I have to go back and redo it.
While it did help the epilepsy, their brain was essentially two brains controlling two halves of the body. With one controlling speech. There were experiments where one eye was shown some instruction text, the corresponding hand performed that instruction, and when asked why they dix that action, the speaking half just made up some plausible, yet completely wrong reason, just like an LLM.
https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r...
When we do meta thinking about that process after the fact, two things happen. 1, we change our total “thought” by adding that meta thinking to it. And 2: it’s a very lossy process, because we don’t have very good data about what our brain or mind was actually doing during that first think and emotional factors are nearly always at play and even more complex.
Now for the more complex AI, the fragmented process of multiple agents and loops and reruns are pretty similar to that first think we do. At least structurally. But the meta think is where they differ. They have no emotion, but they also have even worse data about its own function. They constantly degenerate so I would argue their “changing the thought by thinking about it” factor is also generally way higher than ours.
Getting better at consistent/reproducible thinking, with many ‘steps’, that leaves good documentation of that thinking behind for future analysis, has to be one of the more important areas for the big flagships going forward. I’m certain that “what is this fucker doing and why” is the biggest pain point for AI researchers. Or the math, it’s usually the math.
But you’re correct in the general structure; they generally do the same post hoc analysis we do, just noticeably worse because of their opaque nature(even to themselves) and general degenerative instability.
Sort of, actually. I think we humans actually have some intuition that we'd be more effective if our cognition were augmented more directly by machine strengths: the ability to run precise calculations, more memory, ability to look facts in some sort of knowledge graph.
I think we're on the right track, but instead of augmenting humans with machine strengths, we're building intelligence in hardware in a way where it can access that augmentation. Plus, then we can quickly distribute updates, run parallel instances, etc.
If intelligence is compression, and hallucinations are essentially loss, then as the models grow in size performance (at least as far as hallucinations) should reduce. Or we'll get things fast enough that we can afford to stop relying on model weights for memory and check an increasingly larger set of discrete facts as part of reasoning.
Right now, the models are making trade-offs. As compute grows, and inference gets faster, we can make fewer of those trade-offs and start to use the unique strengths of machines to fill the gaps we're seeing, I suspect.
the inconsistency of LLMs is by far one of the biggest gripes I have with them. Closely related to their apparently deep desire to avoid following instructions.
I know these are both a byproduct of noise (which is somewhat tunable) and noise is inherent to these systems in a lode bearing way.
I still hate it. it’s holding the technology back. I don’t honestly see how we can safely or even successfully approach the idealized realm of AI without bypassing this problem, which to my understanding, probably means not using language models at all and trying a totally different approach. But I really don’t know much about machine learning, I’m a super novice compared to a lot on this website.
If they've worked some minutes/hours/weeks on something and you ask them why did they do that, they will either answer honestly and truthfully, lie, or say "I missed that/didn't seem important so I just chose something at random".
None of these cases are similar to how AI works.
It's not clear to me whether you're aware of a rigorous basis for your claim or you're just inferring based on what you think makes sense, but I can't help but wonder if it's the later, in which case regardless of the mechanism, the outcome certainly seems to resemble what happens with LLMs.
Perhaps up to 50% of people actually don't have an inner monologue, much like many have aphantasia where they can't actually see anything in their mind either.
Yes, but the vast, vast majority of decisions you make either don't take place via an inner monologue, or include details that were not actively/consciously "thought" and reasoned with in your inner monologue.
And yet, when asked why you did something, you're not likely to respond "sorry, that decision was made subconsciously". Instead, you use your inner monologue to try to backfill in a reason why. That reason may be correct, or it may not be. You don't actually know, since you have new data that may be updating your own internal state as you try to rationalize it after the fact.
Sounds like the left hemisphere usually uses something from the right hemisphere to answer those questions and it can't do that if it's been cut off?
We know that our brains are capable of hallucinating due to substances (drugs), being asleep, brain damage (including split brain), hypnosis, etc. Just as RAM damage make your computer do weird shit. That doesn't mean it operates that way normally.
You can't just remove a huge part of a system and then assume that the whole system behaves the same.
Do you know anyone who actually reads and adheres closely to all of the documentation every time it's changed?
But now that I've been working on it a month and there's a lot of documentation, it's pretty clearly ignoring parts of the documentation and parts of the code. It will come up with some ridiculous statement about how something works, and I'll challenge it, and it'll admit I'm right.
It definitely reads more documentation than any programmer I've ever worked with (myself included) but because it doesn't have a memory other than the documentation, it still makes mistakes like that.
I haven't turned on "memory" or tried it with Codex, so I don't know how that'll change soon, though.
Depending on language I've seen this scale past multiple millions of lines of code, as long as you pair it with all of the linting and tooling that you can possibly build.
You can extend this further by using an adversarial agent trying to find mistakes in the other instance's logs in a loop where a 3rd neutral agent weighs the claims of the other two. This is also just another step in reducing error, it does not guarantee elimination of such errors. The latter is an impossible guarantee, even for humans.
Additionally the provided prompts are not what anyone who has used this things would say in either situation.
Sure you can ask it to make one button blue and it can easily make all buttons blue, but they quickly backtrack if told to.
This is fucking constant. I can't deny that this stupid tech saves time prototyping even with having to wrangle it, but it commits a fireable offense several times a day that no human would get away with and is obviously incapable of learning from mistakes in the way a human is. The only reason it's not fired is because it's a slave that works for no more than the cost to feed it.
Everything in life is variable reward. You invite a friend over, they might accept or they might not. Drive to work, traffic might be good or might be bad. You ask a colleague to finish a task, they might do it or might not or might do a good job or might not.
Everything is variable reward. Is everything gambling?
well, no. If you work overtime and get paid overtime, you are not gambling and that is not a variable reward.
Humans engage more with rewards that are intermittent and variable. Like Futurama's scene from 'The Scary Door' where the character says "A casino where I'm winning, I must be in heaven! A casino where I always win, that's boring, I must really be IN HELL!". A constant predictable reward is boring, less engaging. So if you know you get no overtime, but sometimes your boss rewards you with $5 coffee voucher, sometimes a free pizza dinner, sometimes double-time pay for the time worked or a half-day off, now you might be gambling 1hr overtime for an intermittent variable reward.
> "Drive to work, traffic might be good or might be bad."
Good traffic is not a "reward" for driving to work(!) and you have to drive to work regardless so you are not risking anything [you might be risking your life, but you are not making a choice which can reward you with good traffic]. You might say that going a different route is a choice and a gamble which could reward you with good traffic, but traffic engineering does not work that way because if there was a consistently low-traffic route, everyone else would take that route until it was no faster than any other route. Traffic will generally be the predictable and similar every day, plus 'arriving at work early' is not much of a reward.
Perhaps but predictable outcome is a very desirable quality. No one wants a hammer that sometimes drives nails and sometimes doesn’t. All of the current harness engineering work is about squeezing predictability out of the LLM.
> Good traffic is not a "reward" for driving to work(!)
Like hell it’s not. I drove into work last Friday and there was no traffic because of the holiday weekend. It was amazing. Had me considering whether Friday should be one of my standard RTO days.
regarding your examples, i think the difference is that with ai, you’re literally sitting in front of a machine, pressing a button, and (almost instantly) getting a result that, if not desired, can immediately be tried for again. you even spend “tokens” to do this, and at least in my native language, “token” brings to mind the coins you’d stick in a slot machine
If you sit at a slot machine and pump quarters into it, each “turn” is independent. You spin and you win or lose. It’s pure chance and there is no destination. You execute the exact same action over and over and hope random chance brings you more money.
If you sit down in front of a coding harness, the progress is incremental and directed. You ask for a thing, the LLM produces something that is hopefully close to what you wanted. You give it more direction to prod it closer to the end state you want. You are not executing the same action, but incrementally nudging it in the right direction. I’ve literally never restarted from the same initial state with the same prompt and hoped for a different result and I don’t know why anyone would. Rarely I’ve thrown away the progress made and started over but always with a very different prompt that includes learnings from the failed attempt.
most people don’t use ai this way though, and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it (and this is even more obvious with image generation as you chase that perfect output)
perhaps it’s better to compare it to gacha than slots?
How do they use it? Surely no one is just repeating the same prompt over and over (except as a Ralph loop perhaps, which is automated). I’m really struggling with the notion that most people just throw the same prompt repeatedly hoping it eventually works. Because that doesn’t sound like gambling. It sounds crazy (and frustrating).
> and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it
In the sense that you get a dopamine reward when you succeed, sure, but I get the same reward when I code by hand and achieve a successful result.
> and this is even more obvious with image generation as you chase that perfect output
This is fair, because sometimes with image generation the same exact prompt will produce very different output. This is becoming less true as the models get better and it becomes more effective to direct image generation iteratively than to keep starting from scratch with a barely tweaked prompt.
And sometimes the debugging has already veered completely off course at the beginning so it's futile, but each time the fleeting hope that just a few thousand more tokens will magically fix it tempts you to keep going a little longer.
as an example, i was using chatgpt a few weeks back to help me remember the name of a painting i’d seen about a decade ago. i could recall the general shape of the subject and that it was europeanish, but nothing else. after seven turns or so it finally got it, and honestly, the relief of finally remembering the name felt like, well, hitting a jackpot
i’ve had a similar feeling of success after trying to get it to give a comprehensible answer when asking it for a solid counter-argument to philosophical questions. it is indeed often crazy and frustrating
> but I get the same reward when I code by hand
i have only done very simple coding work with llms, so that may be why our ideas differ about the feeling of reward. this is where the comparison to gacha makes more sense than slots, since when you’re coding, you still get a reward each turn whilst chasing the final/desired result
I understand the joy of success but I fail to see how this is gambling. I could have an equivalent conversation with a friend (more likely about a movie in trying to remember than a painting, but still) and get the exact same type of iterative “no, not that one, it was more like X” and feel elated when my friend finally realizes I’m taking about a scene from Hot Tub Time Machine.
This isn’t gambling in any meaningful sense.
Sugar is the original point of the reward system!
Do hard work (takes time), get dopamine for successful completion.
Find berries, taste sweet (hopefully safe), eat all, get calories. Doordash Krispy Kreme instead = few too many calories.
(I’m no Luddite in the sense popularly thought of them pre-‘22 [1], though we have to watch skill atrophy)
[1] regressionist? Decelerationist, too loaded perhaps. Someone remembers or knows the word…
Not everyone has the desire to work around the system, and many are diametrically opposed to the concept of AI. They get this perception that it's a slot machine because of that inconsistency, and then do the human thing of assuming that other people must just be flawed if they're different from them. They're "addicted to gambling".
Obviously, things have changed. Open models can still be like that, but are often so fast and cheap at iterating it doesn't matter. SOTA models aren't perfect, but are to the point that they're generally much better than the average developer.
But once that perception set in and the meme spreads, it's really hard for some to break out of it. Especially at the pace AI development has been moving. It's just that simple.
I got so sick of all this at some point that I slowly stopped doing anything that wasn't my job. But then AI got better and better and I realized it was the ultimate unblocker. When that dreaded malaise started creeping in signaling it was a project's end because I didn't want to waste any more of my life dealing with bullshit orthogonal to what I was trying to do, I'd give it to the AI. It felt like a miracle the first time this worked, and it still does. If we were previously equipped with shovels to dig through bullshit, we now have a fully automated Bagger 288.
The reward schedule now isn't variable anymore; the chance that I finish something in a good state is 100%. I can focus on the parts I actually enjoy - architecting the broader system, making the parts mesh together in a sensible way that's easy to work with and has some mathematical elegance to it, hand coding the bits I want to be really specific about (but now without the endless frustration of bugfixing or import errors and edgecases being immediately discovered, thanks to the AI).
I was working on a side project recently. I had spent months designing the data model in my spare time, thinking through how to make it as elegant and durable to change as possible in the long term, since (if I launched it) the repercussions for getting it wrong would be significant.
Once I had a working design, it probably would have been several more months to build a working prototype and start testing it.
Instead, Claude knocked out the prototype for me in an afternoon. And it immediately became clear that it didn't work: not because the data model didn't solve all the problems I wanted it to solve, but because it didn't fit the shape of how I quickly learned a normal person would need/want to interact with the product. I was so focused on the long term, that I never thought about what the first five minutes of a user with hands on the thing would need. And the changes needed would be significant.
Maybe there's some variable reward mechanism. But I sure was glad to be able to pull that particular slot machine handle and learn that than waste even more of my time on what was a dead end.
At this point, what do the words even mean? Your own patience and available time are always completely random and fairly distributed across a large enough data set?
> Maybe I'd waste hours down the wrong rabbit holes trying to find a library that worked for my use case. Maybe I'd waste a day trying to get an API to do something it turned out it couldn't do. Maybe I'd have to redo my entire approach because of some factor I hadn't considered.
Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
Predicting the time a task will take is impossible. Something that sounds like a 5 minute script can turn into a month of banging your head against unknown unknowns. I lose my patience when the afternoon I allocated is getting overrun by nonsense and I'm missing out on other things I wanted to do or household maintenance.
> Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
Every thought we have has random chance to be wrong despite our conviction that it's correct. Descartes' Evil Demon plays his tricks on all of us. How many times have you typed some line of code only to realize it was obviously wrong afterwards? Even for simpler matters we "hallucinate" all the time. I was deep in thought trying to help someone come up with an acronym the other day and felt convicted that "Goal Oriented Augmented Retrieval" worked for GOAL until I said it aloud.
Our thoughts and actions are consistently wrong some portion of the time because our meat computers are not perfect positronic brains running prolog. We put cereal in the fridge and say "you too" to the waiter. Every thought we put down or action we take is a gamble on the soundness of the thought or action.
> Half the site is blue. I asked for ONE button.
Those are my only options when the site is clearly not blue, two buttons are.
There is a reason for why I am much more specific than this.
So leave emotion at the door and make your words count. Voice your frustrations at the stress doll next to your monitor, sure, but spending tokens on them us just wasting time and money. Explicitly typing out your exasperation will not get you closer to whatever it is you are trying to accomplish. What can you type that will?
Laser focus: clearly, concisely explain what is wrong right now, and how you want it solved. Then do that again until all the moles have been whacked. If the AI is stuck in a sycophancy loop, start a new clean session. Do this frequently anyway: every turn in the same session charges for all preceding conversation again /and/ degrades the LLM's performance.
My best takeaway from the site was: Every time you see Claude Code / Codex doing something you don't want, ask it to add to AGENTS.md. Specially if they churn a lot doing something, and eventually find a way. I ask it to store the way it made something work. It's a constant gardening of AGENTS.md and it has really working well.
If I get frustrated by it because it just would not "listen", I discontinue its use!
Either that, or your coworker themselves would laugh and tell you about it later.
That’s also where this quiz lost me. I wouldn’t respond in either way, I’d say “all buttons look blue now. Can we make it so that just the Add to Cart button is blue? I’m okay with Add to Cart having its own class to make it easier.” or something.
FYI I actually agree, it also surprised me. Anyway everything is deterministic on the website so it's not like that prompt actually has an influence on anything.
I do remember that one of the first things I did with my CLAUDE.md was to tell it to stick to the scope of the task, never to jump ahead and do extra "helpful" things without confirming with me first, and to follow best software practices including around refactoring but also to specifically avoid overengineering. I don't know if that is what's giving me a different experience from whatever the author seems to be "satirizing".
AI isn't magic, and it won't (or, at least, doesn't yet) enable anyone to do All The Things.
But with current models you actively have to sabotage the context to get this kind of behavior, or dramatically underspecify (3 words versus 2-3 sentences)
The user is asserting that this is "not a lived experience"
working out if the user has missed the LLM Cliche
Formulating a suitable response
"did you just argue with a joke?" No, thats to blunt
"Is it satire when it only applies to you?" no again passive aggressive
lets just print the dictionary definition of satire and hope the move on
> Satire is when...
I have misbehaved in this fashion for many people across the full spectrum from casual users to highly experienced software engineers with millions of social media followers, so the statement that it's similar to making a joke about a calculator misadding two numbers because a stray beam of solar radiation flipped a bit at least for my part is not true.
Would you like me to start using bad English and doing things you never asked me to for your sessions, too? Just say the word.
Ask for change A and get unwanted change B happens all the time with bad programmers and tradgedy of the commons (ie poorly architected, no restraint) codebases.
Most normies dont know about this stuff.
Usually just restarting the session helps, though.
My impression is that this is an oversimplified demonstration of what can happen when you prompt Claude in a system with many more variables (than two buttons and two colours).
If I want the button to turn blue and that's it, what instead do I ask? Even in a complicated system with many levers, what do I request other than the desired end result, hoping that Claude pulls the right levers to produce something acceptably close to what I think I asked for?
If I don't understand why changing one button changed them all, I would use an entirely separate session to ask questions about how the site theming works. The fact that the site has a bunch of weird coupling between themes is useful information, and if I don't know how to resolve that I'd ask Claude for ideas about how to safely eliminate the coupling, and once it proposes a reasonable idea tell it to implement that.
The two big things here I'd never do is use emotional languages in prompts, and I'd never tell Claude to revert changes and try again. Once the incorrect change is in the context it's poisoning all of your future results.
And when it doesn't, it's usually because of things outside of the codebase -- iOS layout quirks that aren't documented, buggy Python libraries it's relying on where you then have to tell it to read the source to figure out what's going on, that kind of thing.
You have to change the system so that the AI understands it, via establishing what your beliefs are, how those are reflected in values (especially important if you have e.g. compliance needs), how those values are reflected in the operational and strategic levels, and then a variety of tactical behavior coaching. For example, I ban 2>/dev/null - super tactical, and I say I value simplicity over covering every edge case - a very broad generalization.
Wow, you AI people really have a negative view of the technology y'all are trying to sell as the next Jesus
Once you understand what the problem is, you can give it better instructions. If the architecture is shit, the agent is going to have a rough time of it.
> Why is half the site blue now? I asked you to change one button.
> Half the site is blue. I asked for ONE button
Neither of these is an instruction to fix the problem, they're treating the AI like a person and telling it what it did wrong, expecting the implied admonishment to be enough to steer it back. But without an actual instruction, it just goes and does whatever it thinks will help, which is often arbitrary.
The response I would have used in this situation is
"The Cancel button is also blue now. Make sure the color change is only scoped to the Add to Cart button"
Most of the available responses throughout this "skit" are similar cases of expressing frustration first and guiding the result second.
Skip the emotion and say exactly what you want, and nothing besides that.
Be specific.
That said, GPT always acts up even if I am specific, but I only have the free tier there.
First thing I do when something goes wrong is tell the agent to stop and diagnose. You can't prompt effectively without three proper information.
Only since 4.8 though.
To the people rather lamely doing the "it's satire/a joke", that would require this to be an exaggeration of a reality. But...it isn't.
Got an audible guffaw out of me. This really is what the experience is like sometimes if you're just giving it a result without being specific in implementation, and it comes out of nowhere, some days much worse than others.
I've become patient with it, but whatever this style of output is called or doing - it is both condescending and entirely unhelpful, and it seems designed to frustrate.
Recently I've started using GLM models and noticed they aren't very claudish (just a little bit, compared to DeepSeek which is extremely claudish)
This isn't a defense of it doing those four other things. It would be nice if it did what you wanted correctly. I'm just saying, as long as our programming skills have not completely atrophied, we have the power.
“Ford carried on counting quietly. This is about the most aggressive thing you can do to a computer, the equivalent of going up to a human being and saying "Blood...blood...blood...blood...” ― Douglas Adams, The Hitchhiker's Guide to the Galaxy
I assume this is just a contrived example?
The spinner with random claudisms is what makes the game, mixed with Claude helpfully deciding to pick up some tasks along the way.
I feel like most of my conversations with Claude lately are a battle of “I am a technical user, share technical details, but not random superfluous gibberish”
This got me on "cyanide blue", and I was ROLLING ON THE FLOOR LAUGHING on "Approaching usage limit". I can barely stop laughing now and my stomach hurts. I mean, Thank You!
EDIT: Actually, I used to use Google's AI Studio a lot and fork it after every successful prompt interaction. When I'd encounter a problematic issue like this, I'd revert to the previous fork and try a different prompt until I could get the desired outcome, thus mitigating the need to "argue" with the LLM. Unfortunately the ability to cleanly fork and revert everything including the LLM context was removed some months ago, and I've yet to discover a workflow with any tool that works as well for me.
I wonder if this is just a reflection of some senior folks being arrogant towards junior folks. When the latter finished a task but not to the liking of the senior person they might just get a "that's wrong, try again". Just to have sth similar repeat the second time around. But the arrogant guy got to boss around the junior one, and some junior folks grow up learning that's how you should behave so they also do it later.
Now it's not a person but a machine. And people just make fun of the dumb machine. Well, garbage in, garbage out If you are not specific in what you want, you might get crap back. Or at least sth you didn't envision.
I guess we are all PMs now.
Simple, right? But design guidelines prevent one offs, and no !important. So you create a CSS class, but you discover another element on the header declared itself as primary (the search icon or the sign in button). You talk to that team and they decided to scope what’s primary according to their component. To change the sign in button to grey now you need to talk with the growth team. Growth team wants to run an experiment but they’re backlogged, only next quarter. They say you can innersource, just need VP approval. VP says blue matches a marketing campaign that is about to go out, agency has already been hired. You can’t talk to the agency unless Legal approves.
So you leave the button gray, to revisit decision next planning cycle after you can align all stakeholders.
- The people who won't make any effort to learn the tools, and something as simple as reverting code (via git) needs to be done by AI?
- The awful programmers who we've had to endure working with, who are so bad at simple changes that they have negative productivity?
- Or Claude itself?
---
BTW: I don't have these problems, but I'm also not afraid to do things myself when it's easier.
Edit: If I want to change a button's color, I just change it manually. If I don't know where the code for the button is, I might start with prompting, (because AI can often find the code faster than I can,) and then once the diff is proposed, start adjusting things by hand.
Caveats:
• ONE HONEST CAVEAT
• ONE THING WORTH STATING PRECISELY
• ONE THING WORTH FLAGGING
• WORTH NAMING
• WORTH STATING PLAINLY
• I DON'T WANT TO LEAVE THIS IMPLICIT
• I'D BE DOING YOU A DISSERVICE
• I DON'T WANT TO BURY THIS
• BETTER NOW THAN LATER
• ONE HONEST TRADEOFF
• I DON'T WANT TO PAPER OVER THIS
• ONE SMALL HOUSEKEEPING ITEM
• THE HONEST PART IS SIMPLER
Pushbacks:
• FAIR PUSHBACK
• FAIR HIT
• THAT'S ON ME
• YOU'RE RIGHT ABOUT THAT
• YOU'RE RIGHT
• YOUR INSTINCT IS RIGHT
• FAIR, AND MORE SPECIFIC THAN IT SOUNDS
• RIGHT, FOR A REASON WORTH NAMING
• I'M NOT GOING TO DEFEND THAT
• YOU'RE RIGHT TO PUSH BACK
Reframes:
• LET ME BE PRECISE
• THE SHARPER DISTINCTION
• THE PART THAT MATTERS
• THE USEFUL PART IS NARROWER
• LESS X THAN Y
• VISIBLE ISSUE / UNDERLYING ISSUE
• THOSE ARE DIFFERENT CLAIMS
It is easy to misinterpret this site, and therefore not "get" the joke.
I'm hesitant to email their data emails, as it's common for companies to delete all data upon any request, instead of providing data as they are required to.
This hit a bit too close to home. Sol has the same issue, spawns a lot of agents for no good reasons (besides burning tokens).
I don't know if it works, but I've always had a consistent level of token burn on my plans (I've only heavily used Sol after adding it).
(I am genuinely uncertain, though I presume it’s at least somewhat exaggerated.)
It's a sad parody that people will take as reality.
It's a pisstake, but (in the bits I read, and based on my own personal experience) the writing style is barely exaggerated, while the behaviour doesn't ring true at all.
I am guessing you prompt differently than what is shown in the game?
I've worked with Opus and Sonnet daily, and they are pretty great at generating functions and modules and components that have clear boundaries of concern, but I've recently been working some research tasks into our infrastructure and code and it seems impossible to coerce Sonnet into making only specific changes to a document you are working on.
It also blatantly ignores instructions as a rule. "Don't disassemble java class files, just ask me to pull in the source code" worked less than half the time. The Intellij Copilot plugin just doesn't use the AGENTS.md and similar files, and there doesn't seem to be any meaningful activity in the bug reports of same. "Don't modify code unless I tell you to" had bad adherence as well.
It also will read documentation and inform you that it says the opposite. This problem happened to me across models, across model updates, across months of real time. There's a specific example that I will not mention to avoid having it be trained on specifically. A distinct but similar problem is that it will take bad documentation and just pretend it has a good understanding. Claude gave me absurdly wrong descriptions for Splunk alert settings with absolute confidence.
I don't think any agent can reliably figure out "I don't know"
It’s funny because there are elements of truth in each bit of it, though.
In my experience, Claude Code is great at making a first-pass at a project, but once you start asking it to make changes, it explodes. A bug fix that should only be 2 lines turns into adding 3 functions totaling 100 lines.
Something as simple as "make the button blue" should be done manually.
Surely you still have to give feedback to Claude. How do you do that without talking to Claude directly? By using a different model? But wouldn't that AI have no more common sense than Claude?
For a moment I thought, wow, someone put a lot of work into creating this theme park of frustration.
Next: It would be so easy to create a faux-Claude like this.
Then: How hilarious to watch the transcripts of unsuspecting users in real time.
Finally: I began wondering if this might be relevant to all the redundant, unnecessarily preambled, sentence structure complexifying, indirect referencing, canned phrasing, ambiguity mining, analogy maxxing, over-wordy responses I have recently been getting from Fable...
Maybe this is geared towards pure vibe coders for whom a file and line number is too technical.
I gave it a README specifying the requirements for everything I wanted to build. My interaction with Claude was more or less like this.
I had 5.6-Luna coordinate a code review in which it spawns 2 agents looking for different things. My prompt was "review the currently checked out branch. diff target is `next`. The jira ticket is XX-XXXXX..." My `next` branch was a few commits behind `origin/next` but it still did its review against the stale local version instead of clarifying or inferring that I meant `origin/next`. The findings were very confusing until I realized what I did.
I'm noticing the need to be really specific with any instructions lately, which I don't think is a bad thing. I expect co-workers (or anyone really) to tell me what they need in specific terms so I can get it right. I can do the same for the machine, I guess.
> WebFetch en.wikipedia.org/…/Cyan
> WebFetch en.wikipedia.org/…/Prussian_blue
> WebFetch www.colorhexa.com/16b8c4
Brilliant.
"I was wrong" and "the honest truth" are just forever phrases that are now dead to me.
I use codex now.
I use Opus and Sonnet 5 all the time and I find their language grating. But honest, I prefer to put up with it and get the results than to put up with my own human limitations and not get the results.
I'm sure there's quite a bit of variation from person to person in these sorts of experiences, based on your harness, the way you talk, the stored memory, your CLAUDE.md, etc. But people absolutely have had this Opus 5 style experience the app simulates.
Yes, and it's mine!
The user was providing no context, they had no ability to give the model background or context. A simple why would have prevented 95% of these side quests. “Im trying to increase the relative visibility of the add to cart action on the page. Can we please change it to blue without changing any other buttons. /effort low. Let me know if you have questions and before editing anything tell me what you are going to do”
Hold on let me go commit code without reviewing.
But you can see in this thread that folks definitely have experienced this.
> I'm collapsing this back to the rendered outcome:
And intermixed with SaaS product page idioms from a brain-damaged marketer like:
> No broader cleanup.
> No further architecture work.
> Just the button.
Aside from the patterns everyone knows like em-dashes, "its not X, it's Y", etc. I think the key features of claude diction is it sounds like a junior engineer over their skies who is trying to make up for that with extra verbiage mixed with extremely grating SaaS marketing-ese.
Do people really prompt AI like this? Multiple times the choice was either to yell at the agent, or ask it why it did something, neither of which are very fruitful lines to go down if you know what you're doing
My gripe with Claude is that while investigating how to do this it will report 200 other incidental findings which I overlooked and I realize those are broken too and need urgent fixing, derailing me, not it.
Also, while this little demo is certainly exaggerating the issue, I do find working with Claude to sometimes get quite verbose and tiresome. I doubt I would struggle this much to get it to change a button color, but the patterns of speech, the endless lists, the over-explanations, and the whole song and dance of trying to get it to make the change you want without side-effects is frustratingly familiar to me.
It made me stop using Claude at all. Codex has almost surgical precision, and I like that a lot.
(But nowadays I just use DeepSeek Flash. it does screw up but its cents so ¯\_(ツ)_/¯).