Some conjecture on my part, but I imagine that the bugs being found are not the kind you can just press a button on the LLM to fix anyway, but are instead the sort of bugs which require the reviewer to deeply consider design tradeoffs, of which I believe there are simply more in a DB or programming language codebase, for example. Stuff I am finding in review is often like "this thing isn't like the other thing, which has this behavior, which is wrong if the aim is X, but sort of correct if the aim is Y, and totally perfect is the goal is Z and the use case is J, but not I. Unless it's Q, in which case you probably need to reconsider some fundamental decision making from 12 months ago and have some conversations with people about how users desire it to work in the first place, and then set about understanding the tradeoffs inherent to that design."
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.
You nailed it! I often got bug reports from AI that would touch far more code to fix than what I am willing to touch, and the likelihood of that large piece of new code (which I honestly would not be able to review thoroughly) will likely contain more bugs than the one I am fixing.
So in the end I often ignore these bug reports after evaluating their actual impact on the product.
I like the way you put this. I agree this is where the frontier of software development is right now. You can have LLM chug and bug fix all day long but in the end most real issues are in this layer of intent. I don't think this problem is particular to DB/programming language codebase either - it happens a lot in product design it just doesn't surface in code review. "Everything is working perfectly" yet the resulting workflow or UI is not truly making the user's life easier, or has grown grotesque with additions/modifications.
I will say, if the problem is purely code you occasionally see "sparks of AGI" where the model will step back and say "hold on, there is a massive simplification we can make here...". This started happening with Fable-class models. But rarely if ever will the model push for a huge simplification at the product level.
This is one angle of the productivity story. You don't have any sort of plausible deniability covering all these bugs, test cases etc. when they are comprehensively discoverable by tooling. It's like having typing bugs on a strongly typed language with static analysis, you just can't make a lot of excuses.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
I remember an LWN article on this question a few weeks ago. The debate was going on and they weren’t sure if it was that there were more bugs or just that because of AI they suddenly knew about more bugs than usual.
Either way, unfortunate but a very understandable decision.
"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."
They have been doing Postgres for a long time, and I believe they have been significant contributors from before AI coding.
Their marketing has to mention AI. Everyone has to in a business like that. I would not take it as evidence they are slop coders or hyping AI just on that basis.
Seeing how Fable handled Linux kernel bugs, where out of 82 bugs reported only 6-8 were actually bugs (not already fixed or fixed in newest version or straight up made up) I'm skeptical till humans actually sift through this pile.
I tried that this summer. I encountered this doom loop where the review agent would find bugs, the coding agent would fix them in some overengineered way that introduced new bugs, the review agent found them, the coding agent made more while fixing them, repeat until insanity.
The coding loop and the review loop are orthogonal and probably should remain so. Not least because it's probably worth the money for more intelligent models for code reviews where it's a more focused task while coding loops are all over the place.
Weirdly enough I've found the opposite, using too smart a review model gets you overfit instead of benefit. The coding side can be more complex in a way that benefits from fable/astra, but the review side sometimes not so much.
This is the happy path for LLM assisted coding. Obviously consumes less tokens being more constrained and focused so it's not what gets the most marketing but having the second loop of code reviews seems worth investment even if the coding loop is entirely manual or you're in full LLM autopilot mode
Training data is filled with shortcuts, conveniences, bugs, hacks, hacks that fix other hacks, hot fixes, patches, mappings that map the hack to hack to a better solution and all types of other surprises.
That's the cost of trying to scale so quickly. You dont filter out the crap.
A nice side effect is that the best way to fix it is more AI.
From the article: "He was also at pains to point out that the reverts were not down to 'vibe coding' in development, as the technique was not available when most of the new features were written."
I wonder when people will finally feel comfortable not putting vibe coding in scare quotes. Maybe when it accounts for 50% of all new software, instead of only 10%.
Why read the article when you can read the headline and make up your own?
I think these type of new techniques will ultimately create more jobs for humans. If we think the answer is just “AI” everyone will create their own Postgres with a lot more bugs and vulnerabilities.
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.
So in the end I often ignore these bug reports after evaluating their actual impact on the product.
I will say, if the problem is purely code you occasionally see "sparks of AGI" where the model will step back and say "hold on, there is a massive simplification we can make here...". This started happening with Fable-class models. But rarely if ever will the model push for a huge simplification at the product level.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
Sorry, garbage in garbage out doesnt fly. I feel sad for open source maintainers having to deal with this massive slop-ageddon.
Either way, unfortunate but a very understandable decision.
"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."
Naturally, EDB is an AI shop (https://www.enterprisedb.com):
"Agents need direct, governed access to data at the source. EDB Postgres AI is built for that."
Of course, nothing ever has worked in PostgreSQL before AI!
Their marketing has to mention AI. Everyone has to in a business like that. I would not take it as evidence they are slop coders or hyping AI just on that basis.
Just don’t post the unedited AI “bug report” as an issue please. A human should make sure it’s actually a real problem before dumping it on devs.
Not every time. But often enough.
That's the cost of trying to scale so quickly. You dont filter out the crap.
A nice side effect is that the best way to fix it is more AI.
It's a endless money loop.
I think these type of new techniques will ultimately create more jobs for humans. If we think the answer is just “AI” everyone will create their own Postgres with a lot more bugs and vulnerabilities.