Don't misinterpret this link as representing a final decision. It's actually three separate proposals which will be debated and then voted on.
Proposal A is "expressly forbid any contributions to Debian written with the use or assistance of large language models (LLMs) or other generative AI tools."
Proposal B is "The Debian project allows AI-assisted contributions (partially or fully generated by an LLM), provided the following conditions are met [...]"
Proposal C is "request that all contributors to Debian avoid the use of LLMs in their Debian work" without an outright ban.
Of the 3 proposals, the 2nd/B seems to be more of an “informed consent” model while the other two are seeking a comprehensive exclusion. I hope earnest dialogue and driving to a broad consensus among significant contributors is forthcoming.
Notable are the endorsements, which are balanced over all three options, perhaps indicating only 1/3 are supportive of a more permissive position. If this represents core contributors, this could be worrisome: a clear majority favoring a exclusion, but yet a sizable portion of contributors who don’t want a ban. Losing 1/3 of core contributors would be a significant loss to the project.
I wonder how long software projects will even be able to ban LLM use in an age where exploits are found by LLMs. Like, if you have two forks of Debian and one uses LLMs to fix exploitable bugs and the other one doesn't, the level of security they can offer will be worlds apart. And noone in their right mind would want to use the less secure one. Similar to how noone would want to drive a car that was 100% hand built by humans when we know that machines do a much better job at precision tasks. LLMs are just another tool in the end.
> I wonder how long software projects will even be able to ban LLM use in an age where exploits are found by LLMs.
You can use an LLM to scan your human written code for exploits and patch the relevant ones yourself without any LLM code generation. An LLM is a tool, you can chose how you use it.
Don't ask me how they would know you've used a LLM to "assist" you to find the exploit... I don't even know how they would definitely know if you used a LLM for the code to fix it either.
There are a few of these "maintainer said no AI pulls, so we forked it" in the wild already and I am really interested to see how it all pans out.
Like, okay, sure, maintainer says AI bug monitoring is cumbersome. One fork stops reading the AI bug monitoring, the other leans in. Isn't the latter more likely to find/fix critical/performance/security issues?
>Similar to how noone would want to drive a car that was 100% hand built by humans
Though, of course, this highlights that the market will fragment itself given that there are in fact many humans who will outright refuse machine driven cars despite mounting evidence that they are vastly safer and more efficient in a growing number of situations.
We'll likely see both "individual who won't use AI assisted software gets hacked" and "individual who won't get in a robo-taxi kills jay-walker in broad daylight"
I wonder how they can reconcile the stricter proposals with the LLM usage in kernel development. That seems totally untenable. I mean it all seems untenable, but with the kernel especially.
Also, what about when you inevitably get a situation where a critical vulnerability is discovered, and the only patch available is LLM generated? Do they have to wait to patch until some person who hasn't seen the LLM-generated patch does a clean room implementation?
I understand the objections to LLMs, but rejecting LLM-generated code really doesn't seem realistic.
Kernel development is excluded from this, because it's covered by "Upstream projects using LLMs for development".
On security, I worry that proposal A's "forbid any contributions to Debian written with the use or assistance of large language models", as written, excludes contributions where the LLM assisted in discovering the vulnerability. That's clearly a bad policy, and they should update their wording to clarify that.
I read the question more as, if AI is working out for the kernel, why not a distro? (This is a question with valid answers, like the kernel having more devs paid to handle things while maintaining quality, but it's a fair question)
The proposals which say no LLM generated code mean just for Debian, if upstream allows it then they will still accept it. It’s just contributions to Debian itself which would prohibit or discourage LLM contributions.
> A LLM (...) merely produces syntactically likely combinations of the training data
While doesn't matter much in the rest of the policy, this is a common misconception among AI skeptics. It is not the case for a long time (since RL is used heavily in the training) and a LLM may go beyond its training data.
I think the questionable term is not “syntactically likely” but “merely”. Syntactical likeness is a vast solution space that encompasses the work of a terrible developer and a genius developer. In fact this solution space is a gap wide enough to encompass all coding knowledge and expertise.
I assume RL stands for reinforcement learning, which just means that the weights are adjusted by evaluating some loss function. If the loss function uses the training data (i.e. supervised learning) then the original assumption still holds. So I fail to see how this is a misconception.
Supervised learning and reinforcement learning are mathematically distinct. Supervised pre-training as you said maximizes the likelihood of a token sequence. RL optimizes a policy against an external reward signal or execution environment. For example, RL evaluates code against runtime execution/unit tests/proof checkers (e.g lean). Models learn new strategies and are able to produce novel code if trained in an RL environment.
I don‘t understand what you mean. Supervised learning is reinforcement learning, while not all reinforcement learning is supervised learning (e.g unsupervised learning is also reinforcement learning; i.e. reinforcement learning with unlabelled data).
Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
That would makes sense. This is how AlphaGo became better then the training data, and was the basis for Alpha Zero which used no human data and became even better.
However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.
I might su combining Proposal A (but only for contributions (to Debian) that actually involve the output of LLMs/generative-AI or that blindly trust their results) with Proposal C (for everything else).
However, in the case of item 4 of Proposal C, there are multiple kind of "assistance" that is possible; not all of them involve contributing the output of LLMs (e.g. the assistance might be to check for a mistake such as if a paragraph seems to be redundant or is incomplete or something like that; the result from the LLM might be incorrect but you will have to decide by yourself what (if anything) to do about it when writing your message). Even if some kind of assistance should not be banned entirely, it should still be discouraged (like the rest of Proposal C says). (Also, there are other programs that can sometimes be used for the assistance; e.g. sometimes ordinary spelling/grammar checking will do (although that won't find that an incorrect word that is spelled like a correct word in the correct grammar).)
I do not use LLM/generative-AI for any purpose, and would not accept contributions including their output into my projects, and would discourage other uses. (This seems to be similar to Proposal A and Proposal C.)
Funny because out of all the distros, Gentoo benefits from LLM usage, at least as a power user. I have fixed many-an-issues just having LLMs write patches in a way that portage automatically builds them, or rapidly put together ebuilds. I respect upstream decision though and don't share my changes (and realistically who would want them)
Yes for gentoo and their users it makes sense. Debian is much, much larger and is used pretty extensively in enterprises.
The biggest question, I think, is: does a complete LLM ban interfere with, or slow down, security patches? I think it could, as LLMs are used a lot for vulnerability discovery.
What comprises “assistance?” Getting Claude Code to find bugs, suggest a fix, but a human actually editing the source files and testing the result - is that assistance? Perhaps this policy is as it says: a guiding principle that the community is expected to follow.
> Other projects exploring similar decisions have elicited a common reply: "How will you enforce a ban on LLM contributions?" While enforcement could be a challenge, this is a statement of intent by the Debian community, and we trust this community to adhere to it in good faith.
Proposal A is the end of debian for non-english speakers. For those who don't speak english which is most of the world, using an LLM has become vital, because technical information is not available in their language or is extremely basic. Arch Linux is far more lenient with this.
Un what? There’s several friendly clauses among these proposals including
> 8. Any contributor who feels they cannot write in English without assistance, may write in their native language, and expect readers to use translation tools of their choice. In that case a human-written English summary would be very welcome but is not required. In any case we promise not to shame anyone for any linguistic mistakes.
I would say it leaves communication between individuals in a gray area. And discourages translation work because it has to be done manually. I would say, LLM-translated text on documentation is better than having an LLM translate it again and again on the fly every time someone asks how to do something, because the LLM-translated text can be corrected for everyone to see, whereas conversations with LLMs stay private by default.
For now, I find the quality of LLM translation highly inadequate, I can do much better, and LLMs agree my translation is much better. So I think "no LLM for official documentation" could be a right call, for now. It should be re-evaluated as LLM translation advances.
I believe the bigger risk is not low translation quality but choosing instead to use an LLM in a private by default conversation because the LLM rule discourages translation which causes the no information in native language problem which forces the use of LLMs, and thus the LLM output can contain wrong commands which have bricked some computers on reddit
this doesn't prevent you from reading documentation in whatever language you like. If you're saying this is the end of someone communicating with you through an LLM despite not speaking the language, that should never happen and I've never been in any technical organization where that was considered appropriate, if you want to work in international software projects you need to have at least a functional grasp of English
most documentation is not translated to spanish, for example. working on international projects is the exception, not the rule. Spanish speakers just use LLMs because searching on google in spanish yields no results, because the info they need is in english so google doesn't show it to spanish speakers or by searching it in spanish. Spanish speakers have their search result language set to spanish by default based on IP location and most never change it to english, where the results are far more detailed, and then they almost never search in english.
and they still can do this. Again nobody, by definition, can stop you from consuming documentation using machine translation, what you cannot do per these guidelines is contribute translations or documentation using an LLM or pretend you speak a language to contribute, and that is eminently reasonable.
You will continue to be able to translate the English documentation into Spanish but they will understandably not publish machine translation as authoritative technical documentation.
It doesn't elminate the problems of AI translation, it just shifts it into users in global south countries and makes fixing them harder because there is no transparency in private by default AI chats to know which parts of the translation are wrong
Curious what the "use" or "assistance" of LLM models means. I mostly use Gemini as a front-end to search Google without getting ads or SEO garbage, would that be forbidden?
Asking for good-faith compliance, and failing that, simply waiting for slop enthusiasts to publicly brag about using LLMs to break the rules. All evidence suggests that it is impossible for slop enthusiasts to resist outing themselves for any significant span of time.
Interesting that this makes no distinction from using LLM output and using LLM assistance. An overly strict policy is better than no policy, but I wonder if using llm for conversation and analysis, but not code/config generation, was discussed.
The difference is LLMs are extremely good at analyzing code under the careful hand of an engineer. This is how most vulnerabilities are found now.
Prohibiting that might mean that security falls off.
And, if the concern is code quality, then everything before the code writing shouldn’t matter. A good engineer using an LLM for analysis and then writing a patch by-hand might create a better patch.
A rather mean counter troll to this is rather than submitting any bugs/security flaws to Debian, just immediately zero day them and say it's because of their LLM policy, since the bugs were found by an LLM.
I'm guessing after a few dozen serious risks that are found some people might think twice.
I suspect the debate shouldn't be LLMs or no LLMs, but rather what level of human accountability is required.
We've accepted compilers, static analyzers, and code generators because the maintainer is still responsible for the final result. The interesting question is whether LLMs fundamentally change that responsibility, or just change the kinds of mistakes reviewers need to look for
This set of proposals, are (sorry) just stupid. It's like saying to someone, you are not allowed to saw wood using an electric saw, you must do it by hand. What are we doing here?! LLM(s) are just a tool. Use it as such. You should own the work anyway.
It's more like: food processing tool A is known to be defective. We ask employees of this restaurant and supply chains to avoid food processing tool A. This seems eminently reasonable. The question is whether the tool is in fact defective.
That's just it though. The tool(s) aren't necessarily defective. Just because you can chop off your own hand with an chainsaw or do a botch job of sawing a plank of wood because you weren't watching what you were doing with the electric saw, does not by itself make the tools (Chainsaw, Electric Saw) defective.
The tool makes the food taste bad in ... say ... 20% of the cases, and in 5% of the cases it contaminates the food with salmonella.
Yes you can have the chefs taste the food before serving, and test it for salmonella. But it would be completely reasonable for a regulator to ban such a tool, and even more reasonable for restaurants to have a strict policy against using this tool.
You may as well ban every technological advancement in society then. There are risks to everything we do, build and support. Ban 'em all! Let's go back to the dark ages where everything is safe!
Let's just put this argument to bed shall we? Let's consider another case. Leatherman pocket knives. It has a ~20% chance of cutting you, a 5% change of killing you if it pieces the wrong place in your body. I'm speaking from experience, where I sliced my thumb open (by accident of course!), but it was quite bad that I severed a nerve.
Needless to say, I don't feel compelled to take a moral or ethical stance on the use of such tools -- in this case; pocket knives and their inherent dangers.
Every time no matter how sophisticated or the no. of safeguards you put in place, can still inherently pose risks.
I'm not even arguing about food safety. Just think about the many restaurants that prepare and cook up the Japanese pufferfish. That's not even a tool problem right there, that's a skill and trianing problem. That shit™ will can can kill you if not done right.
Sure, kind of, if the supposed defect is only one of the reasons, and if ‘food processing tool A’ was a hot-button topic that many people have deeply philosophical and ideological opinions about. Oh, and for good measure, if the proposed ban is for ‘food processors’ in general, and a material amount of pushback is coming from a bunch of greybeard industrial cooks that used one particular food processor a couple of years ago, weren’t all that impressed, and have been railing against food processors ever since.
Linus Torvalds said last week that LLMs were just a tool. Lorenzo Stoakes had the impeccable rebuttal that "it's a tool in the same way a nuclear bomb is just a tool" - of course it's useful but there is clearly more to it than that.
I'm happy for Debian to at least consider the other aspects of social impact, ethics, copyright, and maintainability.
Yes, they are. But that’s exactly what Proposal B (the only one without moral panic) says: allow its use, but place the responsibility on whoever uploads it. If you upload broken or unlicensed code, that’s on you. Proposal B is the most pragmatic and realistic option.
Yes. this I agree with. It's teh same thing many corporations like the one I work for are doing as well. Expecting and mandating that even if you use these so-called "AI" tools. that you are a) responsible and b) accountable for the "work".
> This set of proposals, are (sorry) just stupid. It's like saying to someone, you are not allowed to saw wood using an electric saw, you must do it by hand.
Stupid? Since when has there been a lack of clarity in copyright status with electric vs. hand saws?
This has nothing to do with copyright status or copyrighted works. If it did, Debian wouldn't even be a thing. You can't be a puritan about this; How much of Debian can one say, hand on heart, is truly not inspired, or influenced by other people's work, copyrighted or not.
Probably since we all started getting our legal advice from you and every other “I think that it’s illegal so it’s illegal” developer-turned-fake-lawyer on the Internet.
Oh, wait, that never happened.
Proposal A is "expressly forbid any contributions to Debian written with the use or assistance of large language models (LLMs) or other generative AI tools."
Proposal B is "The Debian project allows AI-assisted contributions (partially or fully generated by an LLM), provided the following conditions are met [...]"
Proposal C is "request that all contributors to Debian avoid the use of LLMs in their Debian work" without an outright ban.
Notable are the endorsements, which are balanced over all three options, perhaps indicating only 1/3 are supportive of a more permissive position. If this represents core contributors, this could be worrisome: a clear majority favoring a exclusion, but yet a sizable portion of contributors who don’t want a ban. Losing 1/3 of core contributors would be a significant loss to the project.
You can use an LLM to scan your human written code for exploits and patch the relevant ones yourself without any LLM code generation. An LLM is a tool, you can chose how you use it.
> with the use or assistance
Don't ask me how they would know you've used a LLM to "assist" you to find the exploit... I don't even know how they would definitely know if you used a LLM for the code to fix it either.
>I found this bug with an LLM, now you know and your mind can never be made clean, it's the fruit of the poison tree. You must never patch the bug.
Like, okay, sure, maintainer says AI bug monitoring is cumbersome. One fork stops reading the AI bug monitoring, the other leans in. Isn't the latter more likely to find/fix critical/performance/security issues?
>Similar to how noone would want to drive a car that was 100% hand built by humans
Though, of course, this highlights that the market will fragment itself given that there are in fact many humans who will outright refuse machine driven cars despite mounting evidence that they are vastly safer and more efficient in a growing number of situations.
We'll likely see both "individual who won't use AI assisted software gets hacked" and "individual who won't get in a robo-taxi kills jay-walker in broad daylight"
Also, what about when you inevitably get a situation where a critical vulnerability is discovered, and the only patch available is LLM generated? Do they have to wait to patch until some person who hasn't seen the LLM-generated patch does a clean room implementation?
I understand the objections to LLMs, but rejecting LLM-generated code really doesn't seem realistic.
On security, I worry that proposal A's "forbid any contributions to Debian written with the use or assistance of large language models", as written, excludes contributions where the LLM assisted in discovering the vulnerability. That's clearly a bad policy, and they should update their wording to clarify that.
Why would someone misinterpret this as a final decision?
Proposal D is "Accept AI contributions for Debian specific work"
While doesn't matter much in the rest of the policy, this is a common misconception among AI skeptics. It is not the case for a long time (since RL is used heavily in the training) and a LLM may go beyond its training data.
Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.
However, in the case of item 4 of Proposal C, there are multiple kind of "assistance" that is possible; not all of them involve contributing the output of LLMs (e.g. the assistance might be to check for a mistake such as if a paragraph seems to be redundant or is incomplete or something like that; the result from the LLM might be incorrect but you will have to decide by yourself what (if anything) to do about it when writing your message). Even if some kind of assistance should not be banned entirely, it should still be discouraged (like the rest of Proposal C says). (Also, there are other programs that can sometimes be used for the assistance; e.g. sometimes ordinary spelling/grammar checking will do (although that won't find that an incorrect word that is spelled like a correct word in the correct grammar).)
I do not use LLM/generative-AI for any purpose, and would not accept contributions including their output into my projects, and would discourage other uses. (This seems to be similar to Proposal A and Proposal C.)
They are free to keep pretending that they aren’t receiving covert AI contributions, because I’d bet the farm that they are.
Debian has a little more ‘general interest’
The biggest question, I think, is: does a complete LLM ban interfere with, or slow down, security patches? I think it could, as LLMs are used a lot for vulnerability discovery.
Using LLM encourages data theft or unethical acts.
> 8. Any contributor who feels they cannot write in English without assistance, may write in their native language, and expect readers to use translation tools of their choice. In that case a human-written English summary would be very welcome but is not required. In any case we promise not to shame anyone for any linguistic mistakes.
You will continue to be able to translate the English documentation into Spanish but they will understandably not publish machine translation as authoritative technical documentation.
Prohibiting that might mean that security falls off.
And, if the concern is code quality, then everything before the code writing shouldn’t matter. A good engineer using an LLM for analysis and then writing a patch by-hand might create a better patch.
I'm guessing after a few dozen serious risks that are found some people might think twice.
We've accepted compilers, static analyzers, and code generators because the maintainer is still responsible for the final result. The interesting question is whether LLMs fundamentally change that responsibility, or just change the kinds of mistakes reviewers need to look for
Why do you want Debian to die?
LLMs write bad code / so do people, a lot of really bad code.
LLMs write security bugs / so do people, so many god damned security bugs.
Yes you can have the chefs taste the food before serving, and test it for salmonella. But it would be completely reasonable for a regulator to ban such a tool, and even more reasonable for restaurants to have a strict policy against using this tool.
Needless to say, I don't feel compelled to take a moral or ethical stance on the use of such tools -- in this case; pocket knives and their inherent dangers.
Every time no matter how sophisticated or the no. of safeguards you put in place, can still inherently pose risks.
I'm not even arguing about food safety. Just think about the many restaurants that prepare and cook up the Japanese pufferfish. That's not even a tool problem right there, that's a skill and trianing problem. That shit™ will can can kill you if not done right.
I'm happy for Debian to at least consider the other aspects of social impact, ethics, copyright, and maintainability.
A nuclear bomb is a tool.
And programmers are the biggest tools of them all.
/ba dum tis
Stupid? Since when has there been a lack of clarity in copyright status with electric vs. hand saws?