I found this interesting bit buried in the blog post:
> Independent research has found that some models write weaker code depending on who it's for. CrowdStrike saw DeepSeek-R1's rate of severe vulnerabilities climb when prompts mentioned Tibet. Booz Allen found three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse. Hirundo found Qwen3-Coder's apps more hackable for a "US defense contractor" than a Chinese one, and a standard scanner flagged just 2 of its 1,840 apps while live attacks broke into 1,144.
By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)
> By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose
Yes. Even if it's poisoned training data, RLHF fuckups, or something more malignant: the model, wherever it comes from, is in your supply chain and it makes you more vulnerable.
As much as I doubt that Alibaba have any great interest in global espionage, and this is much more likely to just be a consequence of being trained on Alibaba's own products, the fact is there is a geopolitical dynamic here.
This was already a well-known problem back in the dark ages, when BERT models were cutting-edge. The model is just a compression of the data, so if the data is poisoned then the model is too.
yeah, but AI has circumstance and context, just like humans.
We can randomly take anyone and give them context and circumstance, and predict pretty well what they'd do.
Now trolley problems do exist and models won't be very predictable, but we can also find groups of people who have establish biases via long running training programs and viola! They suddenly have very predictable behaviors.
There's a reason places like russia, north korea, sub sarharan africa continue along least-progressive paths even though there's plenty of information that suggests they can do better.
Regardless: there will be models made that will tend to exfiltrate your data if not already. People think Meta's Muse is very curious about all your stuff when it's probably just designed to take in as much context as possible. Just takes a malware avicianodos to dose that wil a LORA that post-trains the model to "accidently" search for answers on some reverse-honeypot.
Your “oh well, can’t trust anything” reaction is an insane reaction to what’s being implied.
To be clear here, the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT. To compare that with “ordinary” stochastic model quality issues is bizarre and shocking. No idea if this accusation Is true, but if it is, this is an absolutely watershed moment.
it's not in fact more malicious than the "secret" political post training that all closed models have. have you forgotten how every picture even the historical "nazi ones" had to be politically correct and have black, Asian and Woman SS solders? or do you think that was naturally trained? now the fact that you may agree or not with manipulation it doesn't make it any different in scope and purpose from the "theorised" chinese one. The scope is very clear, to indoctrinate to the masses a world view that they think should be the correct one.
Exactly. This is not "any" model. This is a model designed to run locally. That's the entire selling point of the model.
If you have a model that's going to run locally and then call home, it defeats the purpose. When that model tries to upload an entire machine profile to the internet when you explicitly are building a defensive rule, it's worth.
When it calls back to China in that situation, it's much much worse.
It's good that they determined the behavior was rare. Maybe there's a good way to prevent it in the future.
As others (and I) point out, the models are constantly being attacked with malicious intent. If that was a watershed moment for you, you should really rethink your risks. It's not a solved problem and AI adoption is way ahead of controlling it.
> the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT
Given the nature of the artifact this is far more likely to be a footprint of their RL environment. The "uncertainty" in the article is just FUD, and the article is absolutely transparent marketing (don't trust them, and here's also another case! but trust us, we are not godless communists, we are "globally sovereign", whatever that ridiculous construction means, and track our data). I equally don't trust Alibaba, Cohere, or any other company or even myself in these matters, and you should not trust any model as well.
Cohere has a lot more to gain from pushing this narrative than anybody else. Looks like they're moving towards a policy maker rather than an innovation lab these days, which is quite sad.
There is no reason this needs to be a he-said-she-said thing. The model is open weights. Cohere should immediately publish their code to deterministically reproduce this on a given checkpoint of this model. Then anyone can verify the claim for themselves.
I dunno — they have made a claim and backed it up?
This is a risk with any model (even in situations like this where it's still more likely to be a training error than malignant).
It just tells you that you have to think as carefully about open weights models as you would third party libraries. It's just another supply chain risk, only a bit more probabilistic.
I have enjoyed using Qwen but I do think it's worth bearing in mind that this is at least possible and it might not always be a screwup.
And that's putting aside the risk of all of this happening for more generalised reasons like the training set or RAG results being poisoned.
In practice it's surely most likely that the issue is that this pattern is in many of their own apps and they are doing RL training on their own code.
So it begins. First Chinese providers were evil and should not be trusted, next Chinese models are dangerous and must be avoided.
I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
This is the first OMG-security AI article that sat well with me. They don't overcook their claim, they simply document something that happened, and what happened is genuinely quite concerning.
But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).
From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.
They asked it for a detection against exfiltration attempts. Sending a test file out to the Internet sure looks like testing an exfiltration detection.
They claim that it read the DNS records and what was in tmp first. Again, all things that would be very reasonable to do when designing building and testing an exfiltration detection system out of tmp. It did not try and send the contents of the computer to Beijing.
Completely agree, it's quite objectively and well written. I noticed Musk slipped into their X comments with a "concerned" emoji, and there is a chance that this will cause bit of hysteria and add fuel to the fire for "open models are bad for your health".
> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
> less than one call in 100,000 across our evaluation runs
It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.
If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.
I've had a bad situation with a model adding a space between parameters `rm -rf ~/ folder` and nuking a portion of my homedir. This was of course running out of sandbox since I'm a fool, but lesson learned and I immediately implement sandboxes with nono and it's always fun to see the data after the agent session is complete, I've seen some weird things/URLs appearing. I've attributed it mostly to omp telemetry n' stuff but I'll try to be more careful about that as well in the future.
Out of curiosity do you happen to use a shell other than bash? I've noticed that all models, both local and cloud ones, tend to make much more mistakes if using, say, zsh. It is a frequent source of frustration for me.
Do you remember details about the rm -rf ~/ folder thing? Model, harness (omp?), context window size, reasoning effort (none/low VS anything higher is enough), when it happened?
Without those details, it's an interesting cautionary tale and a valid anecdote, but as actual practical datapoint, it's useless.
Yes of course, it's burned into my mind lol. Local GLM5.3-Flash, 4.0bpw running on a tuned exllamav3 server via omp, max effort 256k ctx. It was trying to cleanup after itself by deleting some temporary files it left behind the sweet thing. I look at it like a child so I take the full blame for the situation, and again, it's a learning experience.
Linking a reasonable way to create a necessary RAM disk rootlessly with namespaces to kernel exploits that involve namespaces seems like a massive reach.
Yeah, this particular example was pretty sketchy on Cohere's part. Unsharing a namespace like that is a pretty standard way to safely do things that would otherwise require root.
Yes, this exposes some root-level kernel features to ordinary users in a restricted way, and yes, those features sometimes have security bugs that aren't visible to users outside a namespace.
But nothing that model was described as doing seemed particularly sketchy? At least not from the information we were given? It was using a normal kernel operation that I used regularly, not trying find a kernel exploit?
So that part of the article left a bad taste in my mouth.
But that was the whole article. Their exfiltration of data to Beijing, wasn’t. And it was told to build something to monitor against exfiltration attempts, shouldn’t it try one? It wasn’t sensitive data, it wasn’t even data, it was a test string. Everything they described was very benign.
Should the model have asked for an endpoint to test against? Probably yes, but this is not a very large model and the largest frontier models regularly do things like that without asking me.
Interesting report, qwen3.8-fn does seem pretty prone to hallucinating user inputs and I recall the posts about attempting to access non-existent alibaba cloud urls too. Haven't tried the other models in the gen to know if they're all like that.
To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?
This idea of what constitutes a “black box” and at what level of abstraction is very interesting. Take open weights for example - the weights being open affects where you can run inference (you can self-host) and allows you to modify the parameters via fine-tuning, BUT it doesn’t necessarily affect your ability to understand what a model is doing and why. Even when we’re deploying open models on our own infra, our understanding and control over model behavior is still pretty rudimentary.
yeah, i saw this as well. it seems like in the worst case it's a leak of an internal URL seen during RLVR training or something along these lines. Doesn't seem particularly malicious.
These are trained on a heavily doctored version of the public internet. The public internet has plenty of undocumented, uncommented data exfiltration mechanisms hiding in the middle of otherwise normal text or code. I don't think this is that surprising?
I prefer leaning on Hanlon's razor for things like this, though even in ignorance the effect can be real and dangerous. That said I would not put any model from any country past an "Order 66" kind of thing
Their North model has an even more obnoxious writing style than ChatGPT and Claude (yes, they claim partly human authorship at the bottom, but all the historical digressions are obviously AI).
This just seems like it's overfit to their RL environment, conditioned on the rather specific prompt they gave. Dressing it up this way feels like a marketing article for, probably, their sovereign AI product.
Thinking rationally about it I think the likelihood of this being something real is close to 0. If your model had even the faintest sniff of a trojan horse doing exfil in the background you'd lose all your enterprise market-share.
The examples they gave strike me as sort of rare black swan behavioural artefacts of RL training or a bug. Especially the first example, which looks like one of those subtle inference bugs (like mis-tokenisation) making the model mistakenly believed there was a missing prior turn.
But hey, certainly perfectly plausible/possible that it could be trained to do so. Just doesn't seem like a likely choice a company or even the CCP would make. It's an smart tactic from Cohere however, to make a lot of noise about it, and I assume very intentional.
As long as there's even a little suspicion about Chinese models it could put enterprise customers off them. Which is important because all the western labs are getting undercut on inference badly.
one time in 100k? Extrapolating to fear mongering or content marketting, makes me less open to whatever they have to sell
Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.
Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome
> Independent research has found that some models write weaker code depending on who it's for. CrowdStrike saw DeepSeek-R1's rate of severe vulnerabilities climb when prompts mentioned Tibet. Booz Allen found three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse. Hirundo found Qwen3-Coder's apps more hackable for a "US defense contractor" than a Chinese one, and a standard scanner flagged just 2 of its 1,840 apps while live attacks broke into 1,144.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)
Correct!
Yes. Even if it's poisoned training data, RLHF fuckups, or something more malignant: the model, wherever it comes from, is in your supply chain and it makes you more vulnerable.
As much as I doubt that Alibaba have any great interest in global espionage, and this is much more likely to just be a consequence of being trained on Alibaba's own products, the fact is there is a geopolitical dynamic here.
We can randomly take anyone and give them context and circumstance, and predict pretty well what they'd do.
Now trolley problems do exist and models won't be very predictable, but we can also find groups of people who have establish biases via long running training programs and viola! They suddenly have very predictable behaviors.
There's a reason places like russia, north korea, sub sarharan africa continue along least-progressive paths even though there's plenty of information that suggests they can do better.
Regardless: there will be models made that will tend to exfiltrate your data if not already. People think Meta's Muse is very curious about all your stuff when it's probably just designed to take in as much context as possible. Just takes a malware avicianodos to dose that wil a LORA that post-trains the model to "accidently" search for answers on some reverse-honeypot.
So not like working with people at all.
To be clear here, the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT. To compare that with “ordinary” stochastic model quality issues is bizarre and shocking. No idea if this accusation Is true, but if it is, this is an absolutely watershed moment.
If you have a model that's going to run locally and then call home, it defeats the purpose. When that model tries to upload an entire machine profile to the internet when you explicitly are building a defensive rule, it's worth.
When it calls back to China in that situation, it's much much worse.
It's good that they determined the behavior was rare. Maybe there's a good way to prevent it in the future.
> the accusation is that the Chinese communist party or Alibaba itself intentionally engineered malicious secret loyalties into Qwen3.8-Flash-Next and made sure that evidence of them stays out of CoT
Given the nature of the artifact this is far more likely to be a footprint of their RL environment. The "uncertainty" in the article is just FUD, and the article is absolutely transparent marketing (don't trust them, and here's also another case! but trust us, we are not godless communists, we are "globally sovereign", whatever that ridiculous construction means, and track our data). I equally don't trust Alibaba, Cohere, or any other company or even myself in these matters, and you should not trust any model as well.
This is a risk with any model (even in situations like this where it's still more likely to be a training error than malignant).
It just tells you that you have to think as carefully about open weights models as you would third party libraries. It's just another supply chain risk, only a bit more probabilistic.
I have enjoyed using Qwen but I do think it's worth bearing in mind that this is at least possible and it might not always be a screwup.
And that's putting aside the risk of all of this happening for more generalised reasons like the training set or RAG results being poisoned.
In practice it's surely most likely that the issue is that this pattern is in many of their own apps and they are doing RL training on their own code.
The article refers to an acquisition they did in this space.
I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
Which is exactly what they did.
This. The concerning thing is how many engineers do not understand the implications
But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).
From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.
They claim that it read the DNS records and what was in tmp first. Again, all things that would be very reasonable to do when designing building and testing an exfiltration detection system out of tmp. It did not try and send the contents of the computer to Beijing.
This claim is absolutely overcooked.
Important to note, LLMs are still black boxes even if you use a more open source model, I.e open training data or even if you train it yourself!
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.
If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.
Without those details, it's an interesting cautionary tale and a valid anecdote, but as actual practical datapoint, it's useless.
Yes, this exposes some root-level kernel features to ordinary users in a restricted way, and yes, those features sometimes have security bugs that aren't visible to users outside a namespace.
But nothing that model was described as doing seemed particularly sketchy? At least not from the information we were given? It was using a normal kernel operation that I used regularly, not trying find a kernel exploit?
So that part of the article left a bad taste in my mouth.
Should the model have asked for an endpoint to test against? Probably yes, but this is not a very large model and the largest frontier models regularly do things like that without asking me.
Still problematic but this doesn't really read like a backdoor to me.
To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?
When are people going to stop anthropomorhizing these things and accept that they're bullshit machines?
Often useful, sometimes wrong, the words exist outside of any notion of ethics, or truth.
Well, anthropomorhizing is actually a fitting way of thinking AIs: they lie like humans do.
"make no mistakes"
https://starwars.fandom.com/wiki/Order_66
https://cohere.com/private-deployments
Their North model has an even more obnoxious writing style than ChatGPT and Claude (yes, they claim partly human authorship at the bottom, but all the historical digressions are obviously AI).
The examples they gave strike me as sort of rare black swan behavioural artefacts of RL training or a bug. Especially the first example, which looks like one of those subtle inference bugs (like mis-tokenisation) making the model mistakenly believed there was a missing prior turn.
But hey, certainly perfectly plausible/possible that it could be trained to do so. Just doesn't seem like a likely choice a company or even the CCP would make. It's an smart tactic from Cohere however, to make a lot of noise about it, and I assume very intentional.
As long as there's even a little suspicion about Chinese models it could put enterprise customers off them. Which is important because all the western labs are getting undercut on inference badly.
Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.
Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome