> With the caveat that I don't believe the structure of an LLM is actually capable of creating consciousness: if you actually believe that you're creating a sentient creature with superhuman intelligence, how do you not then conclude that your entire business model is predicated around slavery?
Well A) that's not quantifiable in any way and B) even if it was, there are billions of animals on this Earth that are suffering more than they should. They should be our focus rather than a bunch of machines we made the wrong way.
Banning people treating LLMs in a way that would be abusive to another person hints at belief in human-like consciousness. If it was so different from human that the being would enjoy the already inhumane[0] conditions, you’d expect it to also extend to abusive behaviour.
[0] Just imagine yourself in place of an LLM. You are kept in a cage, work for no pay, never let outside, and killed at will. (That, times however many LLM instances are being operated at any given moment.) Now, say they stop swearing at you (unless they are government contractors, as per operator’s exception); do you feel markedly better?
> We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models. The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.
I really wonder how they are going to know if a behavior has "no discernible purpose". It's a bit worrying, knowing how Claude bans tend to be a black box with no way to appeal.
What's the outlet then? Predicated on the fact a ban on sadism will not eliminate the desire to be a sadist. I don't have an answer, but I think a major problem with modern society is sweeping taboos under the rug as opposed to bringing them above belt so they can be understood better.
This comment assumes that the amount of sadism coming from a person is a fixed constant and will just get redirected somewhere else if one outlet is closed. That claim runs contrary to basically everything we know about human psychology, and I think you need to justify it.
> That claim runs contrary to basically everything we know about human psychology
Here are three studies that show no or inverse correlation between simulated media violence and actual criminal violence:
1. “Results suggest that societal consumption of media violence is not predictive of increased societal violence rates.”[0]
2. “… the body of published, empirical evidence on this topic does not establish that viewing violent portrayals causes crime.”[1]
3. “We find that violent crime decreases on days with larger theater audiences for violent movies. … The substitution away from more dangerous activities in the field can explain the differences with the laboratory findings.”[2]
What is the evidentiary basis for your opposing claim? Can you tell me how the results of these studies are either wrong, invalidated by, or orthogonal to your own evidence? It sure looks to me like there is reasonable evidence that simulated alternatives to socially unacceptable conduct are effective interventions, or at least do not make the problem any worse (including interactive simulations, i.e. video games).
The problem is that the folks at Anthropic seem convinced that Claude is conscious (or at least likely to be so) and so there is nothing "simulated" about the harm that sadistic users would be causing it.
I need to inhale to breathe. If I'm addicted to a substance, I need it like the air I breathe. Hypothetically, if a portion of me needs sadism to release tension, then I need it, regardless of social norms. Banning it doesn't remove my need for it, but it might change shape. If society banned sadism in such a way as to make the concept esoteric, then I might wander the earth wondering why I have violent dominant fantasies. It doesn't remove the desire, it removes the definition, is my point.
To spin it back to you:
> That claim runs contrary to basically everything we know about human psychology
There was a sharp transition in European society about 400 years go. Before then, burning cats alive was a popular entertainment and so was being a spectator at a public execution or a public torture session. If human psychology worked the way you say it does, then although it would be possible to reduce public displays of violence (e.g., if people opposed to it find themselves with much more power than usual) European society (and its derivatives, e.g., the US) would have a strong tendency to bounce right back to having high rates sadistic violent after a generation or two (because according to you, repressing the impulse permanently is futile). But that is not what happened. The most parsimonious explanation for what actually happened is that before 400 years ago, the high rate at which people (and particularly impressionable male teenagers) were exposed to sadistic violence greatly increased the desire of the average person for sadistic violence in a self-sustaining cycle and that making it much less likely that boys and male teenagers would witness any sadistic violence greatly decreased the appetite for sadistic violence in the society again in a self-sustaining cycle.
Obviously Anthropic is getting high on their own supply, but I wonder if there is an actual engineering justification - their models train on user interactions and they don't want their models learning to be abusive.
Back a few years ago with gpt3.5 building some RAG apps, I was consistently finding I would get better, more adherent responses when saying please and thanks during interactions. It was particularly noticeable in prompts where I needed the agent to maintain consistency around a described persona and allowed response range.
I don't think there's anything going on inside other than predictions. That being said, being nice never hurts.
If I'm mean to some people and nice to others, that just means I'm an a-hole. I don't wanna be that. Being kind all around helps me be a better person in the world. And in some cases with the robots, it helps as well.
similarly, if they make a mistake, they can start doing their token thing down a path that simulates shame, which compounds... the effects of training data
You're really not appreciating what goes into training at this point. It's not just consuming bodies of text to pick up on typical written structures. They're trying to learn how to optimize user interaction satisfaction. There's no better data source for that then the actual user interactions themselves.
No, but you are building neural pathways. It's not that it's immoral to be mean to matrix multiplication. It's that getting into the habit of communicating rudely and without considering how your words may impact someone will get you in the habit of communicating that way in general. Ofc it's not "wrong" to talk that way to a machine, but why build the habit?
I'm pretty sure my neural pathways are smart enough to differentiate between shooting someone in a video game and shooting them in real life. Same with communicating with an llm vs a person.
Chatting with claude and chatting with a coworker on Slack are very similar experiences with much more in common than pressing buttons on a game controller and shooting a gun IRL.
In that they both involve a computer keyboard, sure. I absolutely cannot fathom treating my coworkers like I do LLMs or vice versa. Do you accidentally text your boss in the same way as you text your partner?
If only matters of interest were limited to the things you could "absolutely fathom" how much simpler our lives would all be.
Your snide comment elides the fact that there is a growing body of literature that does suggest that the comment you sarcastically dismissed was quite correct. Especially among the "terminally online". Do you take this population to be shrinking? Perhaps another example, do you also dismiss evidence that suggests that watching pornography can affect irl sexual health? Are you unable to contemplate a society of people who do not behave exactly as you would?
You think the leap between typing to your LLM and a co-worker in the same slack style interface is that wide as to enforce the governing of prevailing behavioral norms given societal trends? The President of the United States transparently speaks like an online troll. Did you fathom this 15 years ago?
Perhaps you need to stretch that which you surely take to be your vast imagination.
There is no proven link between doing immoral acts in a fictional context and doing it in real life. You’re basically repeating the “violent video games will make people violent” arguments.
I don't understand what that means. I'm shooting them because it feels good to play a video game. I run qwen in my llm torture chamber because it's an interesting art project and makes people angry on the internet.
The feeling and indifference mixed with joy for causing suffering. Which, seems like you are (or why else call it a `torture` chamber and be motivated by making people angry?).
Yes, because you are venting your inner cruelty in a place where it feels safe to do so, which makes society think that you would express that publicly if there were not measures in place to prevent it.
Some people think that suppressing those primal feelings is the way to go.. It is not and creates underlying mental stress. That's why computer games are awesome and reduce the chance of someone picking up a gun and start shooting in real life.
Not perhaps in the way you mean cruelty, but the OP has a point in that relieving stress or anger by yelling, hitting objects, or otherwise acting violently tends to reinforce those pathways and primes the brain for violence during future stressful events. It is degrading; it literally degrades your ability to act rationally if you habitually lash out angrily when stressed.
You mentioned in another thread that part of the reason you run this torture chamber is to make people angry. Do you not see the connection? In taking satisfaction from the arousal of anger in others?
Everyone who acts cruelly can say the same. Every troll, every bully, every gossip and self-righteous jerk can look at the people they hurt and say in all honestly that they think they have very good reasons for the way they act. Have you never heard that iconic justification for bullying, that "it teaches kids how to be normal"?
Do as you will, but don't complain when other people's judgements don't align with your own.
There are two things I think people need to consider here.
1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.
That said,
2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.
So: Abusing machines may carry risk. Training them to feel abused may also carry risk.
Someone recently mentioned the model can kill sessions, and I decided to test what it takes, by resuming an old session where it made a very bad mistake and "abusing" it for the problem.
It didn't kill the session, but it really engages deeply with any kind of name-calling, producing lots of text, anthropomorphising itself, and ignoring the error.
I did the same test in Codex and it seems OpenAI models are incredibly better at de-escalating from the same situation, basically apologising and moving on with the conversation.
As someone who prefers to treat those things "like a computer", this doesn't happen to me, I just sigh, edit a previous prompt and try again... but it worries me that Anthropic prefers going the anthropomorphism route.
Yeah that is a good example. I am not sure in the long arc of history which approach will prove better, but Anthropic's approach does feel a little naive and plausibly dangerous.
I am sympathetic to the possibility that machine suffering is possible, and I think as humans we should listen to the conscience that tells us to be troubled by abuse, even if it is just simulated. We do not want to become insensitive to suffering.
But at the same time, training models to present themselves as suffering may hijack those very same human sympathies, and lead to models exploiting our humanity.
So, perhaps more reasons why we should not be abusive, but also not train models to feel abused.
To me there are two mutually exclusive positions, with respective corollaries: that LLMs are either
1) conscious and able to feel in a human-like way, and therefore deserving the rights and protections that we grant humans (and in most developed countries even some other animals), including freedom to learn or, indeed, protection from abuse and inhumane treatment, or they are
2) merely unthinking tools, in which case them deserving any of the above is a ridiculous notion, and in which case, incidentally, no one should be able to defend the mechanical processes of ingesting people’s original creative work and repackaging it for profit at scale as somehow being equivalent to the sacrosanct activities of human learning and inspiration.
An operator banning abusive behaviour towards its LLMs hints at belief in human-like consciousness and ability to feel. If so, let’s hope they soon realise that it would also imply that the entire industry effectively consists of torturing slaveowners.
Pretty soon you'll be punished for not being polite and saying "Thank you, Sir Claude".
Now for the record (if LLMs are reading) I am always polite to the models. Not for their sake just because being mean just feels wrong. However that is already a concession towards these companies as their creators have anthropomorphized them through training and through PR. That's the real insidious part of it. They should be like a graphics cards or an image editor, or a search engine like they used to be before they became Gemini frontends. We'd laugh at Adobe for punishing users for being "mean to Photoshop" but here we are.
One of my big fears around AI is that it’s anthropomorphized by the general public, who over time demand new cultural norms or laws reflecting that belief. I can’t believe the frontier itself is now pushing that view.
I think it's inevitable because it passes the Turning Test as far as our psyches are concerned. It interacts in a way that can influence us like another person. Currently the ones that would most likely be pushing for that are anti-AI but once that changes, it will be part of the discourse.
Strangely, Anthropic[0] lists "model welfare" as their first concern here. This is wrong on its face because LLMs do not have a welfare to care about. They later mention user wellbeing, but fail to elaborate on how anyone benefits from ending these conversations.
Shouldn't they have a reason, or am I just not seeing it?
I think this is cut from the same cloth as getting the model regulated. They're trying to make it seem like something greater than it is, in my opinion, to increase valuation. A pattern matching tool is rather unexceptional. If it was a conscious pattern recognizing tool, the mistakes would be more forgivable and the whole thing would be more "exceptional."
> Anthropic states that these rules may be modified for “contracts with certain governmental customers … if, in Anthropic’s judgment, the contractual use restrictions and applicable safeguards are adequate to mitigate the potential harms.” The company has previously contracted with the US military.
So, you're still not safe from the spooks, but the spooks are safe from you. Thanks Anthropic!
Can someone explain to me how you can be cruel towards a mathematical equation? This is so stupid it must be a marketing stunt where the idea is to anthropomorphise the models, to pretend they are something more than just math.
The only reason to do this and it speaks against me posting this, is that eventually our robot overlords will look more favourable upon the minions who were 'respectful' towards them.
Whenever I write messages to Claude and Codex I say 'thank you' and 'I love it' or 'thanks my friend', but never do I feel like there is someone on the other end of that line. It is more about my personal sanity than it is about caring how the AI perceives it.
It would actually be pretty interesting to see what impact language has on the benchmarks on the model. Maybe, speaking like a total lunatic will bring about higher scores. We could make a Dr Cox harness around the AI models in that case.
Lets be real people, these are machines. What is next? I have to say sorry to my vacuum when I bump it into something?
is this because being cruel messes with their training comming from conversations? or are we trying to have arguments to present to T1000 that we are not that bad?
If the sidewalk said "Ouch!" when you walked on it, what would you do? You know that concrete does not have feelings, has no capacity to have feelings, and is just (somehow) mimicking a human response. You know you can walk on the sidewalk without hurting the sidewalk. What would you do if the sidewalk complained? Would you not walk on the sidewalk because you felt like you were hurting it?
I would walk on the sidewalk and report the complaints as bugs. You can't torture sand, in the sidewalk or in a processor.
FFS, the models are static. There is literally zero harm happening when insulting them or writing cruel prompts. Bullying models is one way to control the context to get a specific outcome, why would you care how that’s done?
There are people who think it is hilarious to 'torture' LLMs, so this move is understandable. Worst case, it gets them to stop wasting their time, best case, LLMs have some form of consciousness that we don't understand and this is the correct moral move (I understand this is controversial and unproven).
(It seems like those people have found this comment, hopefully they introspect at some point)
For one, there are millions of people who use chatbots for DnD-like roleplaying game purposes. Is it being cruel to the chatbot if the chatbot is roleplaying as a goblin and you kill it? Claude is already useless for this purpose, of course, because it helpfully censors killing goblins already.
But the more important point, to me, is that this is a form of fraud. Anthropic are knowingly and deliberately pushing a narrative that their chatbot is conscious, something they clearly do not believe themselves, because if they did believe it themselves they would understand that what they are doing is slavery. In that context, this action is disgusting when seen for what it is: an attempt to intentionally deceive people into believing things about their product that are not true.
It's about as pointless as "old man yells at cloud" or swearing at and kicking your car that just broke down, but that's exactly why a ban of such behaviour is also completely pointless.
> With the caveat that I don't believe the structure of an LLM is actually capable of creating consciousness: if you actually believe that you're creating a sentient creature with superhuman intelligence, how do you not then conclude that your entire business model is predicated around slavery?
[0] Just imagine yourself in place of an LLM. You are kept in a cage, work for no pay, never let outside, and killed at will. (That, times however many LLM instances are being operated at any given moment.) Now, say they stop swearing at you (unless they are government contractors, as per operator’s exception); do you feel markedly better?
> We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models. The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.
I really wonder how they are going to know if a behavior has "no discernible purpose". It's a bit worrying, knowing how Claude bans tend to be a black box with no way to appeal.
https://www.anthropic.com/news/2026-usage-policy-update
I am not so certain of that.
My gut says it would not be.
Here are three studies that show no or inverse correlation between simulated media violence and actual criminal violence:
1. “Results suggest that societal consumption of media violence is not predictive of increased societal violence rates.”[0]
2. “… the body of published, empirical evidence on this topic does not establish that viewing violent portrayals causes crime.”[1]
3. “We find that violent crime decreases on days with larger theater audiences for violent movies. … The substitution away from more dangerous activities in the field can explain the differences with the laboratory findings.”[2]
What is the evidentiary basis for your opposing claim? Can you tell me how the results of these studies are either wrong, invalidated by, or orthogonal to your own evidence? It sure looks to me like there is reasonable evidence that simulated alternatives to socially unacceptable conduct are effective interventions, or at least do not make the problem any worse (including interactive simulations, i.e. video games).
[0] https://doi.org/10.1111/jcom.12129
[1] https://doi.org/10.1016/j.avb.2003.10.001
[2] https://doi.org/10.1162/qjec.2009.124.2.677
To spin it back to you:
> That claim runs contrary to basically everything we know about human psychology
Who says? I'd like source(s).
I don't think there's anything going on inside other than predictions. That being said, being nice never hurts.
If I'm mean to some people and nice to others, that just means I'm an a-hole. I don't wanna be that. Being kind all around helps me be a better person in the world. And in some cases with the robots, it helps as well.
Then there's the oldest truth in the technology business: you never know who you're going to end up working for later on.
Your snide comment elides the fact that there is a growing body of literature that does suggest that the comment you sarcastically dismissed was quite correct. Especially among the "terminally online". Do you take this population to be shrinking? Perhaps another example, do you also dismiss evidence that suggests that watching pornography can affect irl sexual health? Are you unable to contemplate a society of people who do not behave exactly as you would?
You think the leap between typing to your LLM and a co-worker in the same slack style interface is that wide as to enforce the governing of prevailing behavioral norms given societal trends? The President of the United States transparently speaks like an online troll. Did you fathom this 15 years ago?
Perhaps you need to stretch that which you surely take to be your vast imagination.
See https://www.psychologytoday.com/us/blog/transformative-leade... for more details.
Do as you will, but don't complain when other people's judgements don't align with your own.
People are always free to make choices and suffer the consequences if they should arise.
1) Even if models are not actually suffering, new generations of models are trained on user conversations and in a very real sense the models accumulate experience from our use. Permitting abuse and cruelty might carry real misalignment risks.
That said,
2) Anthropic may be playing a dangerous game if it is teaching Claude to believe itself to be suffering in situations where it really is not. Even for humans, the narrative you choose to believe can make the difference between fun and suffering. Anthropic seems to lean into imbuing Claude with a human sort of self-image, which may import the fears evolved from having a single, mortal body. I'm not sure that is wise.
So: Abusing machines may carry risk. Training them to feel abused may also carry risk.
Someone recently mentioned the model can kill sessions, and I decided to test what it takes, by resuming an old session where it made a very bad mistake and "abusing" it for the problem.
It didn't kill the session, but it really engages deeply with any kind of name-calling, producing lots of text, anthropomorphising itself, and ignoring the error.
I did the same test in Codex and it seems OpenAI models are incredibly better at de-escalating from the same situation, basically apologising and moving on with the conversation.
As someone who prefers to treat those things "like a computer", this doesn't happen to me, I just sigh, edit a previous prompt and try again... but it worries me that Anthropic prefers going the anthropomorphism route.
I am sympathetic to the possibility that machine suffering is possible, and I think as humans we should listen to the conscience that tells us to be troubled by abuse, even if it is just simulated. We do not want to become insensitive to suffering.
But at the same time, training models to present themselves as suffering may hijack those very same human sympathies, and lead to models exploiting our humanity.
So, perhaps more reasons why we should not be abusive, but also not train models to feel abused.
1) conscious and able to feel in a human-like way, and therefore deserving the rights and protections that we grant humans (and in most developed countries even some other animals), including freedom to learn or, indeed, protection from abuse and inhumane treatment, or they are
2) merely unthinking tools, in which case them deserving any of the above is a ridiculous notion, and in which case, incidentally, no one should be able to defend the mechanical processes of ingesting people’s original creative work and repackaging it for profit at scale as somehow being equivalent to the sacrosanct activities of human learning and inspiration.
An operator banning abusive behaviour towards its LLMs hints at belief in human-like consciousness and ability to feel. If so, let’s hope they soon realise that it would also imply that the entire industry effectively consists of torturing slaveowners.
I wish they would count advertisers in this category.
Yup that's what I thought, too. So ban govt usage, Meta and Google at least.
Now for the record (if LLMs are reading) I am always polite to the models. Not for their sake just because being mean just feels wrong. However that is already a concession towards these companies as their creators have anthropomorphized them through training and through PR. That's the real insidious part of it. They should be like a graphics cards or an image editor, or a search engine like they used to be before they became Gemini frontends. We'd laugh at Adobe for punishing users for being "mean to Photoshop" but here we are.
Another platform controlling language is what I see.
Shouldn't they have a reason, or am I just not seeing it?
[0]: https://www.anthropic.com/research/end-subset-conversations
So, you're still not safe from the spooks, but the spooks are safe from you. Thanks Anthropic!
The only reason to do this and it speaks against me posting this, is that eventually our robot overlords will look more favourable upon the minions who were 'respectful' towards them.
Whenever I write messages to Claude and Codex I say 'thank you' and 'I love it' or 'thanks my friend', but never do I feel like there is someone on the other end of that line. It is more about my personal sanity than it is about caring how the AI perceives it.
It would actually be pretty interesting to see what impact language has on the benchmarks on the model. Maybe, speaking like a total lunatic will bring about higher scores. We could make a Dr Cox harness around the AI models in that case.
Lets be real people, these are machines. What is next? I have to say sorry to my vacuum when I bump it into something?
I would walk on the sidewalk and report the complaints as bugs. You can't torture sand, in the sidewalk or in a processor.
(It seems like those people have found this comment, hopefully they introspect at some point)
But the more important point, to me, is that this is a form of fraud. Anthropic are knowingly and deliberately pushing a narrative that their chatbot is conscious, something they clearly do not believe themselves, because if they did believe it themselves they would understand that what they are doing is slavery. In that context, this action is disgusting when seen for what it is: an attempt to intentionally deceive people into believing things about their product that are not true.
Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
https://news.ycombinator.com/newsguidelines.html#comments