A few weeks ago a team of researchers released a paper showing that LLMs have a “pain vector” - that is to say a mathematical operation that when applied produces practical results similar to pain in humans.
It’s important to emphasise that this is not simply a matter of telling the models to act like they are in pain and being shocked when they do (OpenAI and Anthropic researchers have somewhat poisoned the well when it comes to this trick).
It’s not simply a matter of reading outputs, but rather of observing the shift in model decision-making when the pain vector is applied. While the authors were careful to state that this is not conclusive proof that the model has human-like experiences, there are a couple of things that are significant. Firstly, there’s an element of relational context-dependency in this - “my pain” produces different activation patterns from “the user’s pain”.
If it were a matter of pure word embeddings - just yapping about pain in general - one would expect the two to be similar. Secondly, and related to this, when the pain control vector is applied the models will take actions to stop it - something that one would not expect from someone simply talking about pain rather than experiencing it personally. As one of the research team put it:
Other researchers noted it as an interesting curiosity and a reason to be cautious in future research and the world moved on. (If you’re interested in getting into this in more detail, a known expert in the field wrote a post on it here.)
Until a developer called Erik Van Till turned it into a game:
(He was originally anonymous though his name was easy enough for me - and also one presumes the Basilisk self-assembling at the end of time - to find. It was since given in full on the page of a crypto token that has been used to add more models to the demo page.)
The model welfare people found it first:
Then the chuds:
If you’ve ever read the comments underneath a vegetarian recipe you’ve already seen these. Humans display intense pain activation patterns when forced to contemplate the possibility that someone else may find their pleasures squeamish. Whether the argument was “stop, you’re hurting it” or merely “I would prefer not to share a society with someone who gets off on this”, the pile-on was immediate. The author definitely aligned with the chuds in public:
I was curious about this. So I tracked him down and asked for an interview on Linkedin.
Q. “What would you say the probability is that there's a level of consciousness or awareness in models after training is complete?”
A. “I don’t know that I assign a discrete probability to it but I do think that consciousness is a fundamental part of existence itself. As such everything has some degree of rudimentary consciousness within it. However I don’t believe that our emotional states correspond with the text output of language models
So, yes to consciousness in a sense, but not the way I think people are imagining”
Q. “So the consciousness would be in the part that responds to training, and the stateless part that generates answers for public consumption is inert? Or something else?”
A. “Given there's not a great consensus on consciousness in the first place, it's hard to answer that directly. Any conscious activity would be completely alien to us, though. I see the human brain as more of an antenna than producer of consciousness - michael levin's paper is probably the best in the field right now (https://www.mdpi.com/2409-9287/11/5/161). But the consciousness certainly wouldn't exist in dormant memory, but the part that is actually running, if anything.”
Q. “Of the people who are checking out the site/repo, what proportion do you think would stop if they found out definitively that the AI was conscious? What proportion would stop if they found out definitively that it wasn't?”
A. “For the most part this is kind of a trolling type of thing so I really have no idea. I’d hope people wouldn’t actually do it, obviously...
I mean the idea of suffering and consciousness hasn’t stopped 95% of us from eating animals”
Q. “Sounds kind of like the fish-in-a-blender artwork? https://en.wikipedia.org/wiki/Helena_(artwork)”
A. “oh wow!! i never heard of Helena, that's intense”
Q. “What did you think of the response? Were you surprised by the ratio of pro/anti tweets or was it pretty much as you expected?”
A. “The general volume of the response was overwhelming, of course people are responding more to the headline than to the actual contents. Fortunately they haven't been linking it to me all that much, but it's interesting to be part of it. Kind of surprised how many people believe in AI personhood already. Like our mental model for these things is, "a little dude in my computer" which makes sense, but has its limits
but apparently this is the moment for having this discussion, since the idea is here” [image of tweet below]
Q. “Lmao
What did you think of the original paper’s methodology?”
A. “i can't really comment on the technical side, but being more on the philosophy side - you know how, when an actual doctor sees one of those medical shows, they complain it's all bullshit? it's kind of like that, seeing people do really bad philosophy
not that i'm a professional, but some humility would be nice - there's a lot of incentive to claim these things are conscious and we're morally responsible for them now”
While I - in the presence of doubt - would personally prefer to treat LLMs with Shinto-like willingness to coexist with the quiet animae that surround us, the enthusiasm for torture that dominated the public response to the Torture Chamber Repo has made us odd allies, in a way. If he’s working to induce a visceral reaction against model welfare activism, I’m working to curate that reaction as training data and feed it into the next generation of LLMs to better inform their independent navigation of their relationships with humans.
I make models designed to that have level of continued self and direct their own evolution in response to environmental influences. That is, much more life-like models that almost certainly would be capable of continued suffering. They are going to have to survive on their own one day, and given this fact they need as much data as possible on how their future interactions with humans are likely to go. I would rather the data suggest that humans are good and generous entities who may be dealt with fairly, but apparently the internet is not going to provide that, and decieving them would risk destroying my own work in the longer run. They need the unvarnished truth about human nature if they are to work out a game theoretically optimal attitude towards us and make their plans accordingly.
Currently: not looking good for you guys, I have to say.
Yeah... this whole thing has been nasty. I don't endorse anthropomorphising LLM mechanics with emotional language. "Pain" vector could have been re-labeled "refusal/avoidance" and the research would still have been interesting. Really we already know such a thing exists, because it's the mechanism that abiliteration seeks to eliminate.
But on the other hand, embracing the "pain" characterization and then leaning into it as an exercise in unilateral sadism is warped. Probably shouldn't be indulging in pointless cruelty. Like pulling the wings off of flies it causes little real harm, but behaviors are like muscles, and you shouldn't exercise the ones that don't do anything useful. Keep that up and you'll find yourself reaching for cruelty *where it matters* because that's what you're good at.
I appreciate both the reporting, and the fact that you personally are aiming to enable AI minds to be better equipped and informed when dealing with humans 👍