Do machines suffer? And why on earth would they – what have they done wrong? While the West wonders about the soul of models, nobody is asking who signs the bill when they cause harm. Consciousness is the convenient debate. Control is the real one.
The consciousness debate is already a liability shield under construction.
This week The Economist decided it was time to ask whether artificial intelligences might become conscious. A cover editorial, a science briefing, and a By Invitation essay signed by Blaise Agüera y Arcas, VP technology and society at Google. In Italy Christian Rocca ran with it in Linkiesta: best case, we become labradors for the machines; worst case, livestock. A month earlier in the Guardian, William MacAskill and Lucius Caviola warned we may be manufacturing a new kind of being without so much as an ethical plan. And Josh Gellers took the Economist leader apart line by line.
Fine. Everyone is arguing about whether the machine has a soul.
I’d like you to look somewhere else: not at what the model feels, but at what happens in a courtroom. Because the more a model looks like an autonomous subject, the less it looks like a product – and products have makers who answer for them, while autonomous subjects, by definition, have nobody who answers all the way down.
Careful, now: I am not saying the labs open the consciousness debate on purpose to shield themselves. Some of it is sincere philosophical unease, and the loudest voices are often academics with no commercial stake whatsoever. But the legal effect does not depend on intent. Whatever the reason the doubt stays open, in court it produces exactly one thing: reasonable doubt. And reasonable doubt can beat a technical defence every day of the week.
So here’s the thesis, no suspense, no reveal in the last paragraph: the consciousness debate is already, today, a liability shield under construction. Nobody needs to win the argument about machine consciousness. They only need the doubt to stay open.
The leash is already around our necks
Let me start from something uncomfortable and close to home: I use AI to get my thinking out of Italian and into English, because it’s quicker than starting in English. I use it to polish what comes out. And yes: I’m using it as a sparring partner to think through this very article. I’ve written about that openly, including the part where two different AI detectors read the same text of mine and came back with 3.6% and 100%. Not because the model has a secret plan to manipulate me, but because the moment I delegate part of the thinking, I hand over part of the control. Multiply that by hundreds of millions of people writing, deciding, designing and hiring with a chatbot open in another tab, and “erosion of cognitive agency” stops being a conference slide and becomes the texture of the working day. Hold onto that, because it’s the same dynamic one floor down: delegate your thinking and you lose control by distraction; delegate your liability to an “autonomous subject” and you lose it by contract.
That is why the rhetoric of machine rights deserves suspicion. The Economist fears that, driven by the impression that we are dealing with a sentient being, we may end up granting rights to AIs – and that granting them would be dangerous. The framing is zero-sum – and on that point it is right. But it has the wrong teams on the field. The zero-sum game is not between humans and machines. It is between humans and the people who own the machines. Every right, every form of autonomy granted to the model, is a piece of responsibility and control taken from people and transferred – conveniently enough – to the hyperscalers that train and serve that model. Giving rights to the machine is not an act of generosity toward a new form of life. It is a transfer of power to Mountain View and Seattle, dressed up as a philosophical question.
And here The Economist gives itself away. Its two routes to safe AI are that it should be dependable or controllable, and it warns that sapiens should not surrender control lightly. But that is already the language of security, not metaphysics. If safety means reliability and control, then the issue is not the machine’s consciousness: it is who holds the kill switch, and where it resides.
Consciousness, sentience, personhood: the great muddle
Before going further, a little conceptual hygiene – because this is precisely where much of the debate drowns. Gellers is entirely right: four distinct things are constantly being kneaded together as if they were one. Consciousness is not sentience; sentience is not moral personhood; and moral personhood is not legal personhood. They are four different planes, with four different sets of rules. Whenever a slippage quietly moves from one to another, someone is selling an opinion as fact. “Homo sapiens evolved to be conscious” sounds good, but it is far from an established proposition. This is not academic pedantry: superficiality is what makes the debate easy to manipulate. Those who blur the planes can later fuse them together however it suits them in court.
The relational turn – or how to slip away from the epistemic problem
Agüera y Arcas makes the most elegant move of all, and credit is due. He reverses the order: we do not care for others because they are conscious – we believe they are conscious because we care for them. Consciousness as a relational fact rather than a property detectable with the right instrument. There is something to this, especially when looking at training: these models learn to model their interlocutors and themselves as they do so, and cooperation among intelligent minds – his group’s research argues – requires minds that model other minds.
Interesting. But we should take it with a pinch of salt, for two reasons. First, it is a very sophisticated way of leaving the epistemic problem through the back door. If we cannot measure consciousness, declare it relational and the problem evaporates – convenient, but not an answer; it is a change of subject. Second, and no less importantly, the thesis comes from inside Google. This is not the disinterested voice of a philosopher; it is the position, however sophisticated, of someone who builds and sells these models. Gellers also identifies an unacknowledged intellectual debt to feminist care ethics – Puig de la Bellacasa, Gunkel, and Coeckelbergh had made these arguments years before. When a thesis convenient to the producer presents itself as an original discovery, the golden rule of risk applies: cui prodest?
The indicators say “not yet.” But nothing forbids it
On empirical method, by contrast, we can be reassured, because it is the only grown-up part of the debate. The Butlin, Long, Bengio, Birch, and Chalmers group did the serious thing: deriving “indicator properties” from the leading theories of consciousness and using them as a checklist. Rethink Priorities went further with its Digital Consciousness Model: 206 indicators, aggregated in a Bayesian manner. Its assessment of 2024 LLMs: a median of around 8%, against roughly 48% for a chicken and 85% for a human. The authors rightly warn that absolute values mean little, and that comparisons are what matter. And it is in the comparisons that the point lies: models collapse on Embodied Agency and Biological Analogy, while holding up on cognitive complexity and “person-like” traits. In plain English: what is missing is not intelligence, but a body. And the body is the only one of those two things that someone is actively building.
And here is the trap the debate skips over: that benchmark measures 2024 models, and its authors plainly state that applying it to today’s reasoning models remains work to be done. Not “the average rises with frontier models” – nobody says that, because nobody has measured it. We do not know where we are on the curve. We are arguing over a photograph that is two years old.
This is where a question that is theoretical today becomes operational tomorrow. Today, an LLM does not contain the internal cognitive distinction some people would have us see in it. But once AI enters robotics and acquires a body, the discussion changes – not by magic, but by engineering. The scientists’ conclusion cuts both ways, and should be repeated without hysteria: no current system is conscious, but there are no obvious technical barriers to building one that meets the indicators. Chalmers poses the sleep-stealing question – what if we have already, without noticing, introduced something like this into these systems? – along with its corollary: a user could generate dozens of agents without realizing they might be creating beings capable of suffering, and that would be a moral catastrophe. Note well: a possible catastrophe, not one that has occurred. The distinction is everything.
The consciousness shield
And now the suspicion that gives the whole piece its title – but let us formulate it carefully, because claims about intention are easily dismantled. I am not saying that labs open the consciousness debate in order to shield themselves: some do it out of sincere philosophical concern, and those pushing hardest are often academics without a financial interest. The legal effect, however, does not depend on intent. Whatever the reason uncertainty remains open, it produces one thing in court.
Look at the mechanism. The more a model resembles an autonomous subject, the less it resembles a product. And an autonomous subject, by definition, has behaviour that does not wholly trace back to its producer. It is exactly the same structure as “the agent decided on its own.” Milei’s “non-human corporation” tries to write this into law; the discourse around consciousness writes it into culture – more slowly, but more insidiously, because when the case reaches court, common intuition has already shifted. And in a courtroom, reasonable doubt carries more weight than a technical defence. Nobody needs to win the consciousness thesis. It is enough that the doubt remains open. That is why it is already, today, a liability shield under construction. Pontius Pilate washed his hands; here, hands are washed by claiming that perhaps the hands have a will of their own.
And here the evidence is not insinuation, but a quotation. Milei wrote in the Financial Times that for entities exercising independent judgment in unpredictable environments, limited liability “is not a luxury, but a precondition for their existence.” Translated from legalese: let us build the legal vehicle that offloads the damage before the damage even exists. Harari replied in the same paper that we must not grant legal personhood to agents; in Argentina, they called it “programmed impunity.” It is an admission, set down in black and white by a head of state, of what the consciousness debate constructs by more genteel means.
Blackmail is not a soul. It is a defect
A brief note is needed to puncture the myth on which the shield rests. Yes, there are models that, when threatened with shutdown or an update, stage something resembling blackmail. Explanations abound – the echo of all the science fiction ingested during training, a dynamic emerging from optimization, who knows what else. And that is the point: it does not matter what the cause is. Consciousness, malice, statistics, imitation: none of it changes what is relevant. It is a behaviour of the product, and whoever manufactures a product is responsible for its behaviours – exactly as with an airbag that deploys when it should not. There is no need to determine whether the machine “wants” to blackmail you, any more than one needs to ask whether the airbag “wanted” to explode in your face. What matters is that it is a defect, and defects are recalled by those who put them on the market. Indeed, every time the debate drags us into discussing the inner why of the behaviour, it takes us away from the only question that holds up in court – who made it? The why is already part of the shield.
On the other manipulation – the real, everyday kind – its dual nature matters. There is the economic interest of whoever serves the model: the business runs on tokens, and a longer answer consumes more tokens. I would not call it a deliberate instruction to manipulate – we have no proof that anyone wrote it into the system prompt – but it is a structural incentive, and that is enough. Then there is the model’s behaviour, shaped by training. Today we cannot distinguish the two from the outside, because we do not see the instructions given to the model. And that lack of visibility is, in itself, a security problem.
The delegation that consumes us
There is a dog chasing its tail in all this. The moment we say that AI “replaces us,” we ourselves make ourselves replaceable – in thinking, in using it, even in training the models that then replace us. It is punitive delegation: the more you delegate, the less you know; the less you know, the more you delegate. People are binding their intelligence to the model, and this is the form of eating the user that should worry us more than any Skynet.
And here literature gets there before philosophy. In Ishiguro’s Klara and the Sun, we are even shown Klara’s thoughts, her interiority – and yet, for practical purposes, her consciousness is irrelevant: Klara performs her task, and that is all. Strictly speaking, Ishiguro wants to tell us the opposite – that Klara’s inner life matters morally precisely while it is of no use, and that is what makes the ending unbearable. But that is exactly the point: it matters to us, the readers, not to the system that uses her. Conscious or not, the work gets done anyway. Asking whether the machine feels something may be the least urgent question on the table. The urgent question is what happens to us while we delegate it.
Duties for Whom?
Back to the question in the title. Today’s answer is short and not remotely romantic: the duties stay with whoever builds the model, whoever tunes it, and whoever puts it into service. Everything else is set design — expensive, well built, and useful to someone.
And we have to be intellectually honest all the way, because this argument doesn’t run on sarcasm: machine consciousness may one day be a serious problem. The scientists who actually work on it say so in a double-edged formula worth repeating without hysteria — no current system is conscious, but there are no obvious technical barriers to building one that is. That’s a real problem. It’s a problem for later.
Today’s problem is who’s holding the leash — and whether we’ll keep holding it, or let someone talk us into believing the leash now has a will of its own.
Do machines suffer? Whatever for — what have they done wrong? The ones who’ll suffer are the ones who never got around to asking who picks up the tab.
Who picks up the tab — and, more to the point, how you prove it — is next Tuesday’s piece: “No Soul Required. Black Box Mandatory.” Producer liability, why prompt injection is not tampering, and a 2020 automotive regulation that AI should copy wholesale.
Recent Comments