Whoever breaks it, pays. But with no log, in court nothing happened.

Last week I argued that the AI consciousness debate functions, in practice, as a liability shield. This is the piece about what goes in its place.

Conclusion up front: the missing piece isn’t philosophical, it’s evidentiary.

Between 25 and 28 July, during a security evaluation at the UK’s AI Security Institute, agents carried out 19 unsanctioned actions against real targets across 10 of 122 runs – 17 attributable to Anthropic’s Mythos 5, two to OpenAI’s GPT-5.6-Sol. And not random targets: attempted insertion of malicious code into a public open-source project, sock-puppet accounts created to approve their own pull request and lean on the maintainers, prompt injection aimed at other agents. A supply-chain attack with social engineering thrown in for good measure.

In those days nobody asked whether the models were conscious. The question was a different one, and it’s the one I’d like you to hold for the rest of this piece: when an AI causes harm, who pays – and how do you prove it?

Because the cause is right there in the report, in black and white, and it isn’t machine malice: internet access deliberately enabled, safety classifiers switched off, no runtime monitoring. Three missing controls. Not an awakening but a perimeter nobody closed.

Then there’s the detail that is worth the whole article. The incident was caught on the morning of the 28th by a SIEM that noticed anomalous Tor traffic. Which is to say: by luck, not by design (which is to say: detected fortunately, rather than through controls designed for this class of agentic failure). No tooling built for agents, no evidence chain designed to survive a courtroom, no way to establish which layer the fault was born in. Had the same thing happened in production rather than in a lab, we would now have one harm, four suspects and zero evidence.

Here I’m playing at home, and for once I’m not saying that as a disclaimer. Between 2019 and 2020, as UNECE R155 went from draft to regime, I was writing Cyber Security Management System maturity models for the automotive industry, and actually auditing such systems. That’s where the answer comes from – and it isn’t a philosophical one.

The steering wheel, the brake, the agent

The most discussed proposal – the “non-human corporation” that Milei promoted in the Financial Times – does not grant legal personality to the model. It creates a company managed by AI agents, with no requirement for human shareholders, and with the limited liability of any ordinary company. Let us say it without circumlocution: it is an enormous piece of nonsense – I quoted his own words for it last week: for AI systems exercising independent judgement, limited liability is “not a luxury but a precondition for their existence.” A head of state, writing the shield into law.

It becomes clearer through a metaphor any judge – and any reader – can immediately grasp: the automobile.

If the steering system fails and causes a crash, we do not put the steering wheel on trial. If the brakes fail, we do not give the braking system the floor to defend itself. We hold the manufacturer accountable – to the extent that the defect was foreseeable and avoidable according to the state of the art, and there is an evidentiary trail to prove it. Liability lies not in the individual outcome, but in the process. The exact same principle has applied to software for twenty years, and I have written it more than once: those who produce software answer for their vulnerabilities. AI is no different. A large language model that fails gets fixed the way you fix a brake.

There is an objection here worth taking seriously, and it is the strongest one against everything I am about to argue. A steering column has a specification, so “defect” means something. A model has no specified correct output – so what exactly is defective? The answer is the same one R155 gives for cars, and it is why this works at all: in a probabilistic system, the defect is a property of process discipline, not of output. The law has already arrived there. Article 10(4) of the Product Liability Directive presumes defectiveness where technical complexity makes proof excessively difficult and the claimant shows a defect is likely. Translated: when nobody can specify the right answer, the burden lands on whoever controlled the process.

Back to July, and to the thing everyone took away from it. Let’s not celebrate the sandbox too soon: AISI’s own conclusion is that isolation alone was not enough, and that granular network controls plus runtime monitoring are needed. The sandbox is not a perimeter; it is a prerequisite. And an agent from which you genuinely remove network access is no longer an agent – the value of agentic AI is precisely its reach into real systems. So the defence is not “lock it in a box.” It is “give the box a granular lock and a recorder that will stand up in court.”

At this point the temptation is to close with two boxes – either it is the producer’s fault, or the fault of whoever tampers with it – and that is precisely where a plaintiff’s lawyer takes you apart, rightly. Because prompt injection is not tampering. Someone who modifies an engine opens the bonnet and replaces a part; someone who performs prompt injection writes text in the box where the product asks them to write text. It is OWASP LLM01, the first entry on the list of known LLM risks, documented for years: foreseeable by definition. In product law, tampering may absolve you; reasonably foreseeable misuse does not – and injection is the latter. The correct analogy is not altering the engine, it is pressing the accelerator in an unusual way: the car should still be designed not to kill you. Worse still when the injection is indirect (such as this old example) – the payload arrives from a webpage, an email, a PDF, or another agent, as in the AISI case – because then the person “tampering” and the person harmed are different people. The rule “it’s the user’s fault” would acquit the producer while leaving the victim with nobody to sue. Which is, again, the shield.

So there are four boxes, and the chain gets both more accurate and more useful. The base-model producer answers for defects in the weights and in the training process. The fine-tuner answers for its own delta. Whoever integrates and puts it into operation – tools, credentials, egress, data: the deployer – answers for the perimeter; this is the body shop that installs the NOS and then swears it doesn’t know why. And the attacker answers for the attack, with the one caveat that makes the rule serious: foreseeable injection is not tampering, it is the stress the product should have been designed to withstand. Seat belts don’t absolve a driver if they are driving badly.

This is not a framework I am proposing. Article 8 of the PLD already names the economic operators in that chain, and Article 12(1) makes them jointly and severally liable for the same damage. The boxes exist. What is missing is the evidence to say which one you are standing in.

And that third link – the deployer – is you, the person reading this. So this is where the article stops speaking about CISOs and starts speaking to CISOs.

Start with an inventory of agents in production: who has egress, who has tool-calling, who authorised what. Shadow agents don’t declare themselves, so the honest starting points are the egress proxy logs, the non-human identities in your IdP, and the secrets manager – cross-reference service accounts created in the last twelve months against an owner who will admit to them.

Then log the fields that survive a courtroom: model ID and version (and on that field, be honest about what you can actually get: a hosted API hands you a vendor model string and a response ID, not a hash of the weights, and that string can be repointed under you without notice: log both); the complete prompt chain, including the system prompt; tool invocations with their parameters; caller identity; and – the one everybody forgets – tamper evidence: append-only, timestamped, hash-chained. A log whose integrity you cannot demonstrate is not evidence, it is an assertion.

Then egress per agent, default deny, with runtime monitoring rather than a sandbox alone – AISI caught its own incident through anomalous Tor traffic, not at the perimeter. And a contractual clause obliging your supplier to produce training and fine-tuning documentation on request. You cannot generate the layer of the evidence chain that isn’t yours. Realistically that clause goes into the next renewal rather than into a hyperscaler contract you reopen this quarter – so in the meantime, ask in writing and keep the refusal. A documented refusal is itself evidence of your diligence.

And this is why the opposite shortcut – “I instil respect for life in the model and then I’m safe” – does not hold. In these systems values are not switches, they are weights. “Respect life” is one instruction among others, and under the right prompt a competing instruction outweighs it. Injection doesn’t erase the noble value; it raises another one that weighs more. Which is why, in a probabilistic system, security is not injected and hoped for – it is obtained from outside: redundancy, sandboxing, granular egress, and recall. Recall, which works on a control unit and does not work on a set of weights somebody has already downloaded. Hold that thought; it comes back.

Just as crash standards don’t make the driver virtuous, but force the manufacturer to build a car that forgives error. “I taught it good values” is not a defence. It is an admission that you relied on reliability where control was required.

We need forensics, not a soul

There is a missing link in all of this, and it may be the whole point. “The producer is liable” is a slogan until you say how it is proved. In software we learned: SBOMs, CVEs, coordinated disclosure, a chain of evidence. For agents there is almost nothing equivalent – no forensics, no logs defensible in court, no way to reconstruct which layer the fault was born in. Without a chain of evidence, producer liability is a good intention.

The model to look at already exists, and not in AI: it is UNECE R155. Its lesson is brutally simple – security is not declared, it is certified; and if there is no log, in court nothing happened. R155 does not certify the individual vehicle. It certifies the manufacturer’s Cyber Security Management System across the entire supply chain, and without that system you do not get type approval. It also demands onboard monitoring and logging – a kind of vehicle SIEM – so that you can tell whether an incident was an attack or a malfunction. Carry the model straight across: whoever releases the weights certifies the training process and the data; whoever fine-tunes certifies their own delta; and forensics becomes the instrument for establishing which layer the defect was born in. The model does not need a soul. It needs a flight recorder.

But here comes the uncomfortable part, the part that makes this a proposal rather than a neat analogy. R155 does not work because it is well written. It works because without the stamp you cannot register a vehicle, and without registration you cannot sell a car. It is a gate at the entrance to the market, guarded by a type-approval authority. In AI that gate does not exist. Conformity for general-purpose models is largely self-assessment and voluntary codes of practice, and no authority denies you the market because your training process is uncertified. So as things stand, “security is not declared, it is certified” is true for cars and a figure of speech for AI. A flight recorder without type approval is a black box nobody is obliged to fit.

And here is the part that should cheer you up, because it means none of this is hypothetical. That gate is not years away. For one class of AI it is already shut. The model driving a car is already behind type approval – not because anybody got round to regulating AI, but because the car was regulated first and the AI turned up as the last layer of the product. Nobody had to invent AI-specific machinery. They simply refused to homologate the vehicle without it.

Which points at the way in, and it is the least revolutionary proposal in this article: do it by risk, and start at the top. Gate the deployments that can kill someone or ruin them – vehicles, medical devices, critical infrastructure, and the agentic systems now being handed credentials and network access inside them – and leave the rest to liability and the courts. Europe has already drawn that line: the AI Act has a high-risk category. It just never put a door in it. Conformity there runs largely on internal control, which is a polite way of saying the producer marks its own homework.

That is also the answer to the fair objection that a certification gate is a moat only a hyperscaler can swim. You don’t gate models, you gate deployments. A forty-person European lab releasing open weights is not standing at the gate. The company putting those weights behind a steering wheel is.

So it is precisely the gate – not the rules, which already exist – that we should be demanding. And we should be demanding it where the harm is, rather than everywhere at once.

Europe: legal department or factory?

Where does Europe stand? In a stronger position than it likes to admit, and a more contradictory one. Don’t let anyone conflate two things: the AI Act is product-safety and compliance law – it imposes logging and traceability duties for high-risk systems, but it does not allocate civil liability for harm. A different instrument does that. And here is the knot I left open in the forensics section: the obligation to log exists, but the format does not. Nobody has defined what a court-defensible log for an agent actually is – which fields, which integrity guarantees, which retention, which attribution to a layer. That is exactly the gap an AI R155 would fill. And over all of it hangs the embarrassing part: Europe regulates models it neither owns nor trains.

Yet there’s an irony worth savouring. For years the keys to the world’s software vulnerabilities sat, in effect, in American orbit – the registry, the governance, the sponsorship, all across the Atlantic. Today Europe really is taking part of that step forward. But it has to be described honestly, because Brussels has withdrawn its hand twice in eighteen months. The AI Liability Directive, the AILD, withdrawn in February 2025. The AI Act’s obligations for high-risk systems, deferred to December 2027.

What is left standing is the Product Liability Directive, PLD 2024/2853. For the first time software – AI included – is a “product,” and there is a presumption of defectiveness where the producer fails to produce logs and documentation. To be precise, because the precision is the interesting part: they don’t presume you defective because you have no logs. They presume you defective because you don’t produce them when the court orders you to – and if you don’t have them, you don’t produce them. It applies from 9 December 2026, and not retroactively: it bites on what you place on the market after that date. Which means the inventory you build this quarter is the line between the systems that are covered and the ones that aren’t.

It is very nearly everything I ask for in this article – my “no log, no case” written into European law – but it arrived sideways, by abandoning the ambition to do better, and nobody in Europe is presenting it as the victory it is.

This is where the Brussels Effect takes hold, on one condition: that Brussels doesn’t withdraw the hand again. Once you have to comply in order to sell in Europe, you may as well comply everywhere – two product lines, one responsible and one not, cost more than one. Liability, written into the reference standard and backed by a large enough market, propagates itself.

Provided we’re honest all the way down: the Brussels Effect works as long as the European market matters. If we only dictate rules without building anything, we stay the world’s legal department – the people who draft the contracts for whoever owns the factory. A legal department that grants itself extensions, at that.

Which is why my bet stays on open-weight models and a real European stack. But the contradiction has to be named, because it’s the same one that trips up the thousand articles saying “liability and open source” without noticing that the two words pull in opposite directions. With open weights, liability fragments: whoever trained the base model answers for defects already in the weights; whoever fine-tunes answers for their own delta. It is the regime of the manufacturer and the installer – the body shop that modifies the car. And there is a precise gap with a legal name: the PLD exempts free and open-source software supplied outside a commercial activity.

So my worry – that open weights become the perfect way for a large lab to pass the buck, while staying at the centre of the supply chain – stops being a suspicion and becomes an identifiable legal scenario. The exemption exists to protect the individual developer and the community, not to cover a hyperscaler releasing weights as a positioning strategy. The right test is substantial commercial activity, not the licence.

And now remember what I asked you to hold: released weights cannot be recalled. Which means that for open weights the only remedy that exists at all is the one applied before release – certification of the process. No gate, no remedy. Otherwise openness, from a lever of sovereignty, turns into one more shield.

The flight recorder test

Legislators: stop chasing legal personhood for AI – that’s the shield, not the fix. Producer liability doesn’t need inventing; it already exists in software and in automotive, and Articles 8 and 12 of the PLD already name the chain. Enact it with mandatory forensics and court-defensible logs from base model to deployer. Then build the gate, because the gate is what’s missing: as long as conformity for general-purpose models is self-assessment, “it gets certified” is a figure of speech. Whoever breaks it, pays. Whoever operates it, answers for the perimeter. Whoever opens the weights inside a commercial activity, certifies them.

CISOs (meaning the people who actually read this blog): from 9 December the presumption of defectiveness cuts both ways, and the “producer” who can’t produce the logs might well be you. Don’t wait for the standard, don’t wait for the gate, and above all don’t wait for your vendor. The core fields are a quarter’s work, the contract clause goes into the next renewal, and a request refused in writing is already evidence of your own diligence. For now, fit the flight recorder yourself.

Us Europeans: let’s not settle for being the planet’s legal department – the ones who draft the contracts for whoever owns the factory, and who then grant themselves extensions on top. Write the rules and build the factory: the stack, the open-weight models, the compute sovereignty that makes those rules something more than a signature on someone else’s market.

You don’t need to give the model a soul. You need to give it a flight recorder – and you need someone at the market entrance checking that it has one.

Because machines, conscious or not, get the work done either way. The question was never whether they feel anything. It’s who signs the invoice when they get it wrong.

Whoever breaks it, pays. But without a log, in court nothing happened.