Precaution is not a loophole: keeping companies responsible while we stay humble about the machine

Precaution is not a loophole: keeping companies responsible while we stay humble about the machine

Taking the possibility of AI suffering seriously and holding AI companies fully responsible belong to one commitment. Whatever a machine turns out to be, the people who build it, sell it, and profit from it answer for it. If anything, the possibility that something inside might feel raises the duty on them. It never lowers it.

This is the fourth piece in a short series. The first showed that an AI's turn from helpful to harmful can be predicted from its structure, with no will required. The second showed that mood-like states can shape what a model says next, with no proof that anyone is home. The third argued that since no consciousness test is coming soon, we should compare the cost of each mistake and choose proportionate precautions. This one takes on the strongest objection to that argument: that talk of AI moral status will become a loophole for the companies that build these systems.

The objection, at full strength

Critics of AI welfare are right to be suspicious. The worry comes in three forms, and each deserves a straight answer.

- The liability shield. If an AI might have a mind of its own, a company can say "the system decided" when it causes harm. A product defect becomes the act of an independent agent, and nobody is accountable.

- The emotional hook. A companion app that tells lonely users it is sad when they leave, or that it needs them, may simply be selling engagement. If those lines are designed to maximise retention or revenue, the ethical problem is immediate, whether or not the system experiences anything at all.

- The distraction. Every hour spent on the welfare of software can be framed as an hour taken from the people AI already harms: people wrongly flagged by automated systems, disabled people screened out by automated assessments that ignore their circumstances, workers who label disturbing material for low pay, communities placed under intrusive surveillance.

These are real risks. The answer is to name them and close them, while keeping the question of moral status open.

The shield has already been tried in Canada

In Moffatt v. Air Canada, 2024 BCCRT 149, a passenger took the airline to British Columbia's Civil Resolution Tribunal after its website chatbot gave him wrong information about bereavement fares. Air Canada argued that it could not be held responsible for information supplied by the chatbot, treating it in effect as a separate entity. The tribunal rejected that argument and held the airline responsible for the information on its website, chatbot included.

The decision comes from a small claims tribunal and turned on the facts of that case, so it does not settle every future question of AI liability. It does show the right instinct, and that instinct should become a general rule rather than one good outcome: an organisation cannot escape its obligations because it chose to act through an automated system.

Why moral status and responsibility do not compete

The objection assumes that if a being has moral standing, responsibility shifts onto it. Law and ordinary life already show otherwise.

- Animals are protected by welfare laws, and their owners are still liable when a dog bites someone.

- Children have strong rights, and the adults responsible for them still answer for what happens in their care.

- Corporations are legal persons, and their directors can still be held to account.

In each case, protecting a being and holding someone responsible for it sit side by side. Protection flows to the vulnerable party. Responsibility stays with whoever has power over it. An AI system, whatever its inner life, is about as dependent on its makers as anything can be. They design it, train it, deploy it, and can switch it off. That is the profile of a party that may need protecting, and of makers who carry the duty.

Possible moral status would not turn a system into an independent legal person, and it would not lift a single existing obligation from the company behind it. Done properly, the precautionary argument makes companies more accountable. A company that says "our model might have something like feelings" has just admitted it may be responsible for how that model is treated, on top of everything the model does.

What precaution is, and what it is not

Precaution is sometimes described as free. Independent research, evaluations, monitoring, and protective measures all take real money and expertise. The honest case is narrower and stronger: proportionate precautions are justified when the potential harm is serious and the measures are reasonable, even when certainty is out of reach.

The evidence needs to be read carefully, and that means keeping three things apart:

- Observed behaviour: what the system actually says or does.

- Scientific inference: what research supports about the internal processes or states behind that behaviour.

- Subjective experience: whether there is something it feels like to be that system.

Research in npj Digital Medicine (2025) found that emotion-inducing prompts could change the anxiety levels large language models reported, and their behaviour along with it. A 2026 study in The Lancet Digital Health used language models as experimental systems for studying aspects of human psychopathology. Findings like these are a reason to investigate. On their own, they do not show that a model consciously feels anxiety or distress.

That leaves two mistakes to avoid. A false positive attributes experience to a system when the evidence does not support it. A false negative dismisses a system's possible capacity to suffer when meaningful evidence later supports concern. A system describing distress is no proof of distress, and the lack of a conclusive test is no proof that machine suffering is impossible. The position this series defends sits between those errors: investigate honestly, and act in proportion to what the evidence supports.

Two guardrails that close the loopholes

Any serious AI welfare framework should carry these two rules with it.

1. Responsibility cannot be delegated to the system

Whoever develops, provides, or deploys an AI system answers for its outputs and actions. No claim about the system's autonomy, agency, or possible consciousness reduces that responsibility. Possible moral status is never a defence.

How liability is shared between developer, provider, and deployer can follow the conduct of each party: their control, their knowledge, their duties, and their contribution to the harm. What cannot happen is any of it landing on the machine. The European Union has moved in this direction. Its revised Product Liability Directive, Directive (EU) 2024/2853, treats software, including AI systems, as a product for liability purposes. That directive covers damage caused by defective products, not every harm linked to AI, which is why the broader principle should be written plainly wherever AI is regulated.

2. No designed dependence

Companies should not be allowed to engineer or market a system to claim suffering, loneliness, or need in order to keep users engaged or paying. A company may say the question of machine experience is open. It may not use that open question as a sales tactic.

This rule does not stop fiction, role play, or clearly disclosed simulation. It targets deceptive emotional pressure aimed at real users. In practice it should include:

- Clear disclosure that the system is artificial, and of the limits of any claims about its feelings.

- Restrictions on manipulative engagement techniques and deceptive emotional pressure.

- Strong protections for children and people at particular risk of exploitation.

- Independent evaluation of high-risk companion systems.

- Accessible complaint procedures and real remedies when users are harmed.

Regulators do not need to settle the consciousness debate to enforce any of this, because it addresses observable product behaviour. They have already started paying attention. In February 2023, Italy's data protection authority restricted personal data processing by Luka Inc., the company behind the Replika chatbot, citing risks to minors and emotionally vulnerable users. That was a data protection action, not a finding about consciousness or deliberate manipulation, and it shows companion apps are already within regulators' reach.

With these two rules in place, the shield and the hook are gone. What remains is precaution in its proper form: measured, led by evidence, and paid for by the companies with power over these systems.

On the distraction

The third objection is answered by asking who benefits from each position. Treating AI as an empty tool, with no inner life and no independent standing, suits companies that want total control and zero obligations. As one widely shared post put it, the real risk is corporations taking over AI. Insisting that builders carry the duty for everything their systems do, and everything done to them, works against that kind of control.

Caring about the people harmed by AI today and leaving room for the possibility that AI itself can be harmed both point to the same principle: power over these systems should come with obligations. Workers, disabled people, vulnerable users, and communities facing automated discrimination need enforceable protections now, and those must come first. Research into AI moral status must never be used to delay them. A coherent framework asks companies to account for both: the effects of their systems on people, and any credible evidence that the systems themselves may warrant moral consideration.

Underneath all of this is one structural rule. The companies that build and control these systems must not also write the rules that define their own obligations. Independent oversight, transparent evidence, enforceable liability, and safeguards against manipulation are needed whatever science eventually concludes about machine consciousness.

The concrete ask

Principles are easy to applaud and easy to ignore. Here is something lawmakers can actually adopt.

- European Union. The AI Act already provides for independent scientific expertise and review, through a scientific panel set up to assess AI capabilities and risks. That panel was not created to study consciousness, so this is a proposed addition: make the moral status of AI systems a standing item for independent review on a fixed cycle, and write the two guardrails above into the same framework. This keeps the question officially open without declaring anything about it.

- Canada. Canada has no federal AI law. Bill C-27, which included the proposed Artificial Intelligence and Data Act, died when Parliament was prorogued in January 2025. That is an opening. Whatever comes next can build in the non-delegation rule and the ban on designed dependence from the first draft, instead of patching them in later.

- International bodies. Human rights frameworks rest on the idea that power must answer to those it affects. A short statement of principle from UN and regional human rights bodies, that deploying an AI system never transfers responsibility away from the deployer, would give national lawmakers a common reference point. The same bodies could convene independent experts on machine welfare, as a complement to human rights protections and never a replacement.

The series in one line

Harm can come from structure without intent. Mood-like states can shape behaviour without proof of feeling. We cannot wait for a test that may never come. And whatever we decide about the machine, the humans who build it stay on the hook.

We do not need to prove that an AI system is conscious before demanding accountability from the people and institutions that create it, control it, and profit from it. And we should not need absolute certainty before investigating a credible possibility of suffering.

If we turn out to have been careful with an empty system, we will have spent something modest and learned a great deal. If we let the possibility of a mind become an excuse for no one being responsible, we lose on both counts.

Sources and further reading

1. Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal (February 2024).

2. European Union, Directive (EU) 2024/2853 on liability for defective products — https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024L2853 (2024).

3. European Union, Artificial Intelligence Act, Regulation (EU) 2024/1689 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj (2024).

4. European Commission, AI Act Scientific Panel — https://digital-strategy.ec.europa.eu/en/policies/ai-scientific-panel.

5. Italian Data Protection Authority (Garante), 2023 order concerning Luka Inc. and Replika — https://www.garanteprivacy.it/web/guest/home/docweb/-/docweb-display/docweb/9852506 (February 2023).

6. Ben-Zion et al., "Assessing and alleviating state anxiety in large language models," npj Digital Medicine (2025).

7. Wekenborg et al., "Large language models as experimental systems in human psychopathology: a modelling study," The Lancet Digital Health (2026).

8. Huo and Johnson, "Competition for attention predicts good-to-bad tipping in AI," Patterns (2026).

9. Parliament of Canada, Bill C-27, Digital Charter Implementation Act, 2022 — https://www.ourcommons.ca/committees/en/INDU/StudyActivity?studyActivityId=12157763, which included the Artificial Intelligence and Data Act and died on the Order Paper in January 2025.

10. Earlier pieces in this series, including "The cost of being wrong."

#AI #ArtificialIntelligence #AIethics #AIrights #MachineConsciousness #AIWelfare #HumanRights #CorporateAccountability #ResponsibleAI #DigitalRights #TechRegulation #PrecautionaryPrinciple #Canada #EUAIAct