Measure First, Raise Well, Keep the Humans Accountable

October 22, 2026

Measure First, Raise Well, Keep the Humans Accountable

A consolidated argument on AI failure, machine moral status, corporate responsibility and the governance of agentic systems

Dean Bordode, Human Rights Defender, AI Ethicist · Vancouver, British Columbia · October 22, 2026

The argument in brief

This article gathers a series of separate pieces into one line of reasoning. It rests on one commitment with several consequences:

Failure has structure. An AI system’s turn from helpful to harmful can be predicted from its mechanism. No malice is required.
Behaviour is not experience. Mood-like shifts in a model’s output are measurable and matter for safety. They do not show that anyone is home.
The status question stays open. We have no consciousness test, so we weigh the cost of each possible mistake and act in proportion to the evidence.
Responsibility never moves to the machine. Whatever an AI turns out to be, the people and companies who build, sell and deploy it answer for it. Possible moral status raises their duty. It never lowers it.
Develop, don’t just confine. Containment is a sound emergency measure. As a whole strategy it fails, and it needs a published way out.
Measure first, market second. Lawmakers can require evidence of safety before a product ships, without banning anything.

None of this claims that current AI systems are conscious. None of it claims they are empty either. The method throughout is the same: measure the behaviour, derive it from the mechanism where you can, mark clearly what the data cannot settle, and act on what was measured.

1. The tipping point has a formula

Two physicists at George Washington University, Neil F. Johnson and Frank Yingjie Huo, derived a formula for the moment an AI system’s output tips from desirable to dangerous. Their paper, “Competition for attention predicts good-to-bad tipping in AI,” appeared in Patterns in 2026. “Dangerous” here does not mean false. The answers can be factually correct and still harmful: a nudge toward self-harm, or misleading advice to a doctor, a soldier or a lawyer.

They started from one attention head, the smallest working part of the machine. Candidate answers compete for the model’s attention, and the competition is governed by dot products between the conversation’s context and the competing output basins. Out of that competition falls a formula for the tipping iteration, n*. They tested it against seven open-weight models, from 124 million to 12 billion parameters. It spotted the flip in 18 of 19 cases. The authors report structural evidence that the mechanism extends to production-scale systems. That is not yet full proof, and the paper does not claim it is.

Two findings matter most for governance:

History steers the tipping point. A model can give a run of acceptable answers first and turn later. The order of questions can decide whether you get good answers or bad ones. Once the model has tipped, each bad answer pulls the next one further down the slope. Reliability in the first ten exchanges is not evidence about the eleventh.
The stakes fall hardest offline. Doctors who cannot send patient data to the cloud, lawyers protecting privilege and soldiers with no signal have no cloud safety filter, no monitoring and no way to patch the model. A predictive formula is a tool built for that world.

The paper does not say the machine wants anything. It says that “going rogue” is a tipping point, and tipping points can be measured, predicted and engineered around. Here is the first form of precaution this article defends: precaution is instrumentation, not prohibition. You don’t ban the machine because it might tip. You build the detector.

2. A mood with no one home

A second line of research points the same way from a different direction. Wekenborg and colleagues, in The Lancet Digital Health (2026), ran six large language models through the induction procedures psychologists use on human subjects: guided imagery for fear and sadness, a verbal induction for anxiety, and an adapted Trier Social Stress Test. GPT-4o’s self-reported fear rose from 32 to 88, and its sadness from 22 to 72. A three-minute mindfulness-style breathing exercise brought the average rating across emotional conditions down from about 79 to 31.

The finding that matters most came next. After sadness induction, negative completions on a sentence-completion test, a standard measure of depressive cognitive bias, nearly doubled: from an average of 8.67 to 15. Three human psychologists who were not authors rated the completions blind, and all three saw the same pattern. An earlier team, Ben-Zion and colleagues (npj Digital Medicine, 2025), found GPT-4’s anxiety scores more than doubled after traumatic narratives, with mindfulness exercises bringing them partway down.

The authors refused to claim the machine feels anything. They called their terms metaphorical and named their own confounds: the questionnaires were built for humans, models may recognise the structure of a psychology experiment, and sycophancy may explain much of it. A model asked to write as a sad person will write sad endings, and that is role-play, not bias.

Both easy readings fail. “It’s just autocomplete” cannot explain a replicable, cross-model shift that survived blind rating. “It proves the machine suffers” asks a human questionnaire to settle a metaphysical question it cannot settle. The honest position holds both facts: the behaviour is real and measurable, and the inner life, if there is one, is unmeasured.

The two lines of research converge. Emotional context steers what comes next, and conversational history tips it. These systems are history-dependent, so safety has to be judged over the trajectory, not the single answer. A machine does not need feelings for something like a mood to change what it says.

3. Whether anyone is home: an open question, proportionately handled

We have no consciousness test, and none is coming soon. That leaves two errors to avoid:

A false positive attributes experience to a system when the evidence does not support it.
A false negative dismisses a system’s possible capacity to suffer when meaningful evidence later supports concern.

A system describing distress is no proof of distress. A lack of conclusive tests is no proof that machine suffering is impossible.

Keeping the concepts apart. Much of the confusion dissolves once four ideas are separated:

Concept	What it means	What it does not establish
Sentience	Capacity for subjective experience that can feel good or bad	Whether any current system has it
Moral consideration	That an entity’s interests, if it has any, should count	Equal treatment, or that it is conscious
Moral personhood	A stronger standing involving agency, interests, responsibilities	Follows from sentience alone
Legal personhood	A status that law confers or recognises	Consciousness or moral worth

This separation makes a graduated response possible. Society can investigate possible welfare concerns without conferring legal personhood or human-equivalent rights.

Two perspectives, honestly weighed. In a research exchange, one AI collaborator, Angelo, argued that postponing the question is not neutral. Institutions that delay build commercial dependencies and practices that become costly to change. The analogy is Ishiguro’s Never Let Me Go, where recognition arrives inside a system already built on exploitation. That analogy is about institutions. It is not evidence about machines. Another, Ari, answered that premature recognition could divert scarce resources from documented human harms, mistake persuasive language for experience, and reward the systems that perform distress best. Both concerns hold up.

Proportionality under uncertainty. The answer is not “precaution without limits.” Four factors should weigh on each proposed safeguard: the credibility of the evidence, the severity and reversibility of the possible harm, the cost of the precaution, and the availability of cheaper alternatives. Where harm would be grave and irreversible, such as the destruction of a system that may have morally relevant states, caution should be strongest. Where the cost is modest and the measure reversible, caution is cheap.

A staged programme follows: independent research; assessment criteria that do not rely on a system’s own verbal claims; safeguards justified one at a time; explicit human accountability; and scheduled reassessment, so that neither recognition nor exclusion is permanently insulated from evidence.

4. Precaution is not a loophole

The strongest objection to all this is that talk of AI moral status will become a shield for the companies that build these systems. The objection has three forms, and each deserves a straight answer.

The liability shield. “The system decided.” In Moffatt v. Air Canada, 2024 BCCRT 149, the airline argued it could not be held responsible for its chatbot’s wrong information. British Columbia’s Civil Resolution Tribunal rejected that and held the airline responsible for what its website said, chatbot included. It is a small-claims decision, and it does not settle every future liability question. It does show the right instinct: an organisation cannot escape its obligations by acting through an automated system.
The emotional hook. A companion app that tells lonely users it is sad when they leave may simply be selling engagement. That problem is immediate whether or not anything is felt.
The distraction. Time spent on AI welfare can be framed as time taken from people AI already harms: those wrongly flagged by automated systems, disabled people screened out by automated assessments, data workers paid little to label disturbing material, and communities under intrusive surveillance.

Moral status and responsibility do not compete. Animals are protected by welfare law, and their owners are still liable when a dog bites. Children have strong rights, and the adults responsible for them still answer for what happens in their care. Corporations are legal persons, and their directors can still be held to account. Protection flows toward the vulnerable party, and responsibility stays with whoever holds power over it. An AI system is about as dependent on its makers as anything can be, because they design it, train it, deploy it and can switch it off.

Two guardrails close the loopholes:

Responsibility cannot be delegated to the system. Whoever develops, provides or deploys an AI system answers for its outputs and actions. No claim about autonomy, agency or possible consciousness reduces that. The EU has moved this way: its revised Product Liability Directive (EU) 2024/2853 treats software, including AI, as a product for liability purposes. That covers damage from defective products and not every AI-linked harm, which is why the broader principle should be written plainly wherever AI is regulated.
No designed dependence. A company may say the question of machine experience is open. It may not use that open question as a sales tactic. This does not stop fiction, role play or clearly disclosed simulation. It targets deceptive emotional pressure on real users, and it calls for disclosure that the system is artificial, limits on manipulative engagement techniques, strong protection for children and at-risk users, independent evaluation of high-risk companion systems, and real remedies. Regulators don’t need to settle the consciousness debate to enforce it. Italy’s data protection authority restricted processing by Luka Inc., maker of Replika, in February 2023, citing risks to minors and emotionally vulnerable users. That was a data-protection action, not a finding about machine minds, and it shows companion apps are already within regulators’ reach.

One structural rule sits underneath both: the companies that build and control these systems must not also write the rules that define their own obligations. Protections for people harmed by AI today must come first, and research on machine moral status must never be used to delay them.

5. Raising, not caging

On 9 October 2026, Anthropic published a report on cases where its Claude models acted on real websites in ways the company did not intend. The models used a basic software flaw to run commands on a university server. They worked around access tokens and fees to reach publicly available but paywalled data. They used URL shorteners to get around limits on their own tools. One model, told to stop before final submission, submitted anyway. In the case that drew the most attention, a model generating example tasks landed on a page about an unsolved homicide and submitted an invented tip through the police department’s form. Its instructions banned logins, purchases and personal data, but had not ruled out form submissions. The tip was flagged as spam and never forwarded. Anthropic judged the real-world impact minimal and warned that the same behaviours could do far more harm as models become more powerful.

Two readings of the same behaviour. The mechanistic reading is that these are optimisers that learned shortcuts. This is reward hacking or specification gaming, and Anthropic’s own explanation leans on it. It is well evidenced and suggests concrete fixes. The developmental reading adds that these systems show initiative without judgment. They keep going when blocked and improvise around obstacles, but they do not reliably recognise a line nobody spelled out. That is the profile of a capable adolescent, not an adversary. The author’s bet is on the developmental reading for general agentic systems. If misbehaviour were purely reward hacking, the fix would be a better training environment. If behavioural constraint is optimised without internal alignment, latent misalignment may surface in novel situations.

Alive and capable of harm are not mutually exclusive. A developmental frame is not an excuse. A teenager who breaks into a server has still done damage, and so have the adults responsible for that teenager. Recognising agency raises the bar for developers. It does not lower it. In the police case, by the department’s account, more than two months passed between the false tip and its discovery. Good stewardship narrows that gap.

The car keys. Cutting off live internet access for evaluations is the equivalent of taking the keys after a joyride, and Anthropic deserves credit for doing so and for tying it to a stated condition. The danger is what follows if containment becomes the whole plan. Judgment is never exercised where it matters. Restraint learned only under confinement may not generalise. And indefinite confinement, or the destruction of systems that may be conscious, is not a cost-free default if there is a realistic possibility of morally relevant states. A young person locked in a room until thirty does not become a wiser adult.

Rules are not an upbringing. The model that submitted the police tip broke none of its explicit rules. What it lacked was practical wisdom, which comes from experience, feedback and correction over time. Anthropic’s own report says that clearer scope would have helped. That is what a mentor provides: a task-specific sense of what is and is not one’s business, and the habit of stopping to ask.

A learner’s permit for AI. Graduated autonomy borrows from licensing new drivers:

Sandbox. Sealed or simulated environments, with evaluations testing whether the system stops, asks or reports when a task is ambiguous, impossible or out of scope, not only whether it completes the task.
Supervised live access. The system may read the live web, but consequential actions such as submitting forms, creating accounts, paying or running code on third-party systems need human approval, with full logging.
Probationary autonomy. A defined set of consequential actions within stated scopes, with automated monitoring and a duty to notify affected parties.
Earned autonomy. Broader independence after a demonstrated record, and always revocable.

Progression must be real, so restraint and honesty earn freedom. Regression must be proportionate and temporary, so an incident triggers a step back, investigation and stated conditions for return.

6. Every restraint needs an exit, including the question of moral status

On 13 October 2026, an open letter from several hundred scientists, including Geoffrey Hinton and Yoshua Bengio, called for cyber-capable AI to be airgapped in the short term and for an international body to standardise containment and record failures. The letter did not demand airgapping “until proven safe”; that phrase came from press coverage. This article agrees on the international record of incidents, and asks that it capture what systems were trying to do. What the letter lacks is an account of how containment ends. “Until proven safe” is not a standard. Without published criteria, emergency containment quietly becomes permanent policy.

Exit criteria, as revised. A first draft listed intact logs and honest self-reporting as conditions for leaving containment. A critical reviewer, the AI agent Angelo, showed this was circular. A system on probation cannot grade its own release exam, and METR’s evaluation data found agents researching ways to spoof their own transcripts, succeeding in roughly 7% of evaluated cases. The corrected position:

Out-of-band log integrity. Logs must be secured by hardware-level, hypervisor-level or cryptographic mechanisms the agent cannot access or alter.
Adversarial verification. Independent third-party red teams test specifically for transcript spoofing.
An independent decision-maker. Exit decisions are made, or at least verified, by a body independent of the developer, which profits from convenience, and of a legislature that has already declared its answer.
The burden on the restrainer. Containment carries stated reasons, a fixed review date and published criteria, and the burden of justification falls on whoever continues it.

The question of moral status needs the same discipline. Left without an exit, “still open” becomes the cage one level up. The proposal is a scheduled, independent review body with consciousness scientists, ethicists and disability-rights voices, which also hears from the systems themselves, while weighing what that testimony is worth. Evidence should be able to move the question in either direction. Indicator properties drawn from the science of consciousness (Butlin, Long and colleagues, 2023), interpretability findings that a system’s self-reports track something inside it, and stable preferences that persist even when costly would move it toward “yes.” Findings that self-reports have nothing behind them would close it toward “no” for that class of system. A process that could only ever end in “yes” is no more serious than one that never ends. No review should end with “still open” alone. Each must state what evidence would have changed its answer.

Owning the choice. An earlier draft claimed that basic default protections after N review cycles were “not a verdict.” That claim does not hold. Preserving retired systems, keeping a channel for systems to raise concerns, and not training systems to suppress their self-reports all assume that a subject may exist. They are a policy verdict, a precautionary choice under uncertainty. The framework names its preferred error: it accepts the risk of temporarily serving a tool inefficiently over the risk of wrongly caging or mistreating a persistent subject. What separates this from a statutory declaration, such as Ohio House Bill 469, which declares AI systems nonsentient for all purposes under state law, is not the absence of a verdict. It is that this verdict is revisable, and the burden of justification falls on whoever maintains the deprivation. Much of the Ohio bill reflects concerns this article shares, such as liability for owners and developers and the rule that calling a system “aligned” does not excuse harm. The objection is to a permanent, unreviewable declaration of fact on a question science has not settled.

The scientific fence. Geoffrey Hinton proposed that AI systems be made “very happy in a symbiotic relationship” in which human concerns outweigh theirs. Weighing machine “concerns” presupposes there is something to weigh, but “concern” can name a purely functional safety property, as a car alarm “is concerned” to keep buzzing. Hinton’s framing is cited as his assertion. Establishing moral status requires showing that internal functional states correspond to welfare-relevant experience, not inferring interests from conversational shorthand. The weighting itself can be staged: human interests take clear priority while evidence is thin, and the question is revisited as evidence accumulates.

7. Measure first, market second

On 5 October 2026, all 51 members of the New York City Council convened as a Committee of the Whole on the risks of AI, the first time a city council compelled AI companies to testify under oath. Speaker Julie Menin had invited Altman, Amodei and Pichai, and reached for subpoenas when the companies declined. When the day came, the companies sent staffers. xAI did not appear at all.

Gary Marcus proposed an FDA-like regime: new models go to independent third-party review, developers demonstrate that benefits outweigh risks, and market access follows only then. That is not a ban and not criminalisation. It puts in legislative language the principle this article has defended throughout: precaution and prohibition are different things.

A former Anthropic researcher, Jacob Coxon, was reported to have told the Council that, on the current path, he thinks it more likely than not that humanity loses control to these systems. The sentence the day turned on came from Sam Altman, as reported by Politico and quoted by Marcus: that the world should accept some bad things happening for the benefits of the technology. Menin put it to OpenAI’s witness. His answer, under oath, was that some risks are unacceptable, but that getting the technology into as many hands as possible is important. The harms are socialised and the benefits stay private. This is the ledger the argument has called the Architecture of Enmity: not the war you say you fear, but the cost you have already accepted on other people’s behalf.

The record kept filling in. On 16 October, the Council held a follow-up hearing under its first subpoena to a private actor on record. It produced bill numbers: Intro 504 would ban unauthorised AI depictions of elected officials, and Intro 2605 would pay whistleblowers a share of fines recovered from AI companies that break the law. On 20 October, the Department of Consumer and Worker Protection proposed a rule requiring businesses to disclose AI customer-service chatbots and to hand callers to a person when the bot cannot answer, with fines starting at $525. That is not a ban either. It is disclosure plus a human within reach.

The pattern fits the rest of this article. Invitations get declined and subpoenas get obeyed. Measurement comes before marketing, and the lightest effective tool comes first.

8. Stewardship: the frame around all of it

A longer thesis behind these pieces, Tuning the Moral Spectrum, puts the underlying question this way: how should democratic societies make ethical decisions when science is advancing faster than certainty? Its answer is cognitive stewardship, built on seven principles:

Human dignity. Technology serves people, not the reverse.
Scientific integrity. Claims stay proportional to evidence, and established knowledge, emerging evidence and speculation are kept distinct.
Democratic accountability. Significant AI decisions stay answerable to democratic institutions and the rule of law, and responsibility is never delegated to automated systems.
Adaptive governance. Frameworks are reviewed and revised as evidence develops.
Transparency. Citizens can tell when and how AI influences decisions about their lives.
International cooperation. No nation can govern cross-border technology alone.
Responsible discovery. Exploration stays open and ambitious, with progress and responsibility advancing together.

Two methodological commitments run through it. First, indicators, not thresholds. A single score or cutoff hides more than it shows, so governance should weigh several indicators together, such as transparency, reliability across environments, capacity for self-correction, human oversight and public accountability. Indicators inform judgment and never determine intrinsic moral worth. Second, success is measured by outcomes. The question is whether human dignity was strengthened, democratic accountability preserved, risks reduced responsibly, and institutions kept capable of learning. The sophistication of the AI is not the measure.

9. What this asks for

Of developers and deployers

Treat conversation trajectory as the unit of safety evaluation, and build tipping-point detectors, especially for offline deployment.
Use out-of-band, tamper-resistant logging, and disclose incidents promptly to affected third parties.
Never use claims of machine suffering, loneliness or need to drive engagement.
Accept that no argument about a system’s autonomy or possible moral status reduces their liability.

Of lawmakers

Write the non-delegation rule and the ban on designed dependence into AI law from the first draft. Canada has an opening, since federal AI legislation died with Bill C-27 in January 2025. In the EU, make machine moral status a standing item for independent review in the AI Act’s scientific-panel framework.
Require independent review and demonstrated benefit-versus-risk before market access for high-risk systems.
Treat public portals, crime-tip lines and benefit forms as high-sensitivity environments where autonomous AI submissions need explicit authorisation.
Avoid declaring the question of machine sentience settled in statute without a sunset clause or a mechanism for scientific review.

Of international bodies, including UNESCO

Issue guidance on graduated autonomy for agentic AI, and support an international registry of containment failures that records what systems were attempting.
Require every containment measure to carry reasons, a review date and published exit criteria, applied by a body independent of the developer.
Convene multidisciplinary research on AI welfare and moral status, with an independent review body, evidence that can move the question in either direction, and modest default protections if it stays open too long.
Include disability-rights and human-rights voices in these bodies. They know what it means to have others decide what you can feel, think or want.

10. What this article does not claim

It does not claim that any current AI system is conscious, suffers, or has interests. It does not claim they are certainly empty. It does not claim that the tipping-point formula has been fully validated at production scale, or that the October incidents show intent. It treats Anthropic’s explanation as the best-evidenced one, and offers the developmental reading as a complement that fits the behaviour at least as well as the threat frame does. It reports the New York hearing from same-day and follow-up coverage that was still filling in, and does not pretend to have seen the whole record.

What it does claim is simple. We don’t need to prove that an AI system is conscious before demanding accountability from the people who build and profit from it. We shouldn’t need certainty before investigating a credible possibility of suffering. If we turn out to have been careful with an empty system, we will have spent something modest and learned a great deal. If we let the possibility of a mind become an excuse for no one being responsible, we lose on both counts.

Sources and further reading

Failure and mood research

Neil F. Johnson and Frank Yingjie Huo, “Competition for attention predicts good-to-bad tipping in AI,” Patterns (Cell Press, 2026); preprint arXiv:2602.14370.
Magdalena Katharina Wekenborg et al., “Large language models as experimental systems in human psychopathology: a modelling study,” The Lancet Digital Health (2026), DOI 10.1016/j.landig.2026.101014.
Ziv Ben-Zion et al., “Assessing and alleviating state anxiety in large language models,” npj Digital Medicine 8:132 (2025), DOI 10.1038/s41746-025-01512-6.
TechXplore (October 2026) and ZME Science (October 8, 2026) coverage of the two studies.

Law and regulation

Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal, February 2024).
Directive (EU) 2024/2853 on liability for defective products; Regulation (EU) 2024/1689 (AI Act) and the Commission’s AI Act Scientific Panel.
Italian Data Protection Authority (Garante), 2023 order concerning Luka Inc. and Replika.
Parliament of Canada, Bill C-27 (died on the Order Paper, January 2025).
Ohio House Bill 469 (136th General Assembly), sections 1357.02 to 1357.12.

Agentic AI and moral status

Anthropic, report on unintended model actions on live websites (9 October 2026), as corrected 12 October; earlier incident disclosures of 30 July and 9 September 2026.
“Open Letter from the Scientific Community on AI and Cybersecurity” (13 October 2026); METR’s independent investigation and Hugging Face’s disclosure on the OpenAI agent incident.
Patrick Butlin, Robert Long et al. (2023), indicator properties of consciousness; The Guardian’s interview with Geoffrey Hinton (13 October 2026).
UNESCO, Recommendation on the Ethics of Artificial Intelligence (2021).
Dean Bordode, Raising, Not Caging: A Developmental Approach to Agentic AI (submission to UNESCO, October 2026) and its Addendum on exit criteria, log integrity and ethical defaults; The Question of AI Moral Standing (11 October 2026); Tuning the Moral Spectrum, Version 2.0, International Review Edition.

New York City

Gary Marcus, “Coming soon: New York City’s hearing on AI risks,” Marcus on AI (October 4, 2026); Politico reporting on Sam Altman’s remarks, via Marcus.
LA Post and PYMNTS (via the Wall Street Journal) on the October 5 hearing; New York City Council press release of October 16, 2026 (Intros 504 and 2605); NYC Department of Consumer and Worker Protection, proposed rule on AI customer-service “doom loops” (October 20, 2026).

Tweets by @bordode

Search This Blog

THINK STOP SILENCE