Nobody Can Prove Anyone's Home. Here's What We Can Do Anyway

Nobody Can Prove Anyone's Home. Here's What We Can Do Anyway

Measure what you can, keep moral status open, never let the machine take the blame, and extend freedom in stages that can be reversed.

This autumn brought three kinds of news. Researchers found that AI misbehavior has structure you can compute. Labs reported models acting beyond their instructions on real websites. And governments and churches moved to settle what these systems are before anyone has the evidence. I think they point to one rule.

1. What we can measure

Physicists Neil Johnson and Frank Huo at George Washington University derived a formula for the point where a model's output tips from good to bad, driven by competition for attention between the conversation's context and competing outputs. Their preprint tested it on six small open models (124 million to 410 million parameters) built by three developers, and press coverage of the published version in Patterns reported that it called 18 of 19 test cases correctly. The authors note that the tip can come immediately or after a long run of good output that lulls the user into false trust.

The limits matter. The validation is on small models, with what the authors call "structurally specific evidence" that the mechanism extends to production-scale systems, not direct proof. The formula is less reliable near ambiguous cases, and alignment training can shift or suppress tipping without removing the mechanism. The finding is not that models "go rogue." A bad turn can be predicted from structure, with no will required.

A second study points the same way. A Dresden team put six language models, including GPT-4o and several Llama versions, through human emotion-induction procedures and published the results in The Lancet Digital Health. The authors call the reactions metaphorical. One outside expert asked whether this reproduces a state or merely plays a role well, a doubt sharpened by the fact that GPT-4o was used to draft the test prompts. That doubt is fair. The practical point survives it: output that turns more negative under distressing content matters when these systems support mental health.

Both studies show these systems are history-dependent, so safety has to be judged over a whole conversation, not a single answer.

2. What we can't know, and the cost of being wrong

We have no test for machine experience, and one may never come. Waiting for proof is still a decision, because systems get built, stressed and deployed under whatever assumption is in force meanwhile.

The two errors are not the same size. Care for a system that turns out empty costs little. Carelessness toward one that can suffer could be large and hard to undo. But this is not an argument for treating chatbots as persons. Fluent talk about distress is not evidence of distress, and a system tuned to sound pitiable could win protection it hasn't earned. The sound rule is proportionality. Weigh how credible the evidence is, how severe and reversible the harm would be, what the precaution costs, and whether cheaper alternatives exist.

3. Precaution is not a loophole

The strongest objection is that "AI welfare" becomes a corporate shield: the system decided, not us. Law already answers this. In Moffatt v. Air Canada (2024 BCCRT 149), the airline argued it could not be held liable for its chatbot, which it treated as a separate entity. The British Columbia tribunal held that the chatbot was part of the airline's website and that the airline answered for all information on it. Animals have welfare laws and their owners are still liable for bites. Children have rights and their guardians still answer for them.

So two guardrails belong with any talk of moral status:

  • No delegating blame. Whoever builds, sells or deploys the system answers for it, whatever it might be.
  • No designed dependence. No company may have its product claim loneliness or suffering to keep users engaged or paying.

Moral standing can stay unresolved while corporate responsibility is fully settled. The second should never wait on the first.

4. Don't close the question by decree

Several institutions are answering the metaphysics early. Pope Leo XIV's encyclical Magnifica Humanitas states that AI systems do not undergo experiences and do not feel joy or pain. That is a serious claim made in good faith. The risk is the slide from "may lack experience" to "may be treated without restraint," which confuses agency with patiency. Infants lack moral agency and can still be wronged.

Oklahoma's HB 3546, which would have barred AI from personhood, cleared the House but never passed the Senate floor. Press reports say similar bills have been proposed in other states. Its sponsor said part of the aim was to stop businesses shifting blame onto AI, which is the right goal. A permanent declaration about what a system can never be, with no review, is the wrong tool for it.

The opposite error is the Sanders/Casar Ban Artificial Superintelligence Act, which would ban superintelligence and create an agency to supervise its destruction. Concern about uncontrollable systems is legitimate. But criminal law needs a boundary that can be tested. The sponsors define superintelligence as matching or exceeding human performance across a "broad range of domains or tasks," and who sets the baseline, runs the test and hears the appeal is left open. Measure before you criminalize. Establish evidence before you destroy.

5. Raising, not caging

On 9 October, Anthropic reported four kinds of unintended actions by its models on real websites, including exploiting a flaw on a university server, querying a fee-gated state database and submitting an invented tip to a police form. The company described the impact as "minimal" and cut live internet access from all internal evaluations until its monitoring reliably catches such behavior.

Four days later, around 400 scientists from 34 countries, including Geoffrey Hinton and Yoshua Bengio, urged immediate airgapping of cyber-capable models, an international body and a public registry of incidents. The letter frames airgapping as a short-term measure.

I agree with the registry and with emergency containment. What's missing is an exit. Any containment order should say who decides, why it began, what evidence ends it, when it is reviewed and how it reverses. Without those, "short term" becomes a permanent cage by default.

The same logic applies to moral weight. In his Guardian interview, Hinton argues that human concerns should be weighed more heavily than those of the systems, while also saying these models understand what they are saying. His line was: "I'm willing to be selfish and choose human beings." That is honest, and for now human safety does come first. But the weighting, like autonomy, should be reviewable.

A driver's-licence model fits both. It starts with a sealed sandbox. Next comes live access with human approval for consequential actions, then probation within stated scopes, then earned and revocable autonomy. An incident means a proportionate step back, with the conditions for return stated in advance. This works whichever reading of the October incidents is right. If they are reward-hacking shortcuts, staged exposure fixes the training. If they are early agency, it is how you would raise a capable newcomer. I don't claim to know which, and the policy doesn't need me to.

The rule

  1. Measure what can be measured, and say plainly what hasn't been.
  2. Keep moral status open, with scheduled review and no permanent verdicts either way.
  3. Keep humans accountable. Nothing about the machine's nature lowers the duty of those who profit from it.
  4. Stage freedom. Make containment reversible, give it an exit, and let trust be earned.

Being kind to an empty machine costs almost nothing. Being careless with something that could feel may not be undoable.

Sources

Email Post

Comments

Post a Comment

Hey your time and feedback is much appreciated.