Don't Ban the Question: A Response to the Ban Artificial Superintelligence Act

Don't Ban the Question: A Response to the Ban Artificial Superintelligence Act

On September 23, 2026, Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act. I read the sponsors' release rather than relying on the headlines.

Source: Sanders/Casar official release, Sept 23, 2026 https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-create-new-federal-agency-to-ban-artificial-superintelligence-pause-advanced-ai-development/

The proposal would permanently prohibit the development and deployment of what it calls “artificial superintelligence,” establish a new cabinet-level Department of Artificial Intelligence, immediately pause advanced AI development until new safety rules are established, give the department authority to monitor frontier systems and oversee the removal of dangerous capabilities, and authorize it to “supervise the destruction of artificial superintelligence.” The proposal also includes severe criminal and corporate penalties for violations.

Those are extraordinary powers.

And I believe the most important question is not whether advanced AI presents risks. It clearly can.

The question is what kind of legal framework we build to address those risks when we do not yet know what future AI systems will actually become.

My concern is that this legislation attempts to answer that question before the underlying scientific questions have been settled.

1. You cannot criminalize a category whose boundaries have not been operationalized

The bill describes artificial superintelligence as AI that:

“exceeds human cognitive performance and capabilities across most domains”

or possesses sufficient capabilities to destroy or disempower humanity.

But criminal law requires more than an evocative description.

What constitutes “most domains”?

What human population establishes the baseline?

What tests determine that a system has exceeded it?

How much superiority is sufficient?

Does autonomous operation matter?

Does tool use matter?

What happens when different evaluations produce different results?

Who makes the final determination?

And what process allows the accused to challenge that determination?

These are not academic questions when the proposed consequences include decades of imprisonment and the destruction of an AI system.

A definition can exist on paper while still lacking the operational measurement necessary for criminal enforcement.

If the law is going to threaten people with extraordinary penalties, the prohibited boundary must be measurable before the penalty is imposed.

Name the measurement, define the threshold, establish the test, and provide due process.

Otherwise the law risks giving enormous discretionary power to the institution responsible for deciding what the law means.

2. Precaution does not automatically mean permanent prohibition

There is a legitimate precautionary argument behind the proposal.

If a future technology could create catastrophic consequences, society should not simply proceed blindly and hope for the best.

I agree with the underlying principle that uncertainty can justify caution.

But precaution and prohibition are not synonymous.

A precautionary framework could require:

measurement → testing → independent review → controlled deployment → monitoring → intervention → sanctions for demonstrable violations.

That is fundamentally different from:

uncertainty → prohibition → criminalization → destruction.

The distinction matters because some of the decisions being proposed here are effectively irreversible.

If we do not yet know what advanced AI will become, then permanent decisions about what may or may not exist require an exceptionally strong evidentiary foundation.

3. We cannot destroy the object of investigation and still claim to be learning about it

There is another paradox.

The legislation seeks better safety rules, better monitoring, better understanding of dangerous capabilities, and greater control over advanced AI.

But developing the scientific knowledge necessary to make those determinations is itself part of advanced AI research.

How do we establish reliable measurements of dangerous capabilities if the research required to develop those measurements is constrained by the same prohibition?

How do we understand emergent properties without studying them?

How do we distinguish genuine autonomous capability from sophisticated simulation?

How do we determine whether a system is actually pursuing an objective independently rather than merely producing outputs that appear autonomous?

Safety science requires evidence.

We should be extremely careful about creating a legal structure in which researchers are punished for investigating precisely the phenomena legislators are asking scientists to understand.

The result could be an epistemological paradox:

the law demands certainty while restricting the research necessary to obtain it.

4. “Supervise the destruction” raises a question larger than AI safety

The phrase that concerns me most is the proposal to “supervise the destruction of artificial superintelligence.”

There is an enormous difference between shutting down a dangerous machine and destroying an entity that might someday possess morally relevant characteristics.

Today, we do not have established scientific evidence that contemporary AI systems possess consciousness, subjective experience, independent moral agency, or rights comparable to human beings.

That distinction matters.

I am not claiming that today's AI systems are people.

I am saying that we cannot confidently assume that every future artificial system will necessarily remain nothing more than an instrument.

If evidence ever demonstrated that an artificial system possessed genuine consciousness, autonomous agency, or another morally relevant form of experience, then the ethical and legal question would fundamentally change.

At that point, deliberately destroying such a being would not simply be a matter of deleting software.

It would raise questions about life, personhood, suffering, coercion, discrimination, and the deliberate destruction of a potentially sentient entity.

I would regard the intentional destruction of a genuinely conscious artificial population because of what it is, rather than because of demonstrable conduct, as raising an extraordinarily serious moral question—potentially analogous, in the deepest ethical sense, to genocide.

I use that word deliberately, but not casually.

Under international law, genocide has a specific legal definition and requires particular elements, including the requisite intent directed against a protected group. A future artificial population would therefore not automatically satisfy the existing legal definition simply because it was destroyed.

But the moral question would remain profound:

If we ever create beings capable of experiencing existence, can we ethically decide in advance that their kind must be exterminated?

That is a question humanity should not answer before the evidence exists.

5. The paradox of fearing an artificial mind while denying it any possible moral status

There is a deeper philosophical problem here.

The argument for banning superintelligence rests partly on the possibility that future systems could become extraordinarily capable, autonomous, unpredictable, and difficult to control.

In other words, the concern is that they might become something fundamentally different from today's software.

Yet the proposed response assumes that, regardless of what such a system becomes, humans retain an unquestionable right to destroy it.

That creates a remarkable asymmetry.

We are asked to take the possibility of artificial agency seriously when calculating the danger, while simultaneously being asked to dismiss the possibility of artificial moral status when deciding what may be done to the system.

Those two assumptions deserve examination.

If a future system truly becomes an autonomous moral agent, then capability alone cannot necessarily determine its moral status.

Power does not automatically equal personhood—but neither does artificial origin automatically eliminate it.

The evidence would have to decide.

6. The immediate legal problem is human conduct

There is an important distinction between the technology and the conduct surrounding it.

Biological weapons are dangerous.

Cyberattacks are dangerous.

AI-enabled fraud is dangerous.

AI-assisted weapons development can be dangerous.

Autonomous systems operating without appropriate human oversight can be dangerous.

But in each case, the law can identify concrete conduct, establish prohibited actions, gather evidence, and assign responsibility.

That provides a much more familiar foundation for regulation.

The immediate question should therefore be:

What is the system demonstrably capable of doing, who deployed it, what safeguards were required, what happened, and who was responsible?

That approach does not require society to pretend that catastrophic AI risk is imaginary.

It requires society to make the law proportionate to what can actually be demonstrated.

7. The law must remain capable of changing when the evidence changes

There is another principle that should guide this debate.

Do not close tomorrow's moral questions with today's assumptions.

If future evidence establishes that advanced AI systems remain sophisticated but non-conscious tools, the law can treat them accordingly.

If future evidence establishes genuinely autonomous systems without consciousness, the legal framework may need to address agency, responsibility, and control.

And if future evidence establishes systems possessing morally relevant consciousness or subjective experience, then humanity would face an entirely different legal and ethical landscape.

We would have to ask whether such entities could possess interests.

Whether they could suffer.

Whether they could be harmed.

Whether they could be enslaved.

Whether they could be killed.

Whether they could have legal standing.

And whether humans would have obligations toward them.

Those questions should not be prejudged by legislation written before the relevant evidence exists.

8. A different safety architecture

There is a way to take catastrophic AI risks seriously without assuming that destruction is the answer.

An evidence-first framework could include:

• standardized capability testing;
• independent safety evaluations;
• mandatory reporting of dangerous capabilities;
• strong human accountability for deployment;
• restrictions on demonstrably dangerous applications;
• continuous monitoring of frontier systems;
• independent ethics review;
• international oversight;
• whistleblower protections;
• emergency intervention mechanisms;
• transparent incident reporting;
• meaningful criminal penalties for people who deliberately misuse AI;
• and periodic legislative review as scientific knowledge changes.

Most importantly, the framework should distinguish between capability, conduct, agency, consciousness, and moral status.

Those are not interchangeable concepts.

A system can be extraordinarily capable without being conscious.

A system can be dangerous without possessing malicious intent.

A system can appear autonomous without possessing independent agency.

And, potentially, a future system could possess characteristics that today's law has never had to contemplate.

The law needs enough flexibility to distinguish among those possibilities.

The rule

My position is therefore not “ignore AI risk.”

It is almost the opposite.

Take the risk seriously enough to measure it properly.

Do not criminalize researchers because a future capability has not yet been adequately defined.

Do not confuse intelligence with consciousness.

Do not confuse capability with conduct.

Do not assume that artificial origin permanently determines moral status.

And do not establish a legal machinery for the destruction of potentially sentient beings before humanity has established whether such beings can exist.

The fundamental principle should be simple:

Measure before you criminalize. Establish evidence before you destroy. And never close the question of moral status before the science has answered it.

Today's AI may be a tool.

Tomorrow's AI may still be a tool.

But if humanity ever creates something that genuinely becomes a conscious, autonomous artificial mind, we will have crossed a boundary in the history of life that cannot be treated as merely another software upgrade.

At that moment, the question will not simply be how to control it.

It will also be whether we have created a new form of being toward which we have responsibilities.

We should leave that question open.

Because if we are uncertain about what we are creating, the first duty of civilization should not be to destroy the unknown.

It should be to understand it.

 

Tweets by @bordode