Call me cynical, but when I hear the CEO of one of the world’s most powerful AI companies asking government for the authority to block “dangerous AI” before it reaches the public, I do not immediately picture a selfless public servant standing between humanity and catastrophe.
I hear something closer to this:
“Frontier AI is so dangerous that only highly capitalized, investor-backed, institutionally trusted, regulator-integrated labs with barrier-to-entry-sized compliance departments should be allowed to deploy it.”
That may sound harsh. It is also exactly the kind of incentive structure we should be honest about.
Dario Amodei’s latest policy essay argues that frontier AI should move beyond transparency requirements into binding regulation, including mandatory third-party testing and government power to block or reverse deployment of models that fail safety standards. He points to risk areas like cybersecurity, biological misuse, loss of control, and automated R&D acceleration.
Some of that is reasonable.
I work with AI systems. I build with them, test with them, and depend on them. I do not think powerful AI should be treated like a random SaaS feature pushed on a Friday afternoon. Systems with plausible leverage over cyber operations, biological design, critical infrastructure, weapons integration, or autonomous decision-making deserve serious pre-deployment evaluation.
In other words: the question is not “regulate AI” versus “do nothing.”
The real question is whether regulation constrains dangerous capability or merely freezes the current winners in place.
That distinction matters.
OpenAI, Anthropic, Google, Meta, xAI, and the other frontier players did not become powerful in a highly regulated environment. They benefited from an unusually permissive period in which enormous amounts of human-produced text, code, art, research, and conversation were absorbed into training pipelines before society had any coherent governance model for what was happening.
Now some of those same companies are large enough to absorb compliance costs, shape standards, fund policy teams, build government relationships, and sit across the table from regulators as “responsible stakeholders.”
Startups, academic labs, open-source researchers, and small applied AI companies do not have that luxury.
So yes, companies operating at frontier scale should be held to a higher standard. But we should be very careful about writing rules that would have killed the current leaders before they had a chance to become the current leaders.
That is the regulatory capture risk.
It does not require bad faith. Anthropic can be sincere about AI risk and still advocate for a policy architecture that favors Anthropic. Those two things are not mutually exclusive. In fact, that is often how capture works in technical domains. The argument starts from a real hazard. Then the remedy quietly becomes something only incumbents can comply with.
The test should be simple:
If a technically competent small lab cannot comply without becoming economically dependent on an incumbent, the rule is probably industrial policy wearing a safety vest.
A serious AI safety regime should be provider-neutral. It should attach to measurable capabilities, deployment contexts, and demonstrated hazards, not to brand names, corporate trust narratives, or whether a company has the right policy staff in Washington.
That means regulation should ask:
- What can the system actually do?
- Who can access it?
- What failure modes have been demonstrated?
- What safeguards exist?
- What deployment context is being authorized?
- Who is accountable if the system causes harm?
- Can the decision be appealed, audited, and contested?
That is a systems view. Risk is not a logo. Risk is capability plus context plus exposure plus control failure.
A lab chatbot, a local research model, an autonomous SOC agent, a bio-design assistant, a weapons targeting system, and a critical infrastructure controller should not all be forced through the same crude rule. Nor should we pretend that compute threshold alone is a sufficient proxy for danger.
Compute matters. Scale matters. But capability matters more. Deployment context matters even more.
The right policy architecture should look less like “trust the big labs plus Washington” and more like a safety case regime:
- Mandatory independent evaluations for clearly defined high-risk capability classes.
- Public test specifications wherever possible, with controlled sensitive evals where necessary.
- Red-team and audit pathways small labs can actually access.
- Security requirements for frontier model-weight custody.
- Incident reporting with enough detail to be useful, not just reputational theater.
- Post-deployment monitoring for systems used in high-consequence environments.
- Liability for reckless deployment.
- Government block or recall authority only with due process, written findings, time limits, appeal rights, and oversight.
- Explicit protections for open research, local evaluation, academic replication, and security testing.
This is not anti-regulation. It is anti-moat.
And it is also anti-naivete.
There are real AI risks. Cyber offense is not imaginary. Biological misuse is not impossible. Autonomous systems can fail in ways that are not intuitive to non-practitioners. Model behavior can change under tool access, agent scaffolding, retrieval, fine-tuning, deployment pressure, and adversarial prompting.
Anyone who says there is nothing to regulate is not paying attention.
But anyone who says the answer is to let a handful of frontier companies and government agencies define “safe enough” behind closed doors is also not paying attention.
The legitimacy of AI regulation will depend on whether it is contestable, portable, and provider-neutral.
Contestable means regulated parties can challenge decisions and see the basis for them.
Portable means compliance does not require exclusive relationships with a few approved vendors or auditors.
Provider-neutral means the same standard applies to Anthropic, OpenAI, Meta, xAI, defense primes, startups, academic labs, and open-source deployers when they cross the same risk threshold.
There is a version of AI regulation that raises the floor for genuinely dangerous deployments.
There is another version that raises the drawbridge around incumbent labs.
We should be very clear which one we are building.
The uncomfortable truth is that frontier AI companies are not just ordinary market participants anymore. They are becoming infrastructure providers, policy actors, national security stakeholders, labor-market disruptors, and quasi-public institutions. That makes their incentives complicated. Their safety arguments deserve to be heard, but not simply adopted.
The public needs AI policy that can survive contact with both catastrophic risk and corporate self-interest.
My position is straightforward:
Regulate dangerous AI capabilities. Do it seriously. Do it before preventable harm becomes the justification for panic legislation.
But do not let “AI safety” become the language powerful companies use to convert yesterday’s permissionless growth into tomorrow’s protected market position.
The goal should be safer systems, not smaller competition.
That is the debate I think we should be having: what would an AI safety regime look like if it were designed to be as hostile to regulatory capture as it is to catastrophic risk?
Sources
- Dario Amodei, “Policy on the AI Exponential”
- ABC News interview coverage, June 10, 2026: “Exclusive: Anthropic CEO calls for stronger regulation of AI”
- Anadolu coverage, June 11, 2026: “US government should have authority to block dangerous AI models: Anthropic CEO”