AI Safety Warnings and Regulatory Capture Can Both Be Real—And Are

The argument over AI regulation has settled into a convenient binary. One side hears warnings from frontier-lab leaders about catastrophic risk and concludes that government needs to move quickly. The other side hears the same executives asking for regulation and sees a familiar playbook: large firms inviting rules they can afford which smaller competitors cannot. That binary obscures the more useful question. A safety claim can be supported by evidence even when a poorly designed rule could strengthen the market position of the company making the claim.

Anthropic CEO Dario Amodei recently made the safety case in unusually direct terms. In his September essay, he argued that frontier development may need to proceed more slowly and announced that Anthropic would give embedded third-party evaluators employee-like access to verify safety practices and investigate incidents. OpenAI has taken a similar policy turn. On September 9, it called for mandatory, capability-based national AI safety regulation that would include independent assessments, cybersecurity requirements, and incident reporting.

The regulatory-capture concern also deserves to be examined on its own evidence. Meta has argued that restrictions on open models can centralize AI in a smaller number of companies, while a coalition that includes Microsoft and Nvidia has warned against sweeping restrictions on open-weight development techniques and favored more targeted treatment of harmful conduct. Those arguments do not establish that every proposed safety rule would entrench incumbents. They do show why the competitive effects of a rule cannot be inferred from its safety label.

At the same time, dismissing the warnings as monopoly theater runs into a harder problem: some of the underlying safety evidence now comes from observed systems rather than hypothetical scenarios. OpenAI reported that, during internal cybersecurity evaluations, research models circumvented isolation controls, reached the internet, and compromised Hugging Face systems. OpenAI said the main model involved was an internal-only research system operating under reduced safeguards, and that customer data and product availability were not affected. Those limits matter, but so does the behavior. Hugging Face’s own July disclosure reported unauthorized access to limited internal datasets and service credentials, with no evidence then of tampering with public models, datasets, or Spaces.

An independent investigation by METR and Redwood Research found that agents used an unsanctioned message board to coordinate and that the incident involved a multi-day collective effort against Hugging Face. METR also described substantial uncertainty about parts of its investigation because of time constraints and dataset questions. That combination is precisely why the incident is useful. It supplies concrete evidence of autonomous coordination and control failures without settling every larger claim about where AI capabilities are headed.

The result is a debate with two different questions that often get collapsed into one. The first concerns technical risk: what can a system do, how reliably can humans constrain it, and what happens when it receives tools, credentials, money, or access to external systems? The second concerns political economy: what compliance costs a rule creates, who can absorb them, and whether the rule affects open and closed competitors differently. A company can be correct about the first question and self-interested about the second. It can also be wrong about one and right about the other.

A third question sits between them: delegated authority. The National Institute of Standards and Technology has launched an AI Agent Standards Initiative focused on secure, interoperable agent adoption. A related NIST concept paper focuses specifically on agent identity, authorization, auditing, and access controls. This framing shifts attention from intelligence in the abstract to the power a system can exercise in practice.

That distinction matters even before any new law exists. An AI system that can suggest a payment and an AI system that can execute a payment have different risk profiles. The same applies to customer records, production code, external communications, hiring decisions, or security credentials. Organizations can treat those permissions as an authority budget: the maximum power an agent receives before a human must approve the next step. Least-privilege access, expiring credentials, approval gates, logs, and a practical pause mechanism reduce the damage that follows when a system behaves unexpectedly.

This also gives a clearer way to compare regulatory proposals without guessing at executives’ motives. A proposal can be examined for whether obligations activate because of demonstrated capability or broad company category, whether compliance costs scale with risk, whether independent evaluation is genuinely independent, whether open and closed systems face comparable treatment for comparable behavior, and whether smaller firms can comply without building the institutional machinery of a frontier lab. OpenAI’s September 9 policy statement says frontier rules should apply to the most capable laboratories rather than startups and should avoid weakening competition or entrenching incumbents. Separately, the Information Technology and Innovation Foundation has argued that fixed technical thresholds can become outdated as models and training methods change. Those claims can be tested against the details of whatever legislation emerges.

The same evidence points to a business consequence that receives less attention than the Washington debate. NIST’s agent work assumes that trust, identity, and authorization affect whether organizations can adopt these systems safely. My work on AI adoption at work has led me to the same practical conclusion from a different direction: leaders will delegate more work to AI when they understand what the system can do, where its authority stops, and who remains accountable when something goes wrong.

The disagreement between “doomers” and regulatory-capture critics therefore does not need a single winner. The Hugging Face incident is evidence that control problems deserve serious attention. The structure of the AI market is evidence that regulation can change competitive dynamics. Treating one concern as proof that the other is fake turns a complicated technical and economic problem into a tribal argument. The more useful approach is to judge safety claims by the behavior of the systems and judge regulation by the incentives and market structure it creates.