Anthropic's Draft AI Bill Would Let a Judge Halt a Dangerous Model
A halt could be requested if threats such as biological weapons or cyberattacks pose an imminent risk of killing or seriously injuring more than 50 people, or causing more than $1 billion in damages. The Commerce Department would petition a court, and a judge would decide.
Anthropic has presented US lawmakers with a draft bill that would regulate the safety of high-capability AI, Reuters reported on October 16. The draft lays out a procedure for requesting that the release and use of an AI model be halted when there is a risk of imminent large-scale death or serious injury, or economic damage. An Anthropic spokesperson said the draft incorporates proposals from a policy framework the company had published earlier.
What level of risk triggers a halt
The threshold for requesting a halt is more than 50 deaths or serious injuries, or more than $1 billion in damages. According to the report, that threshold applies when the harm stems from biological weapons, a model escaping its developer’s control, or cyberattacks. A request can be made when such harm is expected to occur imminently and the risk is foreseeable. The design allows a halt to be requested before any actual harm occurs.
Three parties would be involved in sequence in assessing the risk and deciding on a halt: independent evaluators, the Commerce Department, and the courts. An independent evaluator registered with the Commerce Department that judges a risk to be imminent would have to report it to the department within 72 hours of completing its assessment. The Commerce Department could then review the report, assess the risk, and ask the US District Court for the District of Columbia to order a halt, with a judge making the final decision. Reuters also reported that the draft includes a provision limiting the ability of other federal agencies to issue halt orders.
How it differs from the June policy framework
In June 2026, Anthropic published a policy document on regulating high-capability AI called the Advanced AI Framework. It proposed risk testing of models and public disclosure of the results, external evaluation, and government authority to block dangerous deployments. It singled out biological weapons, offensive cyber operations, and loss of control over AI, among others, as priority risks. This latest report reveals the specific halt criteria and request procedure written into the draft bill.
The June document proposed defining the scope of regulation using both the amount of compute used to develop a model and the size of the company. It would cover companies that use more than 10²⁵ FLOPs (a unit for counting compute) to train a model and that also have either more than $500 million in annual AI revenue or more than $1 billion in annual AI research and development spending. External evaluations would begin within six months of the regulation’s enactment, and risk reports would be published at least every six months.
Laws that define risk by the scale of harm already exist. California’s SB 53, enacted in 2025, uses more than 50 deaths or serious injuries, or more than $1 billion in property damage, as its threshold for catastrophic risk. What SB 53 mainly requires is a safety framework and disclosure of risk information. The Anthropic draft now being reported goes a step further, proposing at the federal level who can request a model halt and who decides.
What this means for AI used in Korea
The draft is a proposal aimed at the US Congress and the executive branch. It is not an enacted federal law, and no timeline for implementation has been set. Nothing in it applies directly in Korea.
The core of the draft is a procedure that connects the findings of a risk assessment to a government request and a court decision. For people and companies that use AI, what is at stake is who can halt the use of a model, and on what grounds. A key question going forward is how the reported harm thresholds and the authority assigned to each body get settled during the actual legislative process.