California Gov. Gavin Newsom proposed a stronger system of AI safety oversight Wednesday, including state-set standards, ongoing independent testing and emergency authority to shut down dangerous models. The recommendations would change how powerful AI developers demonstrate that their systems are safe enough to train and release.

The November 18 announcement presents a plan for legislation. Releasing it does not itself impose the proposed shutdown powers or new testing duties. Its focus is the developers of advanced models, whose products can reach people and businesses far beyond California.

What California already requires

California's Transparency in Frontier Artificial Intelligence Act, SB 53, already requires large frontier developers to publish safety frameworks. According to the state attorney general's explanation, those frameworks must describe how companies assess dangerous capabilities, set catastrophic-risk thresholds, apply safeguards and respond to critical incidents.

The requirements also cover risks arising from a company's internal use of its models. Large frontier developers must send the Office of Emergency Services a summary of their assessments of catastrophic risk from that internal use. Oversight therefore reaches work inside a developer, before a product becomes a familiar consumer service.

SB 53 also protects covered employees who disclose specified safety risks or violations. Developers cannot enforce policies or contracts that prevent those protected reports or retaliate against workers for making them. The attorney general provides an existing reporting channel for covered employees.

The original SB 53 signing notice also describes civil penalties for noncompliance, enforceable by the attorney general. It directs the Department of Technology to recommend annual updates based on technological developments, stakeholder input and standards. The law already has an enforcement mechanism and a process for reconsidering its scope as models change.

Who would check the companies

Green desks and chandeliers in the California Assembly chamber.
California Assembly chamber on January 27, 2025. Archival photograph.Antony-22 / Wikimedia Commons, CC BY-SA 4.0. Displayed under the same license.

Two laws signed September 9 created more of the machinery for outside review. SB 813 establishes a framework for independent verification organizations that assess AI systems and models for compliance with state law. AB 1405 creates a registry for AI auditors and standards for their independence, transparency and integrity.

Wednesday's proposal would make independent testing a continuing requirement across training, development and deployment. The state would set the baseline standards companies must meet. That would make outside review part of developing a model, with testing continuing as the system changes.

How emergency shutdowns would work

Rows of server racks with blue indicator lights at NOIRLab in Tucson.
Server racks at NOIRLab headquarters in Tucson, Arizona, in a photograph released in 2022. Contextual archival photograph.NOIRLab/NSF/AURA/T. Slovinský / Wikimedia Commons, CC BY 4.0. Resized for display.

The Government Operations Agency report, dated November 16, recommends tested risk-containment plans covering computing resources, agents, internal access and deployment. Companies would identify responsible people and coordinate with cloud providers and power suppliers.

An emergency order would take effect immediately if officials had credible reason to believe catastrophic danger or a major loss of control was imminent and the developer refused to contain it. A developer could seek court relief, but the order would remain effective pending review. Officials should choose the narrowest workable intervention and remove it when the danger ends.

Ordinary corrective orders would follow a different path: developers could challenge the state's determination in court before it took effect.

The report also recommends authority to limit automated AI research and development when risks become unacceptable. It does not recommend imposing a cap immediately. Such limits should be temporary, reviewed periodically and lifted after adequate safeguards are verified.

The implementation dates already on the calendar

Newsom's September 18 executive order set the work in motion. It instructed agencies to submit recommendations by November 16 on independent evaluations, verified safety documents, shutdown controls and broader incident reporting.

That order also accelerated implementation of the earlier oversight laws. It directs the Government Operations Agency to publish application requirements, procedures and qualification criteria for independent verification organizations by May 1, 2027. For the separate AI auditor framework, it sets a December 1, 2027 deadline for completing the specified requirements and beginning the required actions. Those administrative deadlines are distinct from enacting the new legislative powers proposed Wednesday.