Safety
Governance
Technical safety is necessary and not sufficient. Rules decide who must test, what must be disclosed and who is accountable. This page summarises the frameworks developers have adopted and the law that now applies.
This page is a summary for orientation, not legal advice. Summaries of statutes here rely partly on secondary sources. Check the legal text before acting on any of it.
Developer frameworks
The largest developers publish policies that tie safeguards to capability thresholds.
| Framework | Structure |
|---|---|
| Anthropic Responsible Scaling Policy | AI Safety Levels define sets of safeguards. Thresholds cover chemical and biological weapons uplift, misaligned AI in high-stakes settings, and automated AI research. Risk reports are published at regular intervals. |
| OpenAI Preparedness Framework | Tracked categories are biological and chemical, cybersecurity, and AI self-improvement, each with High and Critical thresholds. |
| Google DeepMind Frontier Safety Framework | Critical Capability Levels cover weapons uplift, cyber, machine-learning research and harmful manipulation. |
A comparison of what these policies share is maintained by METR.
Their limits
- Thresholds are qualitative, and companies assess themselves.
- Commitments increasingly depend on what competitors do.
- Use of models inside the company is weakly covered.
- No agreed method exists for a safety case at the level of automated research.
Law and public reports
EU AI Act. Regulation 2024/1689 places obligations on providers of general-purpose AI models. These have applied since August 2025. A model trained with more than 10^25 floating-point operations is presumed to carry systemic risk, which adds duties of evaluation, risk mitigation, incident reporting and cybersecurity. The voluntary Code of Practice is the main route to compliance.
California SB 53. In force since January 2026. Large developers of frontier models must publish a safety framework and transparency reports, report critical safety incidents, and protect whistleblowers.
International AI Safety Report. The 2026 edition, chaired by Yoshua Bengio and written by more than a hundred experts, is the reference summary of the evidence.
Compute governance. Computing power is detectable, excludable and quantifiable, which makes it a practical point of regulation.
Calls for restraint. In October 2025 the Future of Life Institute published a statement calling for a prohibition on developing superintelligence until there is broad scientific consensus that it can be done safely and controllably, and strong public support. Its signatories include Geoffrey Hinton and Yoshua Bengio.
What researchers expect
A 2023 survey of 2,778 published AI researchers gives the best available picture.
| Question | Result |
|---|---|
| Median probability of extremely bad outcomes, such as human extinction | 5% |
| Share giving at least 10%, depending on wording | 38% to 51% |
| Share who thought good outcomes more likely than bad | 68% |
These figures depend strongly on how the question is worded, and the survey predates the capability gains of 2024 to 2026. They should be cited as ranges.
Open problems
- Verifying compliance, especially across borders.
- Compute thresholds weaken as a proxy when algorithms become more efficient.
- No binding regime addresses the superintelligence threshold itself.
- Internal deployment and automated research sit largely outside regulation.
Recommended practice
- Publish a framework. State thresholds, evaluations, safeguards and who decides.
- Map it to the law. Align internal policy with the EU Code of Practice and SB 53.
- Report incidents. Maintain incident reporting and protected channels for staff.
- Track compute. Record training compute against legal thresholds.
- Commission outside review. Have risk reports reviewed by an independent party, and publish deviations.