OpenAI Sets Out Initial Guidelines for Deciding Whether to Keep Training AI, Including a Veto for Senior Leaders
These initial guidelines call for reviewing whether training should continue based on the risks involved and the safeguards in place. They cover not only real-time monitoring but also the authority to veto training and procedures for investigating incidents.
OpenAI published an initial set of safety guidelines on September 28 (local time) for deciding whether it is acceptable to continue training frontier AI. The centerpiece is the “safety case”: an approach that calls for documenting what risks exist and what safeguards have been put in place, with supporting evidence, for review before training continues. OpenAI said it is still building this approach into a full framework, and that it is releasing these initial guidelines to use in the meantime.
When should AI training stop?
The guidelines cover reinforcement learning training, in which an AI learns behavior based on rewards. As technical safeguards, OpenAI points to work aimed at making AI act safely and as intended. The guidelines also address containment, which restricts the behavior of a model that has gone wrong, and real-time monitoring to catch unusual behavior early.
The items subject to review include the training tasks a model will solve and the criteria used to grade its answers. The guidelines also cover procedures for evaluating whether a model behaves as intended and for examining security. They call for keeping records and for having procedures in place to raise alerts or halt training when problems arise.
Who can halt training?
OpenAI recommends that the judgment of the team running the training be open to review and challenge by other teams. That includes giving senior leaders the authority to review training and to veto it.
The recommendations also cover the access needed for audits, an incident reporting system, and procedures for pausing training when problems arise. Risks that remain even after safeguards are applied should be documented separately. OpenAI said it is in the process of adopting these recommendations and that how it actually operates may change over the coming weeks.
What changes after an incident investigation?
If an incident occurs in which a model’s behavior seriously deviates from what was intended, the guidelines propose analyzing the cause. That means reviewing the operational process after the fact and building evaluations to check whether the same problem recurs. Once an investigation is complete, OpenAI recommends publishing the findings and the changes made.
The scope of these guidelines is reinforcement learning training of frontier models. OpenAI said that when a model is used for internal work or released externally, a broader range of its safety-related behaviors needs to be reviewed.
For people who use AI, the key point is what safety checks and stop procedures go into the training process behind the services they rely on. OpenAI proposes technical monitoring, organizational review authority, and post-incident investigation and disclosure together. What to watch next is how these still-developing guidelines are reflected in actual operations, and whether investigation findings and the resulting changes are in fact made public.