Proposed AI risk evaluators may lack power to prevent disasters

Independent AI risk evaluators proposed by Anthropic and OpenAI would get access to internal systems but not the right to stop development or model releases. Banking regulation experts note that without the power to shut down an organization, the analogy to banking supervision does not hold.

• Amodei proposed embedding third-party evaluators inside AI companies modeled on banking supervision
• Evaluators could publish findings but would not have the right to ban training or model release
• Banking regulation expert: without power to stop an organization, the analogy is flawed
• Model tests reveal failures but may miss rare combinations of conditions for catastrophe