Anthropic’s CEO wants outside watchdogs to watch the company’s peers, and the watchdog he cites comes from the same safety community that birthed his own company.
Dario Amodei, CEO of Anthropic, has proposed that independent “embedded evaluators” supervise AI development companies. These evaluators would verify safety practices, report incidents, and assess model alignment and training pipelines. Amodei wrote: “We are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.”
The proposal puts the safety community at the center of its own oversight. That is the central tension in this story: a group that built its reputation on caution is now asking its own members to judge it.
The Man Behind the Plan
Amodei is a physicist turned AI executive. He earned a Ph.D. in biophysics from Princeton in 2011 and spent years building neural networks at major tech companies before founding Anthropic.
His resume reads like a tour of Silicon Valley’s AI pipeline. At Baidu, he worked on speech recognition through machine learning. At Google, he helped develop neural-net research. At OpenAI, he became vice president of research during the development of ChatGPT 2 and ChatGPT 3.
He left OpenAI in 2020 because he believed the company wasn’t installing guardrails on the technology. He also didn’t know if he could trust the company to set aside financial interests.
“When you feel that you can’t trust someone, when you feel that their values are not what they say they are, when you feel that they’re not honest, when you feel that they’re not in it for the reasons that they say, when you see disturbing patterns of behavior, dishonesty, that makes it very hard to continue to wo”
That mistrust was part of why he left.
The Watchdog He Cites
The organization Amodei points to is METR, an AI safety testing laboratory. Its founder and CEO is Beth Barnes, who worked at OpenAI alongside Amodei during early ChatGPT development.
METR’s other co-founder is Paul Christiano. He led research around OpenAI’s model strategy controls and founded the first iteration of METR. He has framed his approach as effective altruism.
Barnes and Christiano have also provided safety checks for Anthropic’s AI Claude.
The Funding Circle
Amodei’s path to Anthropic passed through a funding circle that now defines the AI governance debate.
Jaan Tallinn led Anthropic’s 2021 Series A financing round. He is a prominent supporter of the Effective Altruism movement. Sam Bankman-Fried led Anthropic’s 2022 Series B financing round before FTX collapsed and he was convicted in a multibillion-dollar fraud case.
The Proposal in Detail
Amodei’s plan calls for evaluators embedded within companies to monitor AI development. Their role would be verification, reporting, and assessment.
The proposal is explicit about what it changes. Amodei wrote that the evaluators would alter “the dynamic” of who chooses what gets included and omitted.
The Safety Community
The safety community grew from the same people who built OpenAI’s early models. Christiano’s work on model strategy controls at OpenAI laid the groundwork for the approach METR now follows.
Barnes’s role at OpenAI placed her directly in the room where ChatGPT was developed. Her later work at METR connects the two organizations.
The overlap is notable. The person who tested Anthropic’s Claude also tested OpenAI’s early GPTs.
Why the Proposal Matters
Amodei’s proposal is a direct attempt to address the problem he saw at OpenAI. He left that company because he believed it wasn’t installing guardrails on the technology, and he now wants the industry to build those guardrails elsewhere.
Whether an evaluator from the same community can truly be independent is a fair question. The source does not answer it.
What We Know
- Dario Amodei is CEO of Anthropic.
- He proposes independent “embedded evaluators” to supervise AI development companies.
- METR is the organization he cites as an example of the guidance he believes the industry needs.
- Beth Barnes is METR’s founder and CEO; she worked at OpenAI alongside Amodei during early ChatGPT development.
- Paul Christiano led research around OpenAI’s model strategy controls and founded the first iteration of METR; he has framed his approach as effective altruism.
- Sam Bankman-Fried led Anthropic’s 2022 Series B financing round before FTX collapsed and he was convicted in a multibillion-dollar fraud case.
- Jaan Tallinn led Anthropic’s 2021 Series A financing round; he is a prominent supporter of the Effective Altruism movement.
- Barnes and Christiano provided safety checks for Anthropic’s AI Claude.
- Amodei originally studied biophysics, earning a Ph.D. from Princeton in 2011.
- At Baidu, he worked on speech recognition through machine learning.
- At Google, he helped develop neural-net research.
- At OpenAI, he became vice president of research during the development of ChatGPT 2 and ChatGPT 3.
- Amodei left OpenAI in 2020 because he believed the company wasn’t installing guardrails on the technology.
- He also doubted whether he could trust the company to set aside financial interests.
Where the paper stands
The paper backs narrow rules forcing companies to disclose safety failures they hid and is against broad rules that hand the market to the incumbents, and it is against letting big companies hide their failures while asking for oversight over others. In this case, the proposal’s central flaw is that the evaluators come from the same community whose judgment is at stake. An evaluator who has tested both Anthropic’s Claude and OpenAI’s early GPTs is hardly independent.
This is a familiar pattern. Big tech asks for oversight of its peers while keeping its own failures hidden. The paper has said before that licensing regimes and compliance costs only giants can afford are a moat, not a safeguard. Here the moat is the same safety community itself, and the proposal does nothing to breach it.
The reader should watch for the moment when the evaluators stop being observers and start being defenders of the status quo. That is when the dynamic changes for real, and it is not the change Amodei promises.
Source material: “Who is Dario Amodei, the Anthropic CEO pushing independent AI oversight?,” Fox News.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.