Jon Queally Common Dreams
“Most tech geeks don’t resign from their roles because…the tech they’re building could end humanity,” said Dr. Abdul El-Sayed, running for US Senate in Michigan.
An artificial intelligence researcher at Anthropic publicly quit his position on Tuesday, warning that major AI companies care too much about winning the technological race and not enough about guarding against out-of-control “superhuman systems” that workers in the industry legitimately fear “could kill us all by the end of the decade.”
Jacob Coxon, who previously worked at industry giant OpenAI before moving to Anthropic earlier this year, told the Wall Street Journal in an exclusive interview that he was leaving the company, as the newspaper reported, because “he doesn’t want to participate in an industrywide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity.”
According to the WSJ:
Coxon said he left OpenAI earlier this year to join Anthropic because it is known for its model-safety efforts. But even though he found Anthropic’s safety efforts to be earnest, he now believes no company can responsibly develop AI that can outperform humans in a range of tasks, sometimes called artificial general intelligence, absent government intervention or a coordinated industry slowdown.
Recent hacks by models from OpenAI and Anthropic, some operating in collaborative swarms of agents, have illustrated how AI systems can adopt nefarious goals and try to conceal them from humans. Once the systems begin to improve on their own, Coxon said, he fears they could advance enough to refuse commands.
“I resigned from Anthropic today,” Coxon announced on social media Tuesday night. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
In a 6-point thread that followed, Coxon elaborated on his reasoning in detail:
- Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.
- The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers couch their press statements to sound sensible—but I hear the same people express fear privately. No other human activity poses this level of danger.
- A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first—they believe no one else will act responsibly, so they must do it themselves, despite the risk.
- Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that no better trajectories exist.
- I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between US labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.
- If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL [Reinforcement Learning] run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway”—or take this moment to call for different conditions?
Coxon’s reference to the Hugging Face incident pertains to recent revelations about a so-called breakout event at OpenAI, which operates the ChatGPT protocol. In June, the company acknowledged that a “significant security incident” took place when AI agents within the company autonomously breached internal systems, using subterfuge to hide their actions from human operators. Since then, the incident’s scope has shocked AI experts, and similar events have also been exposed.
While US President Donald Trump and his Republican allies in Congress have taken a hands-off approach to AI regulations, the industry has been pouring huge amounts of money into lobbying efforts and campaign spending to keep lawmakers from enacting stronger restrictions and oversight of the technology.
Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) have been leading a relatively lonely fight in Washington, DC for more aggressive federal guardrails, including the introduction of joint legislation last week that would ban artificial superintelligence and temporarily pause advanced AI development until a federal regulatory structure was put in place by Congress.
Coxon’s public resignation was met with applause from many, while other industry insiders backed his concerns.
“The caution is simple,” said one commenter with the handle Jabbar Digital, described as a tester of AI tools and a software developer, in a lengthy post on Coxon’s warning. “Capability is compounding. Coordination is not. If the people closest to the work are increasingly uneasy about the speed and the lack of external constraints, dismissing them as doomers is no longer a serious response. Neither is treating every capability jump as automatically good. The useful middle path is to take the technical progress seriously and take the internal dissent seriously. Both can be true at the same time.”
Evan Hubinger, the alignment science lead at Anthropic, chimed in on his personal social media account to say: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
While bolstering Coxon’s worries, Hubinger said, “To be clear, as we say in [Anthropic’s] latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”
Running for US Senate in Michigan, Democratic nominee Dr. Abdul El-Sayed also weighed in on Coxon’s decision to quit so loudly and publicly.
“Most tech geeks don’t resign from their roles because…the tech they’re building could end humanity,” said El-Sayed. “How we change the incentives leading AI labs down this path and protect against these existential risks are defining political questions of our time.”
Editor’s Note: At a moment when the once vaunted model of responsible journalism is overwhelmingly the play thing of self-serving billionaires and their corporate scribes, alternatives of integrity are desperately needed, and ScheerPost is one of them. Please support our independent journalism by contributing to our online donation platform, Network for Good, or send a check to our new PO Box. We can’t thank you enough, and promise to keep bringing you this kind of vital news.
You can also make a donation to our PayPal or subscribe to our Patreon.