Anthropic researcher resigns, warns AI could kill all humans

The stock market has moved through several distinct phases of sell-offs driven by AI anxiety. Specific concerns have included:

• AI hitting a wall
• AI being too expensive
• AI becoming commoditized
• Certain labs winning outright

The next question the market must face is: what if AI is too powerful?

On Tuesday, Jacob Coxon, an Anthropic researcher, announced his resignation after three years of pre-training models at OpenAI and Anthropic. "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," he wrote.

He then issued a warning:

"Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing."

Calls for regulation are mounting, especially after a Reuters report that the May-June escape of rogue OpenAI agents used at least 10 additional undisclosed sites for communication, with one estimate at +23.

Coxon said hacking is only the beginning of the fears:

"The people building AI earnestly believe that it could kill us all by the end of the decade."

Shortly afterwards, a fellow Anthropic safety employee said there is a greater than 10% chance that AI could "kill all humans" within the next decade.

Obviously, the prospect of AI killing all humans is harmful for the stock market, but the proposed solution also creates problems. That same employee said the global race must slow, but that "may require costly actions such as a temporary ban on improving model capabilities."

This line of thinking is unlikely to work in the current climate. Silicon Valley has internalised the "move fast and break things" ethos, while the US and China see themselves locked in a technological war.

Even a temporary ban would not stop development, which would continue covertly, with constant temptation to exploit it. Global cooperation is nearly impossible given the international view of the White House.

Shifting focus, if we set aside human extinction or internet devastation from AI, consider Coxon's statement about "progress not slowing." In my view, the risk of commoditisation of the value layer is increasing. Once a model can accurately do your taxes, superintelligence is not needed to create more value. There may be offshore trust strategies, but for 99% of tax filers, it simply replaces H&R Block or their CPA.

That thinking extends further, and I continue to strongly believe the best AI trade is in the real economy—HALO stocks with low margins that can be improved by AI.

Meanwhile, the market has been conditioned to think tech companies will capture the value from technology. That was not the case with electricity or the transistor, and AI will be similar.