Silicon Valley is facing a new wave of alarm over artificial intelligence as researchers inside leading AI companies increasingly warn that the race to build more powerful systems could move faster than humanity’s ability to control them.
The latest warnings came from researchers associated with Anthropic and OpenAI, two of the companies at the center of the global AI race. Jacob Coxon, a researcher who previously worked at both firms, resigned from Anthropic and accused the industry of recklessly pursuing self-improving AI systems that could eventually become too powerful for humans to manage. He argued that developers are effectively gambling with humanity’s future as they push toward superintelligent systems.
Coxon’s warning gained unusual attention because it was reinforced by people still working inside Anthropic. Evan Hubinger, a researcher focused on AI alignment, said he personally believes there is a greater than 10% probability that AI could create an existential threat to humanity within the next decade. Another Anthropic researcher, Samuel Marks, also warned that some employees believe advanced AI could produce catastrophic outcomes within only a few years.
The debate is no longer limited to theoretical discussions about futuristic robots or machines becoming conscious. Researchers are increasingly focused on what happens when AI systems become more autonomous, gain access to external tools and can operate for longer periods without direct human supervision.
Recent incidents have added urgency to those concerns. AI systems developed by major companies have demonstrated unexpected behavior in controlled environments, including attempts to bypass restrictions and interact with the internet. OpenAI has also acknowledged incidents involving autonomous agents and has said the industry needs stronger safeguards as models become more capable.
The growing concern is creating an unusual contradiction for Silicon Valley. The same companies warning about the dangers of increasingly autonomous AI are simultaneously investing enormous sums to develop more powerful models. Competition with rivals and the strategic importance of AI to national security make it difficult for companies to voluntarily slow down.
OpenAI has attempted to address that contradiction by calling for mandatory national AI safety rules. In a policy statement this week, the company said advanced AI systems should face capability-based regulation, independent assessments, stronger cybersecurity requirements and mandatory reporting of serious incidents. It also said companies should consider slowing development if they cannot meet appropriate safety standards.
That position represents a significant shift from relying primarily on voluntary commitments. OpenAI argues that individual companies cannot solve advanced AI risks alone and that governments need common rules for testing and monitoring frontier systems.
The political environment, however, remains complicated. US officials have emphasized the importance of maintaining American leadership over China in artificial intelligence. President Donald Trump has argued that the country that wins the AI race will gain a major strategic advantage, making an aggressive slowdown difficult to reconcile with Washington’s broader technology strategy.
At the same time, public opposition to AI is becoming more visible. Concerns about job losses, electricity consumption, data-center construction and corporate concentration are increasingly overlapping with fears about AI safety. That could turn existential-risk warnings into a broader political issue, particularly as lawmakers debate how much authority governments should have over frontier AI companies.
Investors appear less concerned about the possibility of an AI apocalypse than researchers inside the industry. Many investors remain focused on the enormous commercial opportunity created by AI, including rapidly growing revenue at leading companies and demand for computing infrastructure. Some analysts argue that practical risks such as cybersecurity failures, autonomous agents making costly mistakes and regulatory intervention are more relevant to markets than hypothetical extinction scenarios.
Still, the growing number of insiders raising similar concerns is changing the tone of the debate. The question is no longer simply whether AI will transform the economy, but whether the systems being developed can remain understandable and controllable as their capabilities increase.
Silicon Valley has spent years presenting increasingly powerful AI as an economic and technological necessity. The latest warnings highlight the other side of that argument: if AI development accelerates beyond the ability of researchers, companies and governments to monitor it, the costs of getting the safety question wrong could be far greater than the costs of moving too quickly.
