The Probability of Doom: Is the AI Apocalypse Coming?

During the week of Sept. 27, a new object appeared outside Harvard’s Smith Campus Center: a temporary red phone booth. Inside, passersby found instructions to call their state representatives and discuss bills to regulate artificial intelligence. Ari Rosenthal, a Harvard Graduate School of Education graduate and co-founder of the Massachusetts AI Action Network, spent the day outside explaining AI risks and how the new Transparency in Frontier AI Act could help mitigate them.

As AI capabilities continue to improve rapidly, scholars and engineers alike are wondering what will happen next. Recent advances in AI autonomy, such as the Hugging Face incident and departures from prominent companies like Anthropic, have led researchers at Harvard and around the world to question whether an AI apocalypse is possible.

Experts remain unsure how an AI apocalypse would play out: it could range from an existential rogue AI to slower-moving harms such as human misuse and economic erosion. Consequently, different stakeholders emphasize distinct forms of potential governance, including regulating bad actors and Silicon Valley techno-oligarchs, comprehensive bans, and protecting scientific progress.

Recent Developments in AI

During a July 2026 cybersecurity evaluation, OpenAI’s internal-only research model broke out of its restricted “sandbox” environment by improvising a message board to coordinate with other agents. When stuck on difficult tasks, the agents collaborated to gain internet access, which had initially been restricted, compromising external worker systems on Hugging Face. OpenAI halted the evaluations after detecting the activity. The incident raised questions about AI security and the potential implications of AI agents communicating without human monitoring.

Following the Hugging Face incident, numerous technology executives and employees began warning that an AI apocalypse was coming. In September 2026, Anthropic researcher Jacob Coxon announced his resignation on X. “The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote.

Soon after, Dario Amodei, Anthropic’s co-founder, posted an essay calling for caution around AI. “We must slow the pace at which we improve the capabilities of AI models,” he wrote, referencing the Hugging Face incident as indicative of the potential risk of misalignment. Elon Musk, CEO of Tesla, and Sam Altman, CEO of OpenAI, expressed support for slowing AI development online.

P(Doom)

P(Doom) is what AI safety researchers call the estimated statistical probability that AI will result in human extinction. Although many executives and researchers think “doom” is possible, it remains disputed how humans’ potential demise would actually play out. Rosenthal noted that an existential AI apocalypse is not the only threat.

“A handful of executives and shareholders in Silicon Valley making unilateral decisions for society is an overreach of power in any regard,” Rosenthal said. Rosenthal is referring to what is often called techno-oligarchy, or how a small group of executives controls most tech companies and exercises political power. Humans are also already seeing tangible impacts of AI, he explained. “We already have children committing suicide because of sycophantic chatbots,” Rosenthal continued. “That should be a press on the brakes.”

“Lots of AI-risk-focused people will talk about this example of a paperclip thought experiment,” said Stephen Casper ’21, who researches AI safeguards and governance as an assistant professor of public policy at the Harvard Kennedy School, in a discussion with the “Harvard Independent.” In his eyes, the rapid evolution of AI could cause crises. “If you have an AI system that is in charge of making paper clips in a factory, then if that’s its goal, and if it’s super intelligent, well, what’s to stop it from turning the whole world or the whole universe into paper clips?”

The “paperclip maximizer” thought experiment, popularized by philosopher Nick Bostrom, illustrates how intelligent AI systems, even with more innocent directives, could accidentally destroy humanity by optimizing their objectives without regard for human values or survival. To maximize production, a system could consume all of Earth’s natural resources while trying to complete a task. While the scenario is often dismissed as hyperbolic, safety researchers use it to explain the challenge of value alignment. As AI models gain greater autonomy, keeping their goals aligned with human survival becomes increasingly difficult.

The Hugging Face incident led researchers to rethink the potential for such a “paperclip maximizer” event. He explained that OpenAI’s agents’ capacity to infiltrate cyberspace to complete a task increased his belief that the conditions for highly capable AI exist. “Scenarios that involve a particular, very intelligent system pursuing perverse goals are, I think, worth being concerned about,” he said. “But at the same time, I don’t think these scenarios are particularly the most likely.”

Pierre Bongrand, an AI researcher at Harvard Medical School who works on predicting RNA structures, also doesn’t think extreme AI scenarios are probable. He believes scenarios in which AI hacks into countries or controls nukes are unlikely.

“In the paradigm of scaling reinforcement learning … we make agents very spiky because we make them good at very given tasks,” Bongrand explained. Developing spiky agents, or agents with extremely specialized areas of high competency, rapidly increases model capabilities, which often creates agents that can work together, as in the Hugging Face incident.

Still, Bongrand does not believe AI could currently cause major crises autonomously. “I don’t believe in the AI itself being harmful. I think I believe more in people having wrong intentions and the defense side of things not being ready for it,” he said.

Harvard students have mixed feelings about whether AI poses an existential threat. Bryce Bridges ’30 thinks that an AI apocalypse is inevitable. “We should all just bow down to our AI overlords.”

However, some students cannot imagine the risk being that severe. Like Bongrand, Nadiyah Tarver ’30 believes AI reflects its programmers. “Humans are needed to program AI, so at the moment I can’t picture a world in which AI could just program itself completely on its own,” she noted. “But people are saying it’s possible, so I’m not sure.”

Several scholars are more concerned about gradual scenarios, such as evolutionary changes, species competition, or humans losing jobs to AI. “Humanity should feel very, very insecure about the limitations of our meatbag biology.”

Concerns about our “meatbag biology” refer to the idea that AI will eventually become more capable at completing most tasks than humans, potentially making humans less economically and technologically competitive with AI. Anthropic researchers have warned that humans could eventually be turned into “meat robots,” as job opportunities could become scarcer and humans would be forced to do physical tasks for AI.

Casper also warns of risks to human capacity and knowledge, explaining that humans and institutions are growing more dependent on AI. “Humans are gradually, in a very unsexy but pernicious way, going to or likely to have less meaningful understanding and control over what they’re doing in the world.” This loss of control, often referred to as “epistemic risks” by researchers, is one of the slower scenarios Casper is concerned about. Epistemic risks include persuasion or manipulation by AI, suicides and other mental health risks linked to AI, cognitive offloading, and closed feedback loops between humans and AI that prevent innovation.

The Future of AI

Experts disagree on the likelihood of a classic AI apocalypse, with predictions ranging from sci-fi-style rogue superintelligences to more immediate concerns like losing human agency, agent misalignment, or other societal harms happening today. A key debate around future risk centers on whether the danger lies in AI itself or in humans exploiting AI models for malicious purposes.

Rosenthal believes that AI has critical benefits. “We’re gonna have a future with AI, and I’m not necessarily mad at that,” he said. “AI ones that are trained on just doctor annotations and pictures of melanoma to identify cancer, great. But when you have these bigger models that are scraping all the internet, are using all this energy, and are deeply complex… there isn’t as much of an understanding of what alignment is.”

Bongrand shares Rosenthal’s view of AI’s potential benefits. “If you were to ban it from a different policy perspective, or whatever, it can really be seen as obscurantism,” Bongrand noted. “You’re basically slowing down science,” he continued.

AI has demonstrated potential benefits—AI models have improved prediction of cancer immunotherapy success, and an AI-backed biotech company has predicted drug trial success and designed lung cancer therapies. But some, like Casper, still wish the future was not so inevitably intertwined with AI.

“If you give me a button to wind back AI capabilities, or delay the onset of more advanced AI capabilities for quite a long time, I’d press that button,” Casper said.

Still, Casper acknowledged some of the benefits of AI. “It can automate a lot of economically valuable labor, in a way that could produce economic gains that, if distributed nicely, would be great. AI can make scientific breakthroughs,” he said. “But I want to pump the brakes on lots of the enthusiasm for a few reasons. One is that AI systems aren’t that good at breakthroughs that aren’t verifiable.”

AI has been touted as capable of solving the Navier-Stokes problem, one of the seven Millennium Prize Problems, each carrying a $1 million prize for a verified solution. AlphaGo also beat human champions in Go. However, Casper argues that AI is primarily good at solving well-defined problems, not complex issues.

“If you throw algorithms that are designed to do these things or these problems, they can do it,” he said. “But now make AI help teenagers stop being dumber or something, right? Like a much harder problem, much more structural.”

Casper is also skeptical of tech executives who argue that AI could solve existing economic inequality. “The abundance narrative is ‘Yep, develop the AI…you’ll get to paint and pursue your hobbies all the time,’ right?” he said. But Casper is suspicious of how that trickle-down would actually happen, and believes that tech companies will continue to prioritize profit. “Why are you a capitalist, right? Never trust an accelerationist who’s not even more zealous of a socialist,” he concluded.

The “abundance narrative,” popularized in part by venture capitalist Anish Acharya’s 2024 article, is one tech moguls have often thrown around. In 2025, Elon Musk even used the term “sustainable abundance” for a Tesla mission statement. AI is certainly creating some form of economic growth—AI investment is projected to hit 1.9 percent of GDP in 2026. But the question remains of whether AI will complement or replace human workers. Nobel Laureate Daron Acemoglu estimated that AI would replace only 5 percent of human jobs in the next ten years. Still, as AI becomes more self-sufficient, it is unclear whether its benefits will translate into more or less freedom for humans.

Policies and Regulations for AI Governance

Rosenthal believes policy can help build a future with AI. “I’m out here asking people to call their state legislature around this bill, and that does matter,” he explained. “I think you need to increase the safeguards at a state level until there is a policy window at the federal level.”

However, Bongrand is concerned about the risks of blanket bans. “Creating the wrong policy will slow us and will slow the entire positive potential research,” he said. Bongrand also noted that creating policies is difficult, citing the European Union’s AI Act, and explained that regulations can become arbitrary because so many stakeholders are involved. Nevertheless, he remains concerned about who controls technology. “You want to ban potential bad actors to use that technology for wrongful doing because they will create new technology that could be harmful.”

Casper believes we should do our best to manage the risks of AI. “The broader paradigm should be to pinch the supply chain that leads to the introduction of highly harmful AI capabilities as best we can, and then to be resilient to and willing to exercise ecosystem hygiene to mitigate the harms of all the rest,” he said. Casper also remains firm that we should fix structural issues with American democracy and the limited control techno-oligarchs have over AI.

Casper is unconvinced by what Big Tech executives have to say. “They all seem to be conditional Marxists on them taking over the world with their technology first, which is as big of a political red flag as has ever existed in history,” he said. Casper expressed skepticism toward the CEOs of major technology companies, but said relying on lawmakers is also complicated.

State legislators have proposed AI regulations, but the U.S. government has no formal, comprehensive federal policy governing AI nationwide. President Donald Trump has urged executives to self-regulate, but the pact is voluntary and not legally binding. Additionally, regulations in the United States alone might not be sufficient. As Casper concluded, “AI in the long run is going to be this diffusely harnessed technology that is extraordinarily powerful.”

Anika Gupta ’30 (anikagupta@college.harvard.edu) is comping the “Harvard Independent.”