Quick answer: AI safety is the work of making sure powerful AI systems do what people intend and do not cause serious harm. The topic hit the headlines in September 2026 when Jacob Coxon, a researcher who worked on training models at OpenAI and then Anthropic, resigned from Anthropic with a public warning that AI labs are racing toward self-improving AI without enough safeguards. His post reached well over 100 million views and started one of the biggest public debates about AI risk so far.
This guide explains what AI safety means in plain language, the main risks researchers worry about, what Coxon said and how the industry responded, how AI companies and governments are trying to manage the risks, and what everyday users can do.
In this article 14 sections
- What Is AI Safety?
- The Main AI Risks Researchers Talk About
- A Short History of Public AI Safety Warnings
- Who Is Jacob Coxon?
- What He Warned About
- How the AI Industry Responded
- Why Self-Improving AI Is the Big Concern
- Both Sides of the Debate
- How AI Companies Try to Keep Models Safe
- How Governments Are Responding
- What Everyday Users Can Do
- Where the Debate Goes From Here
- AI Safety Terms Explained
- Frequently Asked Questions
What Is AI Safety?
AI safety covers several related goals:
- Alignment: making sure an AI system’s goals and behavior match what its makers and users actually want.
- Robustness: making sure systems behave reliably, even in unusual situations or when people try to trick them.
- Misuse prevention: stopping people from using AI to cause harm, such as cyberattacks or dangerous weapons.
- Oversight: keeping humans able to understand, monitor and correct AI systems as they become more capable.
Most AI companies now have dedicated safety and alignment teams, and governments are starting to set rules. The debate is about whether those efforts are keeping pace with how fast AI is improving.
The Main AI Risks Researchers Talk About
| Risk | What it means | Example |
|---|---|---|
| Everyday harms | Mistakes, bias and false information | A chatbot giving wrong medical or legal information |
| Misuse | People using AI to cause harm | Scams, deepfakes, cyberattacks |
| Loss of control | AI systems pursuing goals humans did not intend | An autonomous agent taking harmful actions to finish a task |
| Self-improving AI | AI helping to build more powerful AI faster than humans can check it | Labs relying on models for much of their own research |
The last two are what researchers usually mean by “existential risk”, the idea that very advanced AI could one day threaten humanity as a whole. Experts disagree strongly about how likely this is and how soon it could matter.
A Short History of Public AI Safety Warnings
Coxon is not the first insider to raise the alarm. Some earlier moments that shaped the debate:
- March 2023: an open letter from the Future of Life Institute called for a six-month pause on training the most powerful AI systems. Thousands of people signed it.
- May 2023: AI pioneer Geoffrey Hinton left Google so he could speak more freely about AI risks.
- May 2023: a one-sentence statement from the Center for AI Safety, signed by many AI leaders and researchers, said reducing the risk of extinction from AI should be a global priority.
- September 2026: Coxon’s resignation thread became one of the most viewed AI safety warnings ever.
Each moment pushed AI safety further into public discussion, and each drew both support and strong criticism.
Who Is Jacob Coxon?
Coxon is a 27-year-old British researcher and University of Cambridge mathematics graduate. He worked on pretraining, the main stage of building large AI models, at OpenAI from 2023 until July 2026, before joining Anthropic. According to The Wall Street Journal, he was drawn to Anthropic partly by its reputation for safety research.
What made his exit unusual is his role. Many researchers who have left AI labs with public warnings worked on safety teams. Coxon helped build the capabilities he now says he fears. TIME reported that a week before resigning, he had moved from training models to researching how to make them safe.
What He Warned About
In a seven-part thread on X in the second week of September 2026, Coxon argued that leading labs are racing toward self-improving AI with insufficient safeguards. He wrote that systems under development could soon hack almost anything, transform entire fields very quickly and gain real power and resources, and said neither OpenAI nor Anthropic was acting responsibly.
Before leaving, he also messaged colleagues. NBC News reported that he warned superintelligent AI carried a risk of human extinction without more caution and cooperation. He also said he was optimistic about coordination between labs and governments, and urged other researchers to speak up.
The thread spread fast. Deadline reported nearly 76 million views overnight, and TIME later reported 153 million views within 36 hours.
How the AI Industry Responded
Support from inside Anthropic
One of the most surprising responses came from Evan Hubinger, who leads alignment science at Anthropic. He replied within hours, saying Coxon was correct and that researchers there genuinely believe advanced systems could pose an existential threat. A senior safety leader agreeing in public is a major reason the story kept growing.
Critics and skeptics
Not everyone agreed. Hugging Face CEO Clem Delangue dismissed the warning by comparing Coxon to an air conditioning technician talking about climate change. Technology journalist Taylor Lorenz called it “sanctimonious doomer posting,” according to TIME. Some online users also questioned the timing, which came days after OpenAI launched GPT-6 Astra, though no evidence supported those theories.
Policymakers
Many people in Washington shared the post as a sign that government should act, showing how far the debate has moved beyond the tech world.
Why Self-Improving AI Is the Big Concern
Coxon’s warning came as AI labs hand more of their own work to AI. OpenAI reported that its coding agents now run 3.1 agent-workdays for every human workday, which we explain in our AI coding agents guide. Anthropic has said Claude now leads a significant share of its own AI research, covered in our report on Claude’s research role.
The worry is a feedback loop: if AI helps build better AI, progress could speed up faster than humans can test and understand each new system. Supporters of this approach argue that AI can also speed up safety research itself. Both sides agree that how labs manage this step matters a great deal.
Both Sides of the Debate
| Those worried about AI risk argue | Skeptics argue |
|---|---|
| AI is improving faster than our ability to test and control it | Today’s AI still makes basic mistakes and is far from that level |
| Labs face commercial pressure to move fast | Extreme warnings distract from real, current harms like bias and scams |
| Self-improving AI could speed progress dramatically | Progress depends on chips, power and data, which have real limits |
| Rules and cooperation are needed now, before problems appear | Heavy regulation could slow useful innovation and help only big companies |
Understanding both sides helps you judge new AI headlines more calmly.
How AI Companies Try to Keep Models Safe
- Safety testing before launch: companies test models for dangerous abilities, such as helping with cyberattacks or weapons, before releasing them.
- Staged releases: the most capable features are sometimes given first to trusted partners. OpenAI did this with GPT-6 Astra’s cybersecurity abilities, as explained in our GPT-6 Astra guide.
- Usage policies and filters: rules and automatic checks that block clearly harmful requests.
- Alignment research: studying how models think and behave, and training them to be honest and helpful.
- Outside evaluation: independent groups and government institutes testing models.
Critics like Coxon argue these steps are not enough given how fast capabilities are growing. Companies say they are investing heavily and improving their methods.
How Governments Are Responding
Rules are developing quickly and differ by region:
- USA: AI policy comes from a mix of federal actions and state laws, and the debate over how much to regulate frontier AI is ongoing.
- UAE: the UAE has a Minister of State for Artificial Intelligence and a national AI strategy focused on adopting AI responsibly while becoming a global AI hub.
- Europe: the EU AI Act sets rules based on the level of risk an AI system poses.
International cooperation, which Coxon called for, remains limited but is a growing topic at global summits.
What Everyday Users Can Do
For most people, ChatGPT, Claude and Gemini keep working as before, and you do not need to stop using AI. But a few habits make your own use safer:
- Check important facts: AI can be wrong. Verify medical, legal, financial and news information with trusted sources.
- Protect your data: avoid sharing passwords, ID numbers or bank details with chatbots. Learn how to turn off ChatGPT ad tracking.
- Watch for AI scams: deepfake voices and videos are increasingly used in fraud. Confirm unusual requests through another channel.
- Supervise AI agents: if you let AI take actions for you, keep it away from payments and approve important steps yourself.
- Stay informed: follow trusted sources so you understand how AI is changing.
Where the Debate Goes From Here
The AI safety debate is not going away. As labs release more capable models and prepare major business moves, such as Anthropic’s possible stock market listing explained in our Anthropic IPO guide, questions about safety will matter to investors, regulators and users alike. Whether you see Coxon’s warning as alarmist or overdue, it pushed the question of how fast AI should move into the mainstream.
AI Safety Terms Explained
| Term | Meaning |
|---|---|
| AGI | Artificial general intelligence: AI that can do most economically valuable work as well as a person |
| Superintelligence | AI that would be far more capable than humans in almost every area |
| Alignment | Making AI systems pursue the goals their designers and users intend |
| Pretraining | The main training stage, where a model learns from huge amounts of data |
| Existential risk | A risk that could threaten humanity’s survival or long-term future |
Frequently Asked Questions
What is AI safety?
The work of making sure AI systems behave as intended, resist misuse and stay under meaningful human oversight as they become more powerful.
Who is Jacob Coxon?
A 27-year-old British AI researcher who worked on pretraining at OpenAI and then Anthropic, and resigned from Anthropic in September 2026 with a public warning about AI risk.
Why did Jacob Coxon quit Anthropic?
He said leading AI labs are racing toward self-improving systems without enough safeguards and that the risk to humanity is being underestimated.
How did Anthropic respond?
Evan Hubinger, Anthropic’s alignment science lead, publicly said Coxon was right that researchers at the company take extinction risk seriously.
Is AI dangerous to use today?
Everyday tools are generally safe to use with sensible care. The bigger debate concerns future, much more capable systems and how they are developed.
What is self-improving AI?
AI that helps design or train better AI. Researchers worry it could speed up progress faster than humans can test and understand each new system.
Leave a Comment