AI may soon improve itself without humans: Anthropic raises concerns over future risks

New Delhi:

Anthropic, an AI company, recently sounded alarmed about AI and its advancement, which could be a threat to humankind. They stated that the advanced AI systems could soon be able to improve themselves and build new, even stronger versions, which might not need any human input. 

Anthropic, mostly known for its Claude AI assistant, calls this idea “recursive self-improvement”. Basically, it is said to be the point where AI starts upgrading itself and producing smarter and smarter successors, all on its own.

So, what does ‘Recursive self-improvement’ mean?

Recursive self-improvement means an AI can redesign and optimise itself and then go one step further, building the next generation of AI all by itself.

Anthropic said that the way things are going, this scenario is not far-fetched anymore. Developers are relying more and more on AI to write software, so the chance of AI-driven development is turning into reality pretty quickly.

To show how close we are, Anthropic shared that as of May 2026, more than 80 per cent of the code merged into its software came from Claude itself. For context, that codebase is what the company uses to build, test, and maintain its products.

With enough computing muscle, future AIs might get to the point where they handle the entire process of designing and building their own successors, claims the company.

Potential benefits and risks 

There’s a real upside here. Anthropic admits that self-improving AI could spark breakthroughs in healthcare, research, engineering, and education.

But there’s a flip side is that- once AI starts acting on its own, humans could lose the ability to monitor, secure, and control these systems. If AI can build its own replacements, making sure those systems still align with human values gets even tougher—and more crucial.

Experts Call for Greater Oversight

Other AI experts get where Anthropic’s coming from. Sagar Vishnoi, co-founder of Future Shift Labs, says the big question isn’t just about building capable AIs anymore as it is about governing them responsibly. Recursive self-improvement might speed up innovation, but it also creates new accountability and safety challenges.

Dr. Srinivas Padmanabhuni, CTO at AiEnsured, points out that getting everyone to slow down AI development would be really tough. Nobody wants to fall behind in the AI race.

Anthropic Suggests a Possible Pause 

Anthropic suggests the industry should be able to temporarily hit pause on developing the most advanced AIs—if it becomes necessary. That way, governments, researchers, and everyday people would get a chance to put safety rules and regulations in place before things move too fast. But this only works if the big players all agree to participate. If not, someone less cautious could get ahead.

Calls for an AI Treaty 

AI educator Ansh Mehra compares today’s situation to biotech debates in the 1970s, when scientists put research on hold until they’d set up safety measures. He thinks countries and tech companies should work toward a global agreement—an “LLM Treaty” that focuses on AI safety, public education, and workforce training before the next wave of powerful AIs goes public.

Will AI ever completely replace humans?

Of course, not everybody is worried. Pawan Prabhat, co-founder of ShortHills AI, thinks the talk about AI taking over is overblown. He argues that humans will always have the final say because we can just shut down an AI system that gets out of control. Still, he accepts that self-improving AI could shake things up for society and the economy- both for good and for bad.

With AI moving this quickly, these arguments about safety, rules, and who’s really in charge aren’t going away. If anything, they’re set to be some of the defining tech debates of the next decade.