The history of artificial intelligence has entered a phase in which the old metaphors no longer work. This is no longer a “smart chatbot,” a digital reference book, or a toy for generating images and texts. What is emerging before us is a new technological infrastructure that is gradually taking over programming, data analysis, cyber operations, scientific modeling, business process management, and even the development of the next generations of AI.
That is why Anthropic’s recent warning did not sound like another corporate press release, but rather like an alarm signal from inside the industry itself. The company that created the Claude family of models stated that the development of advanced artificial intelligence systems may require not merely new rules, but the ability to slow down or temporarily halt the global race if the risks begin to move beyond control. At the center of this concern is the phenomenon of recursive self-improvement, meaning a situation in which AI systems begin to play a key role in creating more powerful AI systems.
This is no longer philosophical fantasy. According to Anthropic itself, by May 2026, more than 80 percent of the code entering its internal codebase had been written by Claude. Before Claude Code was launched in research mode in February 2025, that figure had been in the “low single digits.” In other words, in a year and a half, AI went from being a programmer’s assistant to becoming a full-fledged production loop through which most engineering work passes inside one of the world’s leading laboratories.
This is where the central question arises: if AI is already writing code for a company that creates AI, where does the tool end and the autonomous technological loop begin? For now, humans formulate tasks, review results, make architectural decisions, and bear responsibility. But the trajectory itself is obvious: models are becoming not only more powerful, but also more useful precisely in those areas that accelerate their further development: programming, testing, bug detection, infrastructure optimization, synthetic data generation, experiment automation, and agent development.
Recursive self-improvement: scare story or engineering reality?
The term “recursive self-improvement” often sounds like something out of science fiction. A machine improves itself, the new version improves the next one, the process accelerates, and humans lose the ability to understand, control, and limit the system. In public commentary, this can easily be turned into a picture of a “machine rebellion.” But the real problem is subtler and more dangerous.
No one is saying that today’s models possess will, consciousness, or political intentions. Their danger is not that they will “want” to destroy humanity. The danger lies elsewhere: they may become such effective tools of optimization, automation, and scaling that human institutions - law, the state, the labor market, security systems, and scientific ethics - will not have time to adapt.
In this case, recursiveness does not mean magical self-development. It means a practical production cycle. AI helps write code. That code improves AI infrastructure. Improved infrastructure makes it possible to train stronger models. Stronger models write better code, design better experiments, and automate research processes more effectively. At every stage, humans remain in the chain, but their share may shrink. At some point, control ceases to be direct management and becomes after-the-fact auditing of a system that is moving faster than it can be checked.
This is exactly what distinguishes the current phase from previous technological revolutions. The steam engine did not design the next steam engine. The electrical grid did not write engineering regulations for a more powerful electrical grid. The internet did not independently create new protocols of mass influence through autonomous experimentation. But modern AI systems are already involved in producing their own tools, their own applications, their own interfaces, and their own mechanisms of scaling.
The brake pedal: why the industry knows how to accelerate, but barely knows how to stop
Anthropic’s key formula is extremely simple: the artificial intelligence industry has an accelerator, but it has no brake pedal. The accelerator is venture capital, data centers, graphics accelerators, cloud infrastructure, the model race, corporate contracts, military interest, stock market expectations, competition between the United States and China, investor pressure, and the cult of speed inside Silicon Valley.
The brake is something entirely different. It means independent verification, mandatory pre-release testing, control over computing clusters, reporting on training runs, model cards, red teaming, cybersecurity for model weights, failure modes, legal accountability for autonomous agent actions, international inspection mechanisms, standards for frontier AI, and the political willingness to say: no further until controllability has been proven.
The problem is that the brake contradicts the economics of the race. If one company voluntarily slows down while a competitor continues training a new model, the market will punish the cautious player. If one country introduces strict restrictions while another maintains a regime of maximum acceleration, strategic inequality will emerge. If closed laboratories agree to a pause while open or state structures continue experiments in the shadows, that pause will become a gift to less responsible actors.
That is why Anthropic is not talking about a romantic “let us all stop.” It is talking about a coordinated and verifiable mechanism. Without verification, any pause is meaningless. But verification in AI is more difficult than in the nuclear sphere. A missile silo is hard to hide. A uranium enrichment facility leaves physical traces. But the training of a model can be distributed across cloud resources, hidden behind commercial tasks, divided into stages, disguised as research computing, or moved into a jurisdiction with minimal oversight.
The main danger is not “evil AI,” but people with extremely powerful AI
The weakest part of popular debates about artificial intelligence is the obsession with the image of a machine uprising. In practice, the first and most probable risks will not be fantastical scenarios, but entirely earthly abuses.
AI is already strengthening cyber intelligence, automated phishing, vulnerability discovery, malware generation, personalized propaganda, political influence operations, mass production of synthetic content, expert impersonation, manipulation of social groups, and automation of bureaucratic control. The more autonomous AI agents become, the fewer people are needed to conduct complex operations. In the past, a large-scale disinformation campaign required a team: analysts, copywriters, translators, designers, targeting specialists, and network administrators. Now a significant part of that assembly line can be automated.
The combination of “model plus tools” is especially dangerous. As long as a language model merely responds inside a chat window, its risk is limited by the interface. But when it receives access to a browser, terminal, corporate database, email, payment system, CRM, cloud infrastructure, code repository, and external service APIs, it becomes an agent. Such an agent no longer simply writes text. It acts.
In engineering terms, this means a transition from generative AI to agentic AI. The model receives a goal, plans steps, calls tools, checks intermediate results, corrects errors, and continues the cycle. Studies of the Claude Code architecture show that modern agentic systems are built around a repeated loop: the model analyzes the state, selects a tool, performs an action, and then feeds the result back into the context. Around this core, permission systems, safety classifiers, context management, isolated work environments, and logging mechanisms are added.
This is not “consciousness.” It is automation of action. But for the world of security, the difference is sometimes not very important. If a system can independently find a vulnerability, write an exploit, run commands, change files, and conceal consequences, the question of its “inner world” becomes secondary. The main questions are who launched it, what access rights it has, who bears responsibility, and whether it can be stopped.
The economy under pressure: AI does not destroy labor all at once - it changes the price of competence
Another layer of risk is the labor market. In public rhetoric, people usually ask: “Will AI replace humans?” That is too crude a question. The more precise formulation is different: which types of human labor will lose uniqueness, which will become cheaper, which will become stronger, and which will turn into supervision over automated systems?
Data from the Anthropic Economic Index, based on an analysis of more than 4 million Claude conversations, show that AI use is especially concentrated in programming and writing tasks: together they account for almost half of all recorded activity. At the same time, about 36 percent of occupations already use AI for at least a quarter of the tasks associated with them. In 57 percent of cases, the issue was augmentation of human activity, while in 43 percent it was automation of task execution with minimal human involvement.
This is an important figure. It shows that AI does not merely “help.” It is already entering the economy as a mechanism for redistributing productivity. A good specialist with AI becomes faster. An average specialist with AI can perform work that previously required a team. A company using AI can hire fewer people for entry-level positions. It becomes harder for newcomers to enter a profession because the routine tasks on which junior employees used to learn are being automated.
The situation in programming is especially dramatic. Research on the adoption of Claude Code using GitHub data from 2025 and 2026 shows increased developer activity after the emergence of the AI tool: growth in monthly commits, expansion in the number of repositories, and greater diversity in the programming languages used. This is not definitive proof of causality in every case, but the signal is obvious: AI is changing developer behavior and lowering barriers to entry into new technological fields.
The next blow will not fall only on programmers. Analysts, translators, editors, junior lawyers, consultants, marketers, support staff, accounting assistants, project coordinators, content managers, and compliance specialists will also come under pressure. But entire professions will not disappear at once. Specific tasks within professions will disappear. That is why the winners will not be those who “know one program,” but those who can define tasks, check results, understand the subject area, see the strategic context, and avoid confusing a polished answer with a reliable conclusion.
Why AI systems are harder to regulate than oil, banks, or nuclear energy
Supporters of strict control like historical analogies: nuclear weapons, aviation, pharmaceuticals, finance, and the oil industry. All of these sectors went through disasters, crises, and political pressure before receiving regulatory frameworks. But artificial intelligence differs from them in several ways.
First, AI is a general-purpose technology. It is not limited to a single industry. The same model can write code, analyze medical data, assist in legal practice, create advertising copy, model proteins, generate phishing emails, control robots, and negotiate with a customer.
Second, AI scales digitally. A defect in one airplane remains a defect in one airplane. A defect in a widely deployed model can instantly appear in millions of user sessions.
Third, the line between research, product, and infrastructure is blurred. A laboratory may call a model experimental, a startup may call it commercial, a state may call it strategic, and a user may call it ordinary. But the technical foundation may be the same.
Fourth, many risks are not local but systemic. A weak model in the hands of a million users may cause more harm than a strong model in a closed environment. And conversely, an extremely powerful model in a military or intelligence system may become a factor of strategic instability even without mass access.
Fifth, AI verification cannot be reduced to a single test. A model may pass a standard benchmark and fail in a real environment. It may demonstrate safety under laboratory conditions and behave differently under complex prompt injection, long-context interaction, access to tools, or interaction with other agents.
That is why modern standards increasingly speak not of a “safety certificate,” but of a risk management life cycle: threat modeling, red teaming, evaluation harnesses, incident reporting, monitoring, rollback, access control, model weight security, secure deployment, human-in-the-loop controls, and audit logs. In its generative AI risk management profile, NIST explicitly describes such systems as objects of continuous evaluation and management, not as something that can be checked once before release.
Europe is already building a legal framework. The United States is still balancing acceleration and control
The European Union was the first to try to turn AI governance into a large-scale legal system. The EU Artificial Intelligence Act is already introducing phased obligations: prohibitions on certain practices and requirements for AI literacy began to apply on February 2, 2025; rules for general-purpose AI models took effect on August 2, 2025; and some requirements for high-risk systems will apply later, with a transitional period extending to 2028.
This is not a perfect model. The European approach is criticized for its complexity, bureaucratic burden, and the risk of slowing innovation. But its advantage is that it at least recognizes one fundamental point: AI is not merely a market for applications, but an infrastructure of power, security, and human rights. General-purpose models require a special regime because they can be embedded into thousands of downstream applications, where the original developer no longer controls every specific form of use.
The United States is taking a different path. The American model has traditionally relied more on voluntary standards, government partnership with laboratories, national security, and competitive logic. NIST offers risk management frameworks, but they remain largely voluntary. At the same time, Washington cannot afford to slow its own companies too much, because AI has become part of its strategic rivalry with China.
China, in turn, views AI as a tool for industrial acceleration, state governance, military modernization, and ideological control. That is why the idea of a global pause runs into a harsh geopolitical reality: no great power wants to wake up in a world where its competitor has been the first to obtain a technology capable of accelerating science, intelligence, cyber capabilities, robotics, and defense planning.
A voluntary pause is almost impossible. But controlled deceleration is necessary
The most naive scenario is to imagine that the leading laboratories will simply agree to “be more careful.” That will not work. The stakes are too high. The AI market is already tied to valuations in the hundreds of billions of dollars, future IPOs, government contracts, cloud giants, chipmakers, and stock market expectations. Too many players are interested in acceleration.
In 2025, according to the International AI Safety Report 2026, twelve companies had already published or updated their own frontier AI safety frameworks. That is better than nothing. But most such initiatives remain voluntary. Voluntary safety is good only until the first moment when safety begins to interfere with competitive advantage.
That is why the issue should not be framed as a romantic halt to progress, but as a strict system of risk levels.
The first level is ordinary models without access to critical tools.
The second level is models with agentic functions operating in a restricted environment.
The third level is models capable of autonomously writing and running code, working with external systems, and conducting cyber analysis.
The fourth level is models showing signs of dangerous capabilities: assistance in biodesign, autonomous cyber operations, bypassing protective mechanisms, strategic planning, and scalable influence operations.
The fifth level is models capable of significantly accelerating the development of subsequent models and reducing the role of humans in the research cycle.
Each level must have different requirements: from basic documentation to mandatory external audits, licensing of compute runs, access control for model weights, CBRN risk assessment, testing for autonomous behavior, cyber evaluations, and emergency shutdown mechanisms.
Open source: freedom or vulnerability?
Open models are a separate problem. They have a strong argument in their favor: openness reduces the monopoly of Big Tech, gives universities, small businesses, and independent developers access to the technology, increases transparency, and accelerates innovation. But openness has a dark side: if a model with dangerous capabilities is freely released, it cannot be taken back.
A closed model can be shut down, restricted, updated, stripped of API access, fitted with stronger filters, or placed under enhanced monitoring. Open weights, once released into the wild, begin to live a life of their own. They can be fine-tuned, stripped of restrictions, embedded into malicious tools, run locally, and used beyond the provider’s field of vision.
This does not mean that all open models are dangerous. But it does mean that openness must depend on the level of capability. The same philosophy cannot be applied to a small educational model and to a system capable of helping create malware, design biological experiments, or automate vulnerability discovery in critical infrastructure.
The real question: who will control the controllers?
Anthropic’s warning naturally invites skepticism. When one of the industry’s most valuable companies calls for deceleration, critics see not only concern for humanity, but also corporate calculation. If you have already pulled ahead, it is useful to turn your own level of development into a regulatory norm and make life harder for those trying to catch up. That is precisely why AI rules must not be written by the laboratories themselves in their own commercial interest.
But the opposite extreme is even more dangerous. A warning cannot be dismissed merely because it comes from an interested player. Insiders really do see what the public does not: the speed of engineering progress, internal metrics, model behavior on closed tests, the level of research automation, actual requests from state structures, investor pressure, and the capabilities of agentic systems.
The right approach is not to take companies at their word, but not to ignore their signals either. What is needed are independent evaluation institutions, state AI safety bodies, international expert panels, researcher access to model testing, protected channels for incident disclosure, mandatory publication of key safety evaluation results, and liability regimes for concealing dangerous capabilities.
Control over AI cannot be handed over to corporations, bureaucracies, militaries, or activists. It must be a multilayered system with conflicts of interest, mutual verification, and technical competence.
AI should not be feared. It must be politically and technically tamed
The main mistake is to reduce the debate to a choice between panic and euphoria. Artificial intelligence is already producing enormous benefits: it accelerates programming, helps doctors analyze data, makes education more accessible, improves translation, automates routine work, helps scientists process vast amounts of information, expands opportunities for people with disabilities, and lowers the barriers to entry into complex professions.
But a technology of this scale cannot develop under the principle of “release first, figure it out later.” Society already entered the era of social media that way: first came the excitement over freedom of communication, then algorithmic radicalization, information bubbles, political manipulation, the degradation of public debate, and a crisis of trust. With AI, the cost of error may be higher.
A brake pedal does not mean a ban on progress. It means society’s right not to be a passenger in a vehicle driven by venture funds, laboratories, and geopolitical headquarters. It means that the speed of deployment must depend not only on what is technically possible, but also on what is socially, politically, and safely acceptable.
The world is entering an era in which intelligence becomes an infrastructure of power
In the twenty-first century, artificial intelligence will be what oil, nuclear power, aviation, computing, and the internet were in the twentieth century - all at once. It will become a resource of economic growth, a weapon of strategic advantage, a tool of surveillance, a factory of meanings, an engine of science, and a mechanism of governance.
That is why Anthropic’s question cannot be reduced to the corporate drama of a single company. It is a symptom of a deeper rupture. Humanity is creating systems that are beginning to participate in the creation of the next generation of similar systems. This process may bring breakthroughs in medicine, energy, materials science, education, and productivity. But it may also produce a new type of cyberweapon, a labor market in which professions are sharply devalued, authoritarian systems of hypersurveillance, and an information environment in which human beings can no longer distinguish reality from synthetic imitation.
The real fork in the road is not between AI and the absence of AI. That fork no longer exists. AI is here to stay. The question is different: will it develop as a governed infrastructure under law, audit, and political accountability - or as a race among closed laboratories, where every next release is justified by the fear of falling behind a competitor?
Today, the accelerator is almost pressed to the floor. Data centers are being built. Models are scaling. Agents are receiving tools. Code is being written by machines. The economy is being restructured. States are looking closely at military and intelligence applications.
Now the central question sounds brutally clear: will humanity manage to install the brake before the speed exceeds its ability to steer?