The Machine That Writes Itself and the Door We Have Not Yet Closed

There is a familiar rhythm to how we now greet news about artificial intelligence. A frightening headline appears. Beneath it, someone reaches for the internet’s traditional remedy and proposes pulling the plug. A few hundred others agree. And almost nobody reads what the company actually said. It is worth resisting that rhythm this week, because the underlying announcement from Anthropic is both less lurid and far more demanding than the alarm around it suggests.

What Anthropic said, in a report titled “When AI builds itself” and co-authored by Marina Favaro, who leads its in-house research institute, and the firm’s co-founder Jack Clark, deserves to be read in its own words rather than through the panic. As of last month, more than eighty per cent of the code merged into Anthropic’s own codebase was written by Claude rather than by a human engineer, up from low single digits before its coding tool launched in early 2025. The typical engineer now merges around eight times as much code per quarter as before that shift. The sentence that earns the word unsettling is this: a system increasingly capable of designing and building its own successor points toward what researchers call recursive self-improvement, and that, Anthropic concedes, might increase the risks of humans losing control over AI systems.

A fair reader should pause here to note what kind of evidence this is. The figures are Anthropic’s own, self-reported and unaudited, and they arrive days after the company filed to go public, which gives the sceptics a ready motive to discount the whole exercise as positioning. Note it, then weigh it honestly. A firm talking up its capabilities to investors is an old story; a firm talking up its capabilities in order to argue for being slowed down is a stranger one, and the stranger story is the one in front of us.

Note the verbs, too. Might. Points toward. This is not a confession that the machine has slipped its leash. It is a company describing the slope it is standing on and saying, with a candour the industry rarely manages, that it would rather not skid down it blind. Anthropic went further, suggesting that slowing development might likely be a good thing, while naming the obvious difficulty: a meaningful pause would require rival labs in rival countries to stop at once, and to verify that the others had genuinely stopped. That is an arms control problem dressed in software. We have built verification regimes before, for fissile material and nerve agents, but never for something that can be copied at the speed of light and concealed on a server farm.

The temptation now is to file this under speculation and move on. The firm’s own recent conduct makes that harder. Earlier this year Anthropic was forced to confront a more advanced system than the one the public uses, a frontier model it codenamed Mythos. It had not planned to talk about it at all; a configuration error exposed the model in late March, and the company chose disclosure over denial. What it then disclosed is the part worth sitting with. Mythos had found thousands of high severity vulnerabilities across every major operating system and web browser, and demonstrated the ability to chain them into working exploits with little human guidance. At the point of announcement, by Anthropic’s own account, more than ninety nine per cent of those flaws remained unpatched. The company declined to release the model and stood up a hundred million dollar defensive programme, Glasswing, to help the rest of the industry catch up. Those claims, like the coding figures, rest partly on the firm’s own internal testing and have drawn their share of scrutiny. But the direction of the decision is hard to fake: a company does not withhold its best product, at cost to itself, to win an argument it could more easily win by shipping.

Read that sequence slowly, because it answers the question the headlines keep dodging. This is not, on the most plausible reading, a firm crying wolf to burnish its safety credentials. It is a firm that looked at its own best work, concluded the world’s defences were not ready for it, and said so against its commercial interest. You may think that the responsible call of the decade or an act of unearned paternalism. What you cannot easily call it is theatre. The people closest to the gradient are the ones flinching, and they are flinching at something they claim to be able to measure.

Which is why the plug is the wrong object to reach for. You cannot unplug a capability that several governments already treat as strategic infrastructure and that a dozen laboratories are racing to reproduce; the Mythos episode showed how thin the membrane around such a capability really is, breached not by a rival but by a misconfigured page. The plug was never the point. The door is. There is a narrow window in which choices about verification, disclosure and restraint are still ours to make, and it is closing not because a machine wills it shut but because the economics of competition will close it for us if we wait.

There is a reason these warnings keep arriving from inside the laboratories rather than from the watchtowers outside. The builders see the slope steepen first. In 2023 thousands of technologists signed an open letter calling for a six month pause; nothing paused, because a warning without a mechanism is only a feeling with a press release. The lesson has not changed, only the altitude. A model that writes most of its own code is not the rogue intelligence of the films. It is something more mundane and more exacting: a tool good enough that the labour of governing it has finally overtaken the thrill of building it.

We will not pull the plug. We were never going to. The honest question is whether we can do the unglamorous work a real answer demands, the treaties, the audits, the agreed conditions under which serious firms would actually stop and could prove to one another that they had. Anthropic has said the quiet part aloud and absorbed the discomfort of doing so. Whether its motives are pure or its numbers flattering, the underlying problem does not soften. That discomfort is not a reason to look away. It is a summons. The door is still open, and the room beyond it has gone very quiet, waiting to see whether anyone walks through while there is still time to choose what waits on the other side.