From Software as Object to the Engine as Centre
For years we built software the way you build a building. First the walls, then the rooms, then the corridors that tell people where to walk. The application sat at the centre, and intelligence — when it was present at all — lived in the fixed logic of the product itself: not in something that could be corrected or deepened over time, but in paths the designer had drawn before anyone arrived. Whether that made sense or was simply the only thing available — and those are not the same — was not a distinction you could easily see from inside it. Now something has broken open. I cannot say it precisely without either exaggerating the rupture or diminishing it. I include myself among those who still have not fully registered it. Knowing that a framework has changed and actually having the replacement framework are not the same thing. They are coexisting right now — in me and in most of the people I see building in this space.
The application is no longer necessarily the heart. It is becoming a surface — a doorway — while the real operational weight shifts inwards, towards a core that can be trained, corrected, governed.
That is the movement. And instinct has not caught up with it. The old instinct was: we need an application, and inside it we will place some intelligence. What is now appearing works almost the other way round — you have an intelligent engine, and if necessary you give it an interface as a layer of access. Some teams are still building the building while the centre of gravity has already moved. That is not a failure of intelligence. It may be a failure of the available vocabulary, which is not the same problem.
I have sat in rooms with technically capable people — people with real formation, people who were not lost — who still could not answer, with any precision, questions such as these: how do you keep training separate from detection once the system has been running for six months and the boundaries have blurred? How do you reconstruct context after an interruption without quietly importing the contamination of the previous session? How do you audit what went wrong when the failure was not a crash but a slow drift in behaviour that nobody noticed until it had already accumulated? I am putting those questions in a cleaner form than they actually arrived in, which is already a distortion. Databases, APIs, front ends — none of that disappears. But the new centre is not something their formation prepared them for. I do not mean prepared in a merely correctable sense. I mean that the entire direction of the formation pointed elsewhere.
The designer faces something adjacent — not the same problem, and I want to resist the parallelism because it flattens a different difficulty. For a long time design meant screens, journeys, user roles, the texture of interaction. Now it has to reach inside the operational order, and that changes the designer’s relationship to uncertainty. You have to define what the system is permitted to do when it is uncertain. You have to decide what escalates, what gets logged, what gets forgotten, what cannot be forgotten even if someone asks. I have seen systems where nobody had thought carefully about what happens when the engine encounters a case for which it was not trained — and the result was not a clean error. It was a confident wrong answer delivered in the same tone as every other answer. A design failure, yes. But not the kind that appears in a usability test. The kind that appears six months later, when the damage has already been done and the test has already been passed.
The framing I keep running into — AI as a function inside an application, in the way the calendar was once a function, or spell-check — is already too small, and has been too small for longer than people are admitting. I catch myself using it too, which makes it harder to simply name and move past.
The user also changes position, although this is the part about which I find most people resist thinking honestly — and by honestly I mean not dishonestly but incompletely, which is the more common failure. In the old architecture the user handled dead tools. Opened the application, clicked, received an output. The tool did not carry the marks of use. In what is emerging, the user begins to teach — not as a feature, not as a marketing claim, but in the precise and at times uncomfortable sense that the system that operates tomorrow is partly a product of what happened today. I do not think most users are ready for that. I am not sure I have met anyone who is, including people who have thought about it considerably. Whether readiness is even the right frame is something I have not resolved.
What is appearing is a different kind of digital centre: not alive in any biological sense, but capable of continuity, capable of carrying the marks of its formation, and behaving differently — sometimes in ways nobody planned for — according to how it has been governed. The word centre matters. A badly governed engine does not simply produce bad outputs. It produces bad outputs confidently, consistently, and in ways that are difficult to trace back to the original failure — because the failure was not in the output but in the conditions under which the output was formed. That is a different order of problem from a badly designed form.
The decisions now being taken about what gets trained, what gets exposed, what gets protected, will be difficult to reverse.
That is why the ethical question becomes heavier — and I mean heavier in a way the industry has not begun to register, or has registered and set aside, which is not the same thing. Not heavier in the sense of uncomfortable on a conference panel. Heavier in the sense that the wrong decision now is not correctable later without a cost nobody is currently accounting for. Because once a core has acquired continuity and operational depth, its history is partly constitutive of what it is. You cannot easily go back and remove a formation without damaging the structure. There are systems that probably should not be exposed publicly — not because they are dangerous in an obvious sense, but because their value depends on conditions that public exposure would dissolve. I have tried to articulate this more precisely and each time I do I find I have either overstated it or made it sound like a technical constraint when it is not only that. Most people building these systems are not asking that question. They are asking when to launch. I do not know how to make those two questions meet without losing something on both sides. I am not sure I want to.
Nobody planned the economic logic. It followed, whether or not anyone thought it through in advance, and it followed from the same pressure. The most valuable asset in this architecture is not the visible platform. It is the trained core: its operational identity, its memory structure, its discipline under pressure, its ability to remain coherent when it encounters cases at the edge of its formation. I have seen two systems running on the same underlying model, with interfaces that looked almost identical, where one was worth protecting and one was not — and the difference lay entirely in what had been built inside, and in how carefully. The objection that this is not so different from proprietary code is not wrong. The difference is that code could be described, copied, transferred. The density in a trained operational core is not like that. It is relational, historical, and partly tacit. Everything else — the interface, the infrastructure, the model underneath — can be replaced. The density cannot. I have not found a clean way to say that without sounding like I am making a sales argument, but I have also not found a way around it.
Inside the system, none of this becomes simpler. It becomes harder. SOPs that are actually followed — not followed in the specification document and then quietly bypassed the first week there is pressure. Layers of control that are not circumvented when the separation becomes inconvenient, which it will, because separation is always inconvenient for someone. Glossaries that remain stable across the people who use them — and this turns out to be more difficult than any of the technical problems, because the people change and their relationship to the terminology changes with them, and I still have not seen a solution that fully works. Logs that remain readable six months later. Reconstruction protocols that work under the real conditions of an interrupted session, not under the idealised conditions of the protocol document. These are not design preferences, though people treat them that way until they are not. They are load-bearing. When they fail, they fail slowly and silently, and the damage compounds before anyone has the information to name what has gone wrong. I am not sure naming it would be enough anyway.
There is an analogy with human beings that I find I cannot dismiss, even knowing how it tends to land when someone makes it in the wrong room. A person may appear uncomplicated from the outside and still carry memory, contradiction, learning, self-correction, and layered history that shapes each response without announcing itself. The architecture now emerging shares something of that form — not biologically, not even fully as metaphor, but structurally, in ways that feel load-bearing when I press on them. It is historical. It is corrigible in principle but increasingly costly to correct as it deepens. It carries the marks of its formation in ways that are not always visible or traceable. I find myself both wanting to push the analogy further and not trusting myself to do so — and I notice that I have been saying some version of that for two years now without resolving it, which may itself be information. And there is something else underneath that I have not been able to name cleanly: a discomfort that is not quite ethical and not quite technical, and I am not certain it belongs in this essay at all, except that every time I try to remove it the argument feels less true.
I have seen people work in this space who were genuinely competent — people with the technical formation, the intelligence, the commitment — and still building the wrong thing. Not because they were careless. Because the framework they were using was one version behind, and from inside that framework the building looked right. That is the structural shift: not a new set of tools, but a change in what kind of object the work is producing. Tools will increasingly be built as trainable operational cores — with rules, memory, governance, the possibility of sustained formation — rather than as closed objects delivered to users. The people who understand that early will not simply make better applications. They will be working from different premises entirely, which is a different problem from operating better, and I have not seen those two things converge yet. The gap will widen in ways that are not always legible from inside either position. That is not a comfortable thing to say. Nor is it a comfortable thing to be on the wrong side of it.
The application does not disappear. It stays. But it is no longer what the work is organized around, and most people building in this space have not registered that yet, or have registered it and kept building the old way anyway, which amounts to the same thing for now.
The centre passes to the engine. I am not sure what we call that when it has fully happened, because we do not have the word yet, and the ones we are using belong to the previous arrangement.
Whether we are ready to govern that well is a question I keep returning to and do not resolve.
The structure is here. The vocabulary is still catching up.
And that uncertainty is not a position. It is a problem. I am not sure the problem has a solution that does not also cost something I am not prepared to name yet.
