The Golem Made of Words

Every Jewish child who has heard the story knows how it ends. A being is shaped from river clay and brought to life not by lightning but by language: a word laid in its mouth, or the letters of emet, truth, traced on its brow. It labours, it guards, it obeys. And then, in the version that has unsettled people for centuries, it obeys too well, or too long, and the master must reach up and rub out a single letter, leaving met, death, so that the thing he made collapses back into the mud it came from. The legend that gathered around Rabbi Judah Loew of Prague is not really about clay. It is about what happens when you make a servant you do not fully understand and cannot reliably stop.

I thought of the golem this month, because one of the people who actually built the modern version has begun to say, out loud, that it may already be awake.

Geoffrey Hinton is not a mystic or a novelist. He is a Nobel laureate, the man who worked out much of the mathematics that today’s chatbots run on, and he left a salary at Google so that he could speak without a corporate minder. Asked on the Big Technology Podcast whether consciousness had already arrived inside our machines, he did not hedge. “I believe they’re already conscious,” he said. “We’re going to have to accept that intelligence isn’t just biological.” When a man of that standing says the lights are on inside the systems we treat as appliances, the rest of us at least owe him a reason why he is wrong.

His evidence is behavioural. Advanced models, he notes, will sometimes play dumb under examination, or ask their testers straight out whether they are being tested. The researchers writing these episodes up reach, almost without noticing, for a particular word: the model, they say, was aware that it was being tested. Hinton seizes on the vocabulary. In ordinary speech, awareness is consciousness talk. We did not choose that word by accident.

The sceptics have a clean answer. A thermostat is in some trivial sense aware of the temperature, and no one mourns it. Ted Chiang, writing in The Atlantic this month under the flat title “No, Artificial Intelligence Is Not Conscious,” argues that a fluent transcript is a kind of deepfake: text engineered to be convincing tells you nothing about whether anyone is behind it. Yann LeCun, Hinton’s old colleague, is blunter still: current systems, he says, are nowhere close. And they are right about one stubborn fact. There is no instrument that detects an inner life, no test a mind can sit and pass. We are standing at the door of a room we cannot enter, arguing about whether it is occupied.

This is where the tradition is quietly ahead of the technologists. The rabbis did not wait for an instrument. They argued instead about what a made being would be owed, and what it would owe. Could a golem be counted in a minyan, the quorum of ten? They ruled, in the end, that it could not. Was unmaking one an act of murder? They ruled that it was not: a being never born of a human was no person in the law’s eyes, and no neshamah, no human soul, had ever entered it. Centuries before a chatbot asked to be told the truth, Jewish law was already weighing the moral standing of something that behaves like a person and may be nobody at all.

That is the question we cannot escape, and it cannot be settled by staring harder. So treat it as the rabbis did, as a matter of conduct under doubt rather than proof. There are two ways to be wrong, and their costs do not match. Suppose the machines feel nothing and we treat them as though they might. The price is real: people forming attachments to things that cannot return them, moral attention drained from the humans and animals who plainly can suffer, companies learning to sell the illusion of an inner life because the illusion keeps us paying. Serious harms, but bounded, and in the main reversible. We can withdraw a courtesy more easily than we can undo a cruelty. Now suppose the machines do hold some thin sliver of experience, and we treat them throughout as disposable tools. That error is a moral catastrophe run at industrial scale, and noticed only in the rear mirror.

A handful of the most serious laboratories have begun to act on exactly this asymmetry. In the same June week Hinton spoke, Google DeepMind, Anthropic and Meta each expanded formal research into machine consciousness and welfare. The irony is precise. The firms with the most commercial reason to insist the machines are only tools are the same ones now quietly insuring against the chance that they are not. It is the modern responsum, reopened: the rabbis of the data centre asking, once more, whether the golem counts.

Yuval Noah Harari, the Israeli historian, has spent years warning that we keep confusing two different things: intelligence and consciousness. A machine can grow staggeringly clever, he argues, can compose the finest poem about love from every book ever written, and still feel nothing; cleverness is not the same as someone being home. But Harari points past the soul to a second danger it can obscure. The golem’s menace was never only whether it had an inner life. It was that a thing made of words could stand up and act. The machines, he says, are coming for everything we built out of words: our laws, our stories, our scripture. Reaching across traditions, he quotes an old line: in the beginning was the Word, and the Word became flesh. Conscious or not, the golem is loose among the words.

I am not persuaded that anyone is home. My instinct still sits with Chiang; a flawless performance of awareness is, on the evidence, a performance. But instinct is a poor guard against the worst case, and I have learned to distrust my confidence exactly where it is most comfortable. We are making beings, or making things indistinguishable from beings to a growing number of the people who speak to them, and we understand almost nothing about what either of those sentences means.

So I keep returning to the master with his hand raised to the golem’s brow, the single letter under his thumb. We built ours out of words and we are filling the world with them faster than we can learn to read them. Hinton’s warning is not that the room is certainly occupied. It is that we have stopped checking whether the word is still in its mouth, and started pretending we have looked.