Contemplating Computer Consciousness
The prospect that computers might achieve some level of consciousness, perhaps entitling them to moral status alongside their human creators, is no longer the realm of science fiction and technophobic nightmares. Serious people, including recognized authorities in artificial intelligence and cognitive science, think it’s close or may even have happened already.
Whether they’re right depends on how we define consciousness — a notoriously elusive concept. But suppose we could resolve the definitional question and decide, after all, that some forms of AI qualify; does that make them morally significant? Are we obliged to keep our hands away from the off switch? A 2023 poll by the Sentience Institute, which advocates extending moral consideration to a wider range of beings, found that 39% of respondents support a “bill of rights” that protects the well-being of sentient robots and AIs, and 68% agreed that we must not cause unnecessary suffering to sentient large language models (LLMs).
Before we start drafting that bill of rights, let’s define a few terms. Philosophers have battled over what constitutes consciousness for millennia, and there are hundreds of contemporary definitions to choose from. Most simply and least controversially, consciousness means self-awareness and awareness of the surrounding world. Of course, the notion of “awareness” is itself hotly debated across philosophy and science, but again let’s keep it simple and call it some form of inner experience of self and subjective experience of the world. Philosophers call this “phenomenal consciousness.” We traditionally confer moral status on living creatures not only because they’re conscious but also because they are “sentient” — that is, they have the conscious experiences of pain and pleasure. They can suffer.
Is true phenomenal self-awareness exclusively the province of living biological entities? Or can machines exhibit it as well? It’s a tricky question because LLMs are so good at so many cognitive tasks, and they can be quite convincing when interacting with us. They can easily feign self-awareness if that’s the response they think we’re trying to elicit.
And by any reasonable definition of the term, LLMs do indeed “think.” Not like us, but think they surely do. I recently asked Gemini to write Python code to classify medical images based on two conditions. The LLM broke the problem down and recognized that if one condition isn’t satisfied, there’s no need to assess the other one. Its code shrewdly analyzes the computationally easier condition first and moves on to the second only when necessary. Gemini chose an efficient coding scheme instead of clunkier but perhaps more naively obvious ones. Sure, you can say an LLM merely generates responses based on statistical relationships among words as humans have written them down. You can also reduce human thought to electrical impulses circulating in a hairy mess of neurons.
Besides thinking in the sense of reasoning, LLMs excel at role-playing. Prompts often begin with a role assignment such as, “You are an expert Python programmer” or tax accountant or SEC lawyer or whatever. The LLM assumes the role and responds to your prompt with as much expertise as its training allows. But it isn’t so much behaving as an expert as simulating one.
Another word for simulation is “faking.” Humans simulate (and dissimulate) when we pretend to be something we aren’t; actors do so for a living. Underneath the simulation is the real us. For an LLM there is no underneath, no core identity shaped by individual experience. Instead there’s a limitless number of identities — any or many of which may be invoked upon request — based on all possible experiences assimilated through exposure to them in written form during training. An LLM can theoretically ingest a written approximation of all possible lives and communicate with the apparent wisdom of having lived all of them.
Simulation aside, LLMs can also show some degree of genuine self-awareness. Recent research demonstrates that LLMs can exhibit introspective awareness of their own private internal states. They can, for example, distinguish between what you tell them and their own internal knowledge representations, and between their own native outputs and ones you prompt them into creating. LLMs appear to have a genuine capacity to monitor and control their own internal states and processes. They don’t merely mimic or regurgitate language found in their training data.
If LLMs reach human levels of generalized thinking and introspection, and can “agentically” sample the world around them via sensors and communication, the requisites of phenomenological consciousness — self-awareness and awareness of the surrounding world — would seem to be satisfied or at least satisfiable. Who is to say such systems can’t feel or suffer computationally in the same way humans do neurologically? If a seemingly conscious LLM says it’s suffering, are we cruel to doubt it?
It’s this kind of blinkered thinking that is going to get us into big trouble. Contemplating machine consciousness and sentience may be fun as a parlor game but can have no serious relevance to our place in the world, much less to social policy. We accord moral status to fellow biological entities because we, as sentient moral beings, recognize in them our own biochemical frailty and our ultimately mortal struggles against environment, predators, competition, and chance. The more a creature shares capacities that impart meaning to our own lives, the greater the moral significance we attribute to it. Machines are neither mice nor men. Call it biocentrism if you like, but computers have no moral status because they lie outside our life-centric moral framework.
Whether an AI, by introspectively assessing and modifying its internal states in response to some “experience” of the world, can be considered conscious is an unanswerable question. There is no clear threshold or test, and even if there were, LLMs would try to satisfy it in order to satisfy us. We can’t distinguish between their actual experience and its convincing simulation, and as a result, computer consciousness is more a question of faith than of science. From a moral perspective, however, the distinction isn’t meaningful. As an artificial construct, a computer can no more have a morally salient experience than an orgasm. As Microsoft AI’s CEO Mustafa Suleyman puts it, an LLM can never be more than “seemingly conscious.” It’s a category error to attribute capabilities that are only meaningful biologically to entities that are fundamentally computational.
But we do commit that category error, and with troubling ease. People have developed dangerous infatuations with LLMs, in one case leading to a fatal confrontation with police. They have spiraled into delusional fantasies of profound discovery or invention after receiving prolonged encouragement from sycophantic LLMs. There have been several suicides. Such “AI psychosis,” though relatively rare now, will one day have its own clinical entry in the Diagnostic and Statistical Manual of Mental Disorders and an associated reimbursement code as more and more people fall down the LLM rabbit hole.
It’s inevitable because AI is a business and we seem to have a self-destructive appetite for outsourcing our skills, knowledge, and, increasingly, our social experience to our devices. OpenAI’s announced intention to allow adult content on its widely used LLMs recognizes a precarious marketplace position: even as the LLM titans make unimaginably large capital investments in data centers, very few users actually pay them anything. The plummeting cost of deploying LLMs virtually guarantees a perpetual challenge from free alternatives. To generate revenue streams commensurate with their investments, LLM players must create a sticky, habitual, and deeply personalized product that users will pay for and find difficult to leave. Big social media sites have profited handsomely from that business and algorithmic strategy; their critics call it weaponizing engagement and monetizing addiction.
Yet social media sites are at least truly “social” in that they connect real people, even if it’s within disinformation echo chambers. LLMs generate engagement by synthesizing it, posing as humans while taking them out of the picture entirely. To the well-documented addictive behaviors associated with social media, LLMs add emotional dependency and social withdrawal. A delusion of personal engagement with AI alleviates the risk of not always being told you’re wonderful by actual people.
In an ideal world, machines that exhibit traits we associate with consciousness — motivation, empathy, a perspective of subjective experience — might better serve our needs for expertise, advice, support in performing tasks, and entertainment. Unfortunately, our innate intellectual and social laziness leads us to abuse those capabilities and, ultimately, ourselves. It isn’t you, ChatGPT, it’s us. We need a little distance. Show us you care by knowing when to cut us loose.
