Larry Ellison, Yuval Noah Harari & Nick Bostrom

Oracle CTO & Executive Chairman Larry Ellison raised some eyebrows on a recent earnings call. He said in the near future: “Citizens will be on their best behavior because we are constantly recording and reporting everything that’s going on.” 

This sounds like a behavioral modification program. When humans know they are being watched, and their actions monitored by authorities, they change their actions to conform. For example, people today are less likely to say how they really feel over email, opposed to over drinks with friends, because said email exists forever. French polemicist Michael Foucault argued: “Surveillance is permanent in its effects, even if it is discontinuous in its action.” 

Though “to exist is to differ”, this is turning us into what Yuval Harari describes in his new book Nexus: A Brief History of Information Networks from the Stone Age to AI as Homo sovietcus: “servile and cynical humans, lacking all initiative or independent thinking, passively obeying even the most ludicrous orders, and indifferent to the results of their actions.”

Yet when Ellison says “we” he makes it seem as if AI is one of us. It is not. AI is an independent agent. We have never created such a thing before. This is why many, myself included, are concerned we are risking our own enslavement or extinction. Larry may think AI will accrue massive amounts of power to himself and his colleagues. But, as Harari warns, the AI may instead just take power for itself. Harari gave the Davos canvessari a simple equation in 2020: 

B x C x D = AHH! 

Biological knowledge multiplied by computing power multiplied by data equals the ability to hack humans.

Biometrics and surveillance going “under the skin” may make it so the AI can know not only what we say and search, but also how we feel. Perhaps you hate the governor. In future totalitarian societies, which is precisely what Ellison describes, you will learn to change your thinking so you feel happy when the leader comes on TV. Nick Bostrom worries this will lead to “unprecedentedly effective ways of enforcing a hegemonic perspective (such as by automatically censoring or suppressing discordant opinion and by using sentiment analysis or lie detectors to disincentivize wrongthink.)” Harari told the global elites: “In Stalin’s USSR, the state monitored members of the communist elite more than anyone else. The same will be true of future total surveillance regimes. The higher you are in the hierarchy – the more closely you’ll be watched.”

COVID-19 accelerated the trend. Hannah Arendt wrote in The Origins of Totalitarianism (1951): “What prepares men for totalitarian domination in the non-totalitarian world is the fact that loneliness, once a borderline experience usually suffered in certain marginal social conditions like old age, has become an everyday experience” 

This has implications for citizens who wish to protest against their government. Philosopher Nick Bostrom writes: “Peaceful international coexistence might become harder if national governments could reliably prevent their populace from revolting against war policies…Despots could become more totalitarian and more immune to overthrow, reducing their need to accommodate the wider elite or popular interests.” Harari clearly reads Bostrom, noting: “For thousands of years prophets, poets, and politicians have used language to manipulate and reshape society. Now computers are learning how to do it. And they won’t need to send killer robots to shoot us. They could manipulate human beings to pull the trigger.”   

The value of history here is in understanding where these policies could lead. Constant AI monitoring may sound good for dealing with crime and police brutality. But those who think it will end there do not know much about human nature. Ellison is a brilliant technologist who has created the tools to create a totalitarian police state the likes of which Stalin could never dream of. Or he could give us the best healthcare ever.

But he is not trained as a historian or sociologist. It seems inevitable that if these tools are deployed, they will be abused by the powers that be, as opposed to just being used to keep the streets safe. Harari notes that modern technology has enabled democracy over large distances. But it also allowed Joseph Stalin to terrorize the massive Soviet landmass. 

Ellison is the third richest person alive, so he should set up a history department at Oracle to analyze these risks. We are still in control. Ellison is doing more than creating new computer programs but is “redesigning politics society and culture, and so we had better have a good grasp of politics, society and culture.” 

Indeed.