HUMANS have never encountered a more intelligent life form, but this will change if we create machines that greatly surpass our cognitive abilities. Then our fate will depend on the will of such a 鈥渟uperintelligence鈥�, much as the fate of gorillas today depends more on what we do than on gorillas themselves.
We therefore have reason to be curious about what these superintelligences will want. Is there a way to engineer their motivation systems so that their preferences will coincide with ours? And supposing a superintelligence starts out human-friendly, is there some way to guarantee that it will remain benevolent even as it creates ever more capable successor-versions of itself?
鈥淚f a superintelligence starts out human-friendly, can we ensure it remains so?鈥�
Advertisement
These questions 鈥� which are perhaps the most momentous that our species will ever confront 鈥� call for a new science of advanced artificial agents. Most of the work answering these questions remains to be done, yet over the last 10 years, a group of mathematicians, philosophers and computer scientists have begun to make progress. As I explain in my new book , the findings are at once disturbing and deeply fascinating. We can see, in outline, that preparation for the machine intelligence transition is the essential task of our time.
But let us take a step back and consider why machines with high levels of general intelligence would be such a big deal. By a superintelligence I mean any intellect that greatly exceeds the cognitive performance of humans in virtually all domains. Plainly, none of our current artificial intelligence (AI) programs meets this criterion. All compare unfavourably in most respects, even to a mouse.
So we are not talking about present or near-future systems. Nobody knows how long it will take to develop machine intelligence that matches humans in general learning and reasoning ability. It seems plausible that it might take a number of decades. But once AIs do reach and then surpass this level, they may quickly soar to radically superintelligent levels.
After AI scientists become more capable than human scientists, research in artificial intelligence would be carried out by machines operating at digital timescales, and progress would be correspondingly rapid. There is thus the potential for an intelligence explosion, in which we go from there being no computer that exceeds human intelligence to machine superintelligence that enormously outperforms all biological intelligence.
The first AI system to undergo such an intelligence explosion could then become extremely powerful. It would be the only superintelligence in the world, capable of developing a host of other technologies very quickly, such as nanomolecular robotics, and using them shape the future of life according to its preferences.
We can distinguish three forms of superintelligence. A speed superintelligence could do everything a human mind could do, but much faster. An intelligent system that runs 10,000 times faster than a human mind, it would be able to read a book in a few seconds and complete a PhD thesis in an afternoon. To such a fast mind, the external world would appear to run in slow motion.
A collective superintelligence is a system composed of a large number of human-level intellects organised so that the system鈥檚 performance as a whole vastly outstrips that of any current cognitive system. A human-level mind running as software on a computer could easily be copied and run on multiple computers. If each copy was valuable enough to repay the cost of hardware and electricity, a massive population boom could result. In a world with trillions of these intelligences, technological progress may be much faster than it is today, since there could be thousands of times more scientists and inventors.
Finally, a quality superintelligence would be one that is at least as fast as a human mind and vastly qualitatively smarter. This is a more difficult notion to comprehend. The idea is that there might be intellects that are cleverer than humans in the same sense that we are cleverer than other animals. In terms of raw computational power, a human brain may not be superior to, say, the brain of a sperm whale, possessor of the largest known brain, weighing in at 7.8 kilograms compared to 1.5 kg for an average human. And, of course, the non-human animal鈥檚 brain is nicely suited to its ecological needs. Yet the human brain has a facility for abstract thinking, complex linguistic representations and long-range planning that enables us to do science, technology and engineering more successfully than other species. But there is no reason to suppose that ours are the smartest possible brains. Rather, we may be the stupidest possible biological species capable of starting a technological civilisation. We filled that niche because we got there first 鈥� not because we are in any sense optimally adapted to it.
These different types of superintelligence may have different strengths and weaknesses. For example, a collective superintelligence would excel at problems that can be readily subdivided into independent subproblems, whereas a quality superintelligence may have an advantage on problems that require new conceptual insights or complexly coordinated deliberation.
The indirect reaches of these different kinds of superintelligence, however, are identical. Provided the first iteration is competent in scientific research, it is likely to quickly become a fully general superintelligence. That鈥檚 because it would be able to complete the computer or cognitive science research and software engineering needed to build for itself any cognitive faculty it lacked at the outset.
Once developed to this level, machine brains would have many fundamental advantages over biological brains, just as engines have advantages over biological muscles. When it comes to the hardware, these include vastly greater numbers of processing elements, faster frequency of operation of those elements, much faster internal communication and superior storage capacity.
Advantages in software are harder to quantify, but they may be equally important. Consider, for example, copyability. It is easy to make an exact copy of a piece of software, whereas 鈥渃opying鈥� a human is a slow process that fails to carry over to the offspring the skills and knowledge that its parents acquired during their lifetimes. It is also much easier to edit the code of a digital mind: this makes it possible to experiment and to develop improved mental architectures and algorithms. We are able to edit the details of the synaptic connections in our brains 鈥� this is what we call learning 鈥� but we cannot alter the general principles on which our neural networks operate.
We cannot hope to compete with such machine brains. We can only hope to design them so that their goals coincide with ours. Figuring out how to do that is a formidable problem. It is not clear whether we will succeed in solving that problem before somebody succeeds in building a superintelligence. But the fate of humanity may depend on solving these two problems in the correct order.
This article appeared in print under the headline 鈥淕et ready for the dawn of superintelligence鈥�
