Act Now to Keep New Technologies Out of Destructive Hands

Bill Joy was co-founder and Chief Scientist of Sun Microsystems and was co-chairman of the presidential advisory commission on information technology. This article appeared in the Summer 2000 issue of NPQ.

Aspen, Colo.--In the 20th century, nuclear, biological and chemical weapons of mass destruction (WMDs) were developed by the military, having little or no commercial value. WMD development and manufacture required large-scale activities and often-rare raw materials. The knowledge of how to create these WMDs was not made widely available.

Three new 21st-century technologies--genetic engineering, nanotechnology and robotics (GNR)--are being aggressively pursued by the commercial sector because of their promise to create almost unimaginable wealth. Using them we will be able cure many diseases and extend our lives, eliminate material poverty and grinding physical labor, and heal the Earth.

But these new technologies may also pose an even greater danger to humankind than weapons of mass destruction.

It is critical to notice that the scale of the activities needed to practice the GNR technologies is rapidly declining, and that these do not need rare raw materials. More and more, the knowledge needed to design with these technologies is freely available on the Internet. The advancing power of computing will allow this design to be done on a personal computer, and manufacture of these designs is becoming inexpensive, using widely available equipment, for good or evil purposes.

Writing in the Seattle Times, William Calvin, a neurobiologist at the University of Washington, said: "There is a class of people with 'delusional disorders' who can remain employed and pretty functional for decades. Even if they are only 1 percent of the population, that's 20,000 mostly untreated delusional people in the Puget Sound area. Even if only 1 percent of these has the intelligence or education to intentionally create sustained or widespread harm, it's still a pool of 200 high-performing sociopathic or delusional techies just in the Puget sound area alone."

The malevolent actions of such individuals and small groups using the GNR technologies pose a large and even mortal danger to our civilization.

This threat will manifest itself, for example, as genetic engineering techniques provide the ability, perhaps in about 20 years, to use software to create new, highly contagious and deadly "designer pathogens." Nanotechnologists have similarly recognized that out-of-control nanobots could destroy the biosphere; a first quantitative study of this possibility of "Global Ecophagy" by Robert Freitas was published in response to the article I wrote on this subject in Wired, April, 2000. His study is quite troubling, showing the clear dangers we face from unrestricted nanotechnology and the extreme difficulty and enormous scale required of any "defense."

Such pestilences are beyond our direct experience. The Black Death killed a third of the population of 14th-century Europe and smallpox devastated the native population in the Americas in the 16th century, but these are distant historical events. Even the influenza pandemic of 1918 is largely out of living memory. Antibiotics and improved sanitation have given us grace from such disasters, at least for a time. But to believe such things cannot recur is untrue and our failing memory of them is quite dangerous.

Since technologies are creating these new dangers some have hoped for the answers to be technological too, some sort of defense. But a strong defense against genetically engineered pathogens would seem to require a nearly perfectly augmented immune system, which seems quite unlikely in the timeframe of interest; this may even prove to be impossible without large-scale reengineering of the germline of our species. A "doomsday nanoshield" appears to be so outlandishly dangerous that I can't imagine we would attempt to deploy it. As with nuclear technology, the destructive offensive uses here have a seemingly deeply sustainable advantage over defensive efforts.

Robotics poses a different threat--the creation of a new life form that may escape our control. Some have romantically imagined that we would achieve near-immortality by becoming robots. But replacing our bodies with silicon while retaining our humanity will not eliminate the risk I am discussing here, unless we can somehow simultaneously eliminate human evil. I think it is also clear that we are not the natural life form in this imagined new computational substrate, where there would be little need for sex, no need for relearning, and perhaps no strong notion of individuality. It's not clear--Hollywood notwithstanding--that there is any practical way to protect our continued existence in the presence of a more powerful robotic species.

An alternative to defense might be escape--to the stars, as Carl Sagan dreamed. But there seems insufficient time and most people couldn't go, so we have to look for answers closer to home.

Illimitable Individuals | We live in an economic and political system that puts its faith strongly in the individual. Freedom was born in Greece roughly 2,500 years ago when we agreed to limit our actions, this giving birth to modern civilization. This idea of putting faith in the individual is the basis not only of Greek democracy, but also underlies the Enlightenment ideal, modern democracy and capitalism. But we must now realize that we are creating such incredible power that we cannot sanely give this power to all individuals, some of whom are clearly not sane. Our civilization, a society of laws, is grounded in the benefits we receive by limiting our actions; this social contract is clearly threatened by illimitable individual power.

The risk of our extinction as we pass through this time of danger has been estimated to be anywhere from 30 percent to 50 percent. I believe that such high risks are far beyond completely unacceptable and that we must therefore take some strong action to reduce this risk. Though we can't eliminate the risk through technology, we should still build some partial defenses to reduce the risk and also need to look for sensible non-technical steps. Historically, the doctrine of Mutually Assured Destruction (MAD), which seems to have helped to contain the nuclear threat in the past half-century, is such a non-technical approach; going forward let us hope we can find less morally repugnant mechanisms.

We can do some simple and obvious things to reduce the risk: have scientists and technologists take a Hippocratic oath, do assessment of the risks of new technologies in an open public process, force enterprises which wish to use dangerous technologies to take insurance against the catastrophic risk so that less risky paths are favored, limit the access to dangerous technologies by practicing them in secure international laboratories even though the work being done in these laboratories is on behalf of commercial enterprises and finally relinquish development of the most dangerous forms of the new technologies such as unrestricted nanotechnology. We should engage a wide discussion of these and other sensible steps.

If we do not retain, as a civilization, control over these new technologies, and allow individuals to release self-replicating GNR technologies into the world, then we will cede control over our future to extreme individuals and accidents.

A half-century ago, Einstein warned us that the nuclear age had come and changed everything but our way of thinking, and that we were thus drifting toward unparalleled catastrophe, bequeathing the power of widespread destruction to the nation states. Now, with the confluence of powerful, widely available information technology with these new self-replicating GNR technologies, we are drifting toward a further, even larger potential catastrophe, on course to put our collective fate in the hands of the extreme individuals that undeniably exist in the world.

We must act collectively to reduce this grave threat, while getting most of the benefits of the new technologies. We can do this only if we fully face the new dangers, act decisively and soon.