Act Now to Keep New Technologies Out of Destructive Hands
Bill Joy is co-founder and Chief Scientist of
Sun Microsystems and was co-chairman of the presidential advisory commission
on information technology. He wrote this article for NPQ responding to
the comments by Jacques Attali, Francis Fukuyama, Amory Lovins and Alvin
and Heidi Toffler on his article "Why the Future Doesnt Need
Us," which appeared in the April issue of Wired Magazine and which
is available from www.wired.com.
Aspen, ColoradoIn the 20th century, nuclear, biological and chemical
weapons of mass destruction (WMDs) were developed by the military, having
little or no commercial value. WMD development and manufacture required
large-scale activities and often-rare raw materials. The knowledge of
how to create these WMDs was not made widely available.
Three new 21st century technologiesgenetic engineering, nanotechnology
and robotics (GNR) are being aggressively pursued by the commercial
sector because of their promise to create almost unimaginable wealth.
Using them we will be able cure many diseases and extend our lives, eliminate
material poverty and grinding physical labor, and heal the Earth.
But, these new technologies may also pose an even greater danger to humankind
than weapons of mass destruction.
It is critical to notice that the scale of the activities needed to practice
the GNR technologies is rapidly declining, and that these do not need
rare raw materials. More and more, the knowledge needed to design with
these technologies is freely available on the Internet. The advancing
power of computing will allow this design to be done on a personal computer,
and manufacture of these designs is becoming inexpensive, using widely
available equipment, for good or evil purposes.
Writing in the Seattle Times, William Calvin, a Neurobiologist at the
University of Washington, wrote: "There is a class of people with
delusional disorders who can remain employed and pretty functional
for decades. Even if they are only one percent of the population, thats
20,000 mostly untreated delusional people in the Puget Sound area. Even
if only one percent of these has the intelligence or education to intentionally
create sustained or widespread harm, its still a pool of 200 high-performing
sociopathic or delusional techies just in the Puget sound area alone."
The malevolent actions of such individuals and small groups using the
GNR technologies pose a large and even mortal danger to our civilization.
This threat will manifest itself, for example, as genetic engineering
techniques provide the ability, perhaps in about 20 years, to use software
to create new, highly contagious and deadly "designer pathogens."
Nanotechnologists have similarly recognized that out of control nanobots
could destroy the biosphere; a first quantitative study of this possibility
of "Global Ecophagy" by Robert Freitas was recently published
in response to the article I wrote on this subject in Wired in April.
His study is quite troubling, showing the clear dangers we face from unrestricted
nanotechnology and the extreme difficulty and enormous scale required
of any "defense."
Such pestilences are beyond our direct experience. The Black Death killed
a third of the population of 14th century Europe and smallpox devastated
the native population in the Americas in the 16th century, but these are
distant historical events. Even the influenza pandemic of 1918 is largely
out of living memory. Antibiotics and improved sanitation have given us
grace from such disasters, at least for a time. But to believe such things
cannot recur is untrue and our failing memory of them is quite dangerous.
Since technologies are creating these new dangers some have hoped for
the answers to be technological too, some sort of defense. But a strong
defense against genetically engineered pathogens would seem to require
a nearly perfectly augmented immune system, which seems quite unlikely
in the timeframe of interest; this may even prove to be impossible without
large-scale reengineering of the germline of our species. A "doomsday
nanoshield" appears to be so outlandishly dangerous that I cant
imagine we would attempt to deploy it. As with nuclear technology, the
destructive offensive uses here have a seemingly deeply sustainable advantage
over defensive efforts.
Robotics poses a different threatthe creation of a new life form
that may escape our control. Some have romantically imagined that we would
achieve near-immortality by becoming robots. But replacing our bodies
with silicon while retaining our humanity will not eliminate the risk
I am discussing here, unless we can somehow simultaneously eliminate human
evil. I think it is also clear that we are not the natural life form in
this imagined new computational substrate, where there would be little
need for sex, no need for relearning, and perhaps no strong notion of
individuality. Its not clearHollywood notwithstandingthat
there is any practical way to protect our continued existence in the presence
of a more powerful robotic species.
An alternative to defense might be escapeto the stars, as Carl Sagan
dreamed. But there seems insufficient time and most people couldnt
go, so we have to look for answers closer to home.
Illimitable Individuals | We live in an economic and political
system that puts its faith strongly in the individual. Freedom was born
in Greece roughly 2500 years ago when we agreed to limit our actions,
this giving birth to modern civilization. This idea of putting faith in
the individual is the basis not only of Greek democracy, but also underlies
the Enlightenment ideal, modern democracy, and capitalism. But we must
now realize that we are creating such incredible power that we cannot
sanely give this power to all individuals, some of whom are clearly not
sane. Our civilization, a society of laws, is grounded in the benefits
we receive by limiting our actions; this social contract is clearly threatened
by illimitable individual power.
The risk of our extinction as we pass through this time of danger has
been estimated to be anywhere from 30% to 50%. I believe that such high
risks are far beyond completely unacceptable and that we must therefore
take some strong action to reduce this risk. Though we cant eliminate
the risk through technology, we should still build some partial defenses
to reduce the risk, and also need to look for sensible non-technical steps.
Historically, the doctrine of Mutually Assured Destruction (MAD), which
seems to have helped to contain the nuclear threat in the past half-century,
is such a non-technical approach; going forward let us hope we can find
less morally repugnant mechanisms.
We can do some simple and obvious things to reduce the risk: have scientists
and technologists take a Hippocratic oath, do assessment of the risks
of new technologies in an open public process, force enterprises which
wish to use dangerous technologies to take insurance against the catastrophic
risk so that less risky paths are favored, limit the access to dangerous
technologies by practicing them in secure international laboratories even
though the work being done in these laboratories is on behalf of commercial
enterprises, and finally relinquish development of the most dangerous
forms of the new technologies such as unrestricted nanotechnology. We
should engage a wide discussion of these and other sensible steps.
If we do not retain, as a civilization, control over these new technologies,
and allow individuals to release self-replicating GNR technologies into
the world, then we will cede control over our future to extreme individuals
A half century ago, Einstein warned us that the nuclear age had come and
changed everything but our way of thinking, and that we were thus drifting
toward unparalleled catastrophe, bequeathing the power of widespread destruction
to the nation states. Now, with the confluence of powerful, widely available
information technology with these new self-replicating GNR technologies,
we are drifting toward a further, even larger potential catastrophe, on
course to put our collective fate in the hands of the extreme individuals
that undeniably exist in the world.
We must act collectively to reduce this grave threat, while getting most
of the benefits of the new technologies. We can do this only if we fully
face the new dangers, act decisively, and soon.
Back to Index