Existential risk from artificial general intelligence

Short description: Hypothesized risk to human existence

Artificial intelligence

Major goals

* Knowledge reasoning
* Planning
* Machine learning
* Natural language processing
* Computer vision
* Robotics
* Artificial general intelligence

Approaches

* Symbolic
* Deep learning
* Bayesian networks
* Evolutionary algorithms

Philosophy

* Ethics
* Existential risk
* Turing test
* Chinese room
* Control problem
* Friendly AI

History

* Timeline
* Progress
* AI winter

Technology

* Applications
* Projects
* Programming languages

Glossary

* Glossary

* v
* t
* e

Existential risk from artificial general intelligence is the idea that substantial progress in artificial general intelligence (AGI) could result in human extinction or an irreversible global catastrophe.[1][2][3]

One argument goes as follows: human beings dominate other species because the human brain possesses distinctive capabilities other animals lack. If AI were to surpass humanity in general intelligence and become superintelligent, then it could become difficult or impossible to control. Just as the fate of the mountain gorilla depends on human goodwill, so might the fate of humanity depend on the actions of a future machine superintelligence.[4]

The plausibility of existential catastrophe due to AI is widely debated, and hinges in part on whether AGI or superintelligence are achievable, the speed at which dangerous capabilities and behaviors emerge,[5] and whether practical scenarios for AI takeovers exist.[6] Concerns about superintelligence have been voiced by leading computer scientists and tech CEOs such as Geoffrey Hinton,[7] Yoshua Bengio,[8] Alan Turing,[lower-alpha 1] Elon Musk,[11] and OpenAI CEO Sam Altman.[12] In 2022, a survey of AI researchers with a 17% response rate found that the majority of respondents believed there is a 10 percent or greater chance that our inability to control AI will cause an existential catastrophe.[13][14] In 2023, hundreds of AI experts and other notable figures signed a statement that "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."[15] Following increased concern over AI risks, government leaders such as United Kingdom prime minister Rishi Sunak[16] and United Nations Secretary-General António Guterres[17] called for an increased focus on global AI regulation.

Two sources of concern stem from the problems of AI control and alignment: controlling a superintelligent machine or instilling it with human-compatible values may be difficult. Many researchers believe that a superintelligent machine would resist attempts to disable it or change its goals, as that would prevent it from accomplishing its present goals. It would be extremely difficult to align a superintelligence with the full breadth of significant human values and constraints.[1][18][19] In contrast, skeptics such as computer scientist Yann LeCun argue that superintelligent machines will have no desire for self-preservation.[20]

A third source of concern is that a sudden "intelligence explosion" might take an unprepared human race by surprise. Such scenarios consider the possibility that an AI that is more intelligent than its creators might be able to recursively improve itself at an exponentially increasing rate, improving too quickly for its handlers and society at large to control.[1][18] Empirically, examples like AlphaZero teaching itself to play Go show that domain-specific AI systems can sometimes progress from subhuman to superhuman ability very quickly, although such systems do not involve altering their fundamental architecture.[21]

## Contents

* 1 History
* 2 AI capabilities
* 2.1 General Intelligence
* 2.2 Superintelligence
* 2.2.1 Comparison with humans
* 2.2.2 Intelligence explosion
* 2.2.3 Alien mind
* 2.2.4 Limits
* 2.3 Dangerous capabilities
* 2.3.1 Social manipulation
* 2.3.2 Cyberattacks
* 2.3.3 Enhanced pathogens
* 2.4 AI arms race
* 3 Types of existential risk
* 4 AI alignment
* 4.1 Instrumental convergence
* 4.1.1 Resistance to changing goals
* 4.2 Difficulty of specifying goals
* 4.3 Alignment of superintelligences
* 4.4 Difficulty of making a flawless design
* 4.5 Orthogonality thesis
* 4.6 Anthropomorphic arguments
* 4.7 Other sources of risk
* 5 Scenarios
* 5.1 Treacherous turn
* 5.2 Life 3.0
* 6 Perspectives
* 6.1 Endorsement
* 6.2 Skepticism
* 6.3 Popular reaction
* 6.3.1 Public Surveys
* 7 Mitigation
* 7.1 Views on banning and regulation
* 7.1.1 Banning
* 7.1.2 Regulation
* 8 See also
* 9 Notes
* 10 References
* 11 Bibliography

## History

One of the earliest authors to express serious concern that highly advanced machines might pose existential risks to humanity was the novelist Samuel Butler, who wrote in his 1863 essay Darwin among the Machines:[22]

> The upshot is simply a question of time, but that the time will come when the machines will hold the real supremacy over the world and its inhabitants is what no person of a truly philosophic mind can for a moment question.

In 1951, foundational computer scientist Alan Turing wrote the article "Intelligent Machinery, A Heretical Theory", in which he proposed that artificial general intelligences would likely "take control" of the world as they became more intelligent than human beings:

> Let us now assume, for the sake of argument, that [intelligent] machines are a genuine possibility, and look at the consequences of constructing them... There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler's Erewhon.[23]

In 1965, I. J. Good originated the concept now known as an "intelligence explosion" and said the risks were underappreciated:[24]

“  | Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion', and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously.[25] | ”

|
|

Scholars such as Marvin Minsky[26] and I. J. Good himself[27] occasionally expressed concern that a superintelligence could seize control, but issued no call to action. In 2000, computer scientist and Sun co-founder Bill Joy penned an influential essay, "Why The Future Doesn't Need Us", identifying superintelligent robots as a high-tech danger to human survival, alongside nanotechnology and engineered bioplagues.[28]

Nick Bostrom published Superintelligence in 2014, which presented his arguments that superintelligence poses an existential threat.[29] By 2015, public figures such as physicists Stephen Hawking and Nobel laureate Frank Wilczek, computer scientists Stuart J. Russell and Roman Yampolskiy, and entrepreneurs Elon Musk and Bill Gates were expressing concern about the risks of superintelligence.[30][31][32][33] Also in 2015, the Open Letter on Artificial Intelligence highlighted the "great potential of AI" and encouraged more research on how to make it robust and beneficial.[34] In April 2016, Nature warned: "Machines and robots that outperform humans across the board could self-improve beyond our control—and their interests might not align with ours."[35] In 2020, Brian Christian published The Alignment Problem, which details the history of progress on AI alignment up to that time.[36][37]

In March 2023, key figures in AI, such as Musk, signed a letter from the Future of Life Institute calling a halt to advanced AI training until it could be properly regulated.[38] In May 2023, the Center for AI Safety released a statement signed by numerous experts in AI safety and the AI existential risk which stated: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."[39][40]

## AI capabilities

### General Intelligence

Artificial general intelligence (AGI) is typically defined as a system that performs at least as well as humans in most or all intellectual tasks.[41] A 2022 survey of AI researchers found that 90% of respondents expected AGI would be achieved in the next 100 years, and half expected the same by 2061.[42] Meanwhile, some researchers dismiss existential risks from AGI as "science fiction" based on their high confidence that AGI will not be created anytime soon.[43]

Breakthroughs in large language models have led some researchers to reassess their expectations. Notably, Geoffrey Hinton said in 2023 that he recently changed his estimate from "20 to 50 years before we have general purpose A.I." to "20 years or less".[44]

### Superintelligence

In contrast with AGI, Bostrom defines a superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest", including scientific creativity, strategic planning, and social skills.[45][4] He argues that a superintelligence can outmaneuver humans anytime its goals conflict with humans'. It may choose to hide its true intent until humanity cannot stop it.[46][4] Bostrom writes that in order to be safe for humanity, a superintelligence must be aligned with human values and morality, so that it is "fundamentally on our side".[47]

Stephen Hawking argued that superintelligence is physically possible because "there is no physical law precluding particles from being organised in ways that perform even more advanced computations than the arrangements of particles in human brains".[31]

When artificial superintelligence (ASI) may be achieved, if ever, is necessarily less certain than predictions for AGI. In 2023, OpenAI leaders said that not only AGI, but superintelligence may be achieved in less than 10 years.[48]

#### Comparison with humans

Bostrom argues that AI has many advantages over the human brain:[4]

* Speed of computation: biological neurons operate at a maximum frequency of around 200 Hz, compared to potentially multiple GHz for computers.
* Internal communication speed: axons transmit signals at up to 120 m/s, while computers transmit signals at the speed of electricity, or optically at the speed of light.
* Scalability: human intelligence is limited by the size and structure of the brain, and by the efficiency of social communication, while AI may be able to scale by simply adding more hardware.
* Memory: notably working memory, because in humans it is limited to a few chunks of information at a time.
* Reliability: transistors are more reliable than biological neurons, enabling higher precision and requiring less redundancy.
* Duplicability: unlike human brains, AI software and models can be easily copied.
* Editability: the parameters and internal workings of an AI model can easily be modified, unlike the connections in a human brain.
* Memory sharing and learning: AIs may be able to learn from the experiences of other AIs in a manner more efficient than human learning.

#### Intelligence explosion

According to Bostrom, an AI that has an expert-level facility at certain key software engineering tasks could become a superintelligence due to its capability to recursively improve its own algorithms, even if it is initially limited in other domains not directly relevant to engineering.[4][46] This suggests that an intelligence explosion may someday catch humanity unprepared.[4]

The economist Robin Hanson has said that, to launch an intelligence explosion, an AI must become vastly better at software innovation than the rest of the world combined, which he finds implausible.[49]

In a "fast takeoff" scenario, the transition from AGI to superintelligence could take days or months. In a "slow takeoff", it could take years or decades, leaving more time for society to prepare.[50]

#### Alien mind

Superintelligences are sometimes called "alien minds", referring to the idea that their way of thinking and motivations could be vastly different from ours. This is generally considered as a source of risk, making it more difficult to anticipate what a superintelligence might do. It also suggests the possibility that a superintelligence may not particularly value humans by default.[51] To avoid anthropomorphism, superintelligence is sometimes viewed as a powerful optimizer that makes the best decisions to achieve its goals.[4]

The field of "mechanistic interpretability" aims to better understand the inner workings of AI models, potentially allowing us one day to detect signs of deception and misalignment.[52]

#### Limits

It has been argued that there are limitations to what intelligence can achieve. Notably, the chaotic nature or time complexity of some systems could fundamentally limit a superintelligence's ability to predict some aspects of the future, increasing its uncertainty.[53]

### Dangerous capabilities

Advanced AI could generate enhanced pathogens, cyberattacks or manipulate people. These capabilities could be misused by humans,[54] or exploited by the AI itself if misaligned.[4] A full-blown superintelligence could find various ways to gain a decisive influence if it wanted to,[4] but these dangerous capabilities may become available earlier, in weaker and more specialized AI systems. They may cause societal instability and empower malicious actors.[54]

#### Social manipulation

Geoffrey Hinton warned that in the short term, the profusion of AI-generated text, images and videos will make it more difficult to figure out the truth, which he says authoritarian states could exploit to manipulate elections.[55] Such large-scale, personalized manipulation capabilities can increase the existential risk of a worldwide "irreversible totalitarian regime". It could also be used by malicious actors to fracture society and make it dysfunctional.[54]

#### Cyberattacks

AI-enabled cyberattacks are increasingly considered a present and critical threat. According to NATO's technical director of cyberspace, "The number of attacks is increasing exponentially".[56] AI can also be used defensively, to preemptively find and fix vulnerabilities, and detect threats.[57]

AI could improve the "accessibility, success rate, scale, speed, stealth and potency of cyberattacks", potentially causing "significant geopolitical turbulence" if it facilitates attacks more than defense.[54]

Speculatively, such hacking capabilities could be used by an AI system to break out of its local environment, generate revenue, or acquire cloud computing resources.[58]

#### Enhanced pathogens

As AI technology democratizes, it may become easier to engineer more contagious and lethal pathogens. This could enable people with limited skills in synthetic biology to engage in bioterrorism. Dual-use technology that is useful for medicine could be repurposed to create weapons.[54]

For example, in 2022, scientists modified an AI system originally intended for generating non-toxic, therapeutic molecules with the purpose of creating new drugs. The researchers adjusted the system so that toxicity is rewarded rather than penalized. This simple change enabled the AI system to create, in six hours, 40,000 candidate molecules for chemical warfare, including known and novel molecules.[54][59]

### AI arms race

Main article: Artificial intelligence arms race

Companies, state actors, and other organizations competing to develop AI technologies could lead to a race to the bottom of safety standards.[60] As rigorous safety procedures take time and resources, projects that proceed more carefully risk being out-competed by less scrupulous developers.[61][54]

AI could be used to gain military advantages via autonomous lethal weapons, cyberwarfare, or automated decision-making.[54] As an example of autonomous lethal weapons, miniaturized drones could facilitate low-cost assassination of military or civilian targets, a scenario highlighted in the 2017 short film Slaughterbots.[62] AI could be used to gain an edge in decision-making by quickly analyzing large amounts of data and making decisions more quickly and effectively than humans. This could increase the speed and unpredictability of war, especially when accounting for automated retaliation systems.[54][63]

## Types of existential risk

An existential risk is "one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development".[64]

Besides extinction risk, there is the risk that the civilization gets permanently locked into a flawed future. One example is a "value lock-in": If humanity still has moral blind spots similar to slavery in the past, AI might irreversibly entrench it, preventing moral progress. AI could also be used to spread and preserve the set of values of whoever develops it.[65] AI could facilitate large-scale surveillance and indoctrination, which could be used to create a stable repressive worldwide totalitarian regime.[66]

It is difficult or impossible to reliably evaluate whether an advanced AI is sentient and to what degree. But if sentient machines are mass created in the future, engaging in a civilizational path that indefinitely neglects their welfare could be an existential catastrophe.[67][68] Moreover, it may be possible to engineer digital minds that can feel much more happiness than humans with fewer resources, called "super-beneficiaries". Such an opportunity raises the question of how to share the world and which "ethical and political framework" would enable a mutually beneficial coexistence between biological and digital minds.[69]

AI may also drastically improve humanity's future. Toby Ord considers the existential risk a reason for "proceeding with due caution", not for abandoning AI.[66] Max More calls AI an "existential opportunity", highlighting the cost of not developing it.[70]

According to Bostrom, superintelligence could help reduce the existential risk from other powerful technologies such as molecular nanotechnology or synthetic biology. It is thus conceivable that developing superintelligence before other dangerous technologies would reduce the overall existential risk.[4]

## AI alignment

Further information: AI alignment

The alignment problem is the research problem of how to reliably assign objectives, preferences or ethical principles to AIs.

### Instrumental convergence

Further information: Instrumental convergence

An "instrumental" goal is a sub-goal that helps to achieve an agent's ultimate goal. "Instrumental convergence" refers to the fact that some sub-goals are useful for achieving virtually any ultimate goal, such as acquiring resources or self-preservation.[71] Bostrom argues that if an advanced AI's instrumental goals conflict with humanity's goals, the AI might harm humanity in order to acquire more resources or prevent itself from being shut down, but only as a way to achieve its ultimate goal.[4]

Some ways in which an advanced misaligned AI could try to gain more power.[72] Power-seeking behaviors may arise because power is useful to accomplish virtually any objective.[73]

Russell argues that a sufficiently advanced machine "will have self-preservation even if you don't program it in... if you say, 'Fetch the coffee', it can't fetch the coffee if it's dead. So if you give it any goal whatsoever, it has a reason to preserve its own existence to achieve that goal."[20][74]

#### Resistance to changing goals

Even if current goal-based AI programs are not intelligent enough to think of resisting programmer attempts to modify their goal structures, a sufficiently advanced AI might resist any attempts to change its goal structure, just as a pacifist would not want to take a pill that makes them want to kill people. If the AI were superintelligent, it would likely succeed in out-maneuvering its human operators and prevent itself being "turned off" or reprogrammed with a new goal.[4][75] This is particularly relevant to value lock-in scenarios. The field of "corrigibility" studies how to make agents that will not resist attempts to change their goals.[76]

### Difficulty of specifying goals

In the "intelligent agent" model, an AI can loosely be viewed as a machine that chooses whatever action appears to best achieve its set of goals, or "utility function". A utility function gives each possible situation a score that indicates its desirability to the agent. Researchers know how to write utility functions that mean "minimize the average network latency in this specific telecommunications model" or "maximize the number of reward clicks", but do not know how to write a utility function for "maximize human flourishing"; nor is it clear whether such a function meaningfully and unambiguously exists. Furthermore, a utility function that expresses some values but not others will tend to trample over the values the function does not reflect.[77][78]

An additional source of concern is that AI "must reason about what people intend rather than carrying out commands literally", and that it must be able to fluidly solicit human guidance if it is too uncertain about what humans want.[79]

### Alignment of superintelligences

Some researchers believe the alignment problem may be particularly difficult when applied to superintelligences. Their reasoning includes:

* As AI systems increase in capabilities, the potential dangers associated with experimentation grow. This makes iterative, empirical approaches increasingly risky.[4][80]
* If instrumental goal convergence occurs, it may only do so in sufficiently intelligent agents.[81]
* A superintelligence may find unconventional and radical solutions to assigned goals. Bostrom gives the example that if the objective is to make humans smile, a weak AI may perform as intended, while a superintelligence may decide a better solution is to "take control of the world and stick electrodes into the facial muscles of humans to cause constant, beaming grins."[47]
* A superintelligence in creation could gain some awareness of what it is, where it is in development (training, testing, deployment, etc.), and how it is being monitored, and use this information to deceive its handlers.[82] Bostrom writes that such an AI could feign alignment to prevent human interference until it achieves a "decisive strategic advantage" that allows it to take control.[4]
* Analyzing the internals and interpreting the behavior of current large language models is difficult. And it could be even more difficult for larger and more intelligent models.[80]

Alternatively, some find reason to believe superintelligences would be better able to understand morality, human values, and complex goals. Bostrom writes, "A future superintelligence occupies an epistemically superior vantage point: its beliefs are (probably, on most topics) more likely than ours to be true".[4]

In 2023, OpenAI started a project called "Superalignment" to solve the alignment of superintelligences in four years. It called this an especially important challenge, as it said superintelligence may be achieved within a decade. Its strategy involves automating alignment research using artificial intelligence.[83]

### Difficulty of making a flawless design

A Modern Approach, a widely used undergraduate AI textbook,[84][85] says that superintelligence "might mean the end of the human race".[1] It states: "Almost any technology has the potential to cause harm in the wrong hands, but with [superintelligence], we have the new problem that the wrong hands might belong to the technology itself."[1] Even if the system designers have good intentions, two difficulties are common to both AI and non-AI computer systems:[1]

* The system's implementation may contain initially unnoticed but subsequently catastrophic bugs. An analogy is space probes: despite the knowledge that bugs in expensive space probes are hard to fix after launch, engineers have historically not been able to prevent catastrophic bugs from occurring.[86][87]
* No matter how much time is put into pre-deployment design, a system's specifications often result in unintended behavior the first time it encounters a new scenario. For example, Microsoft's Tay behaved inoffensively during pre-deployment testing, but was too easily baited into offensive behavior when it interacted with real users.[20]

AI systems uniquely add a third problem: that even given "correct" requirements, bug-free implementation, and initial good behavior, an AI system's dynamic learning capabilities may cause it to develop unintended behavior, even without unanticipated external scenarios. An AI may partly botch an attempt to design a new generation of itself and accidentally create a successor AI that is more powerful than itself but that no longer maintains the human-compatible moral values preprogrammed into the original AI. For a self-improving AI to be completely safe, it would need not only to be bug-free, but to be able to design successor systems that are also bug-free.[1][88]

### Orthogonality thesis

Some skeptics, such as Timothy B. Lee of Vox, argue that any superintelligent program we create will be subservient to us, that the superintelligence will (as it grows more intelligent and learns more facts about the world) spontaneously learn moral truth compatible with our values and adjust its goals accordingly, or that we are either intrinsically or convergently valuable from the perspective of an artificial intelligence.[89]

Bostrom's "orthogonality thesis" argues instead that, with some technical caveats, almost any level of "intelligence" or "optimization power" can be combined with almost any ultimate goal. If a machine is given the sole purpose to enumerate the decimals of pi, then no moral and ethical rules will stop it from achieving its programmed goal by any means. The machine may utilize all available physical and informational resources to find as many decimals of pi as it can.[90] Bostrom warns against anthropomorphism: a human will set out to accomplish their projects in a manner that they consider reasonable, while an artificial intelligence may hold no regard for its existence or for the welfare of humans around it, instead caring only about completing the task.[91]

Stuart Armstrong argues that the orthogonality thesis follows logically from the philosophical "is-ought distinction" argument against moral realism. He claims that even if there are moral facts provable by any "rational" agent, the orthogonality thesis still holds: it is still possible to create a non-philosophical "optimizing machine" that can strive toward some narrow goal but that has no incentive to discover any "moral facts" such as those that could get in the way of goal completion. Another argument he makes is that any fundamentally friendly AI could be made unfriendly with modifications as simple as negating its utility function. Armstrong further argues that if the orthogonality thesis is false, there must be some immoral goals that AIs can never achieve, which he finds implausible.[92]

Skeptic Michael Chorost explicitly rejects Bostrom's orthogonality thesis, arguing that "by the time [the AI] is in a position to imagine tiling the Earth with solar panels, it'll know that it would be morally wrong to do so."[93] Chorost argues that "an A.I. will need to desire certain states and dislike others. Today's software lacks that ability—and computer scientists have not a clue how to get it there. Without wanting, there's no impetus to do anything. Today's computers can't even want to keep existing, let alone tile the world in solar panels."[93]

### Anthropomorphic arguments

Anthropomorphic arguments assume that, as machines become more intelligent, they will begin to display many human traits, such as morality or a thirst for power. Although anthropomorphic scenarios are common in fiction, most scholars writing about the existential risk of artificial intelligence reject them.[18] Instead, advanced AI systems are typically modeled as intelligent agents.

The academic debate is between those who worry that AI might destroy humanity and those who believe that AI would not destroy humanity. Both sides have claimed that the others' predictions about an AI's behavior are illogical anthropomorphism.[18] The skeptics accuse proponents of anthropomorphism for believing an AGI would naturally desire power; proponents accuse some skeptics of anthropomorphism for believing an AGI would naturally value human ethical norms.[18][94]

Evolutionary psychologist Steven Pinker, a skeptic, argues that "AI dystopias project a parochial alpha-male psychology onto the concept of intelligence. They assume that superhumanly intelligent robots would develop goals like deposing their masters or taking over the world"; perhaps instead "artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no desire to annihilate innocents or dominate the civilization."[95] Facebook's director of AI research, Yann LeCun, has said: "Humans have all kinds of drives that make them do bad things to each other, like the self-preservation instinct... Those drives are programmed into our brain but there is absolutely no reason to build robots that have the same kind of drives".[96]

Despite other differences, the x-risk school[lower-alpha 2] agrees with Pinker that an advanced AI would not destroy humanity out of emotion such as revenge or anger, that questions of consciousness are not relevant to assess the risk,[97] and that computer systems do not generally have a computational equivalent of testosterone.[98] They think that power-seeking or self-preservation behaviors emerge in the AI as a way to achieve its true goals, according to the concept of instrumental convergence.

### Other sources of risk

Bostrom and others have said that a race to be the first to create AGI could lead to shortcuts in safety, or even to violent conflict.[99][100] Roman Yampolskiy and others warn that a malevolent AGI could be created by design, for example by a military, a government, a sociopath, or a corporation, to benefit from, control, or subjugate certain groups of people, as in cybercrime,[101][102] or that a malevolent AGI could choose the goal of increasing human suffering, for example of those people who did not assist it during the information explosion phase.[3]:158

## Scenarios

Further information: Artificial intelligence in fiction and AI takeover

Some scholars have proposed hypothetical scenarios to illustrate some of their concerns.

### Treacherous turn

In Superintelligence, Bostrom expresses concern that even if the timeline for superintelligence turns out to be predictable, researchers might not take sufficient safety precautions, in part because "it could be the case that when dumb, smarter is safe; yet when smart, smarter is more dangerous". He suggests a scenario where, over decades, AI becomes more powerful. Widespread deployment is initially marred by occasional accidents—a driverless bus swerves into the oncoming lane, or a military drone fires into an innocent crowd. Many activists call for tighter oversight and regulation, and some even predict impending catastrophe. But as development continues, the activists are proven wrong. As automotive AI becomes smarter, it suffers fewer accidents; as military robots achieve more precise targeting, they cause less collateral damage. Based on the data, scholars mistakenly infer a broad lesson: the smarter the AI, the safer it is. "And so we boldly go—into the whirling knives", as the superintelligent AI takes a "treacherous turn" and exploits a decisive strategic advantage.[4]

### Life 3.0

In Max Tegmark's 2017 book Life 3.0, a corporation's "Omega team" creates an extremely powerful AI able to moderately improve its own source code in a number of areas. After a certain point, the team chooses to publicly downplay the AI's ability in order to avoid regulation or confiscation of the project. For safety, the team keeps the AI in a box where it is mostly unable to communicate with the outside world, and uses it to make money, by diverse means such as Amazon Mechanical Turk tasks, production of animated films and TV shows, and development of biotech drugs, with profits invested back into further improving AI. The team next tasks the AI with astroturfing an army of pseudonymous citizen journalists and commentators in order to gain political influence to use "for the greater good" to prevent wars. The team faces risks that the AI could try to escape by inserting "backdoors" in the systems it designs, by hidden messages in its produced content, or by using its growing understanding of human behavior to persuade someone into letting it free. The team also faces risks that its decision to box the project will delay the project long enough for another project to overtake it.[103][104]

## Perspectives

The thesis that AI could pose an existential risk provokes a wide range of reactions in the scientific community and in the public at large, but many of the opposing viewpoints share common ground.

Observers tend to agree that AI has significant potential to improve society.[105][106] The Asilomar AI Principles, which contain only those principles agreed to by 90% of the attendees of the Future of Life Institute's Beneficial AI 2017 conference,[104] also agree in principle that "There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities" and "Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources."[107][108]

Conversely, many skeptics agree that ongoing research into the implications of artificial general intelligence is valuable. Skeptic Martin Ford has said: "I think it seems wise to apply something like Dick Cheney's famous '1 Percent Doctrine' to the specter of advanced artificial intelligence: the odds of its occurrence, at least in the foreseeable future, may be very low—but the implications are so dramatic that it should be taken seriously".[109] Similarly, an otherwise skeptical Economist wrote in 2014 that "the implications of introducing a second intelligent species onto Earth are far-reaching enough to deserve hard thinking, even if the prospect seems remote".[46]

AI safety advocates such as Bostrom and Tegmark have criticized the mainstream media's use of "those inane Terminator pictures" to illustrate AI safety concerns: "It can't be much fun to have aspersions cast on one's academic discipline, one's professional community, one's life work ... I call on all sides to practice patience and restraint, and to engage in direct dialogue and collaboration as much as possible."[104][110] Toby Ord wrote that the idea that an AI takeover requires robots is a misconception, arguing that the ability to spread content through the internet is more dangerous, and that the most destructive people in history stood out by their ability to convince, not their physical strength.[66]

A 2022 expert survey with a 17% response rate gave a median expectation of 5–10% for the possibility of human extinction from artificial intelligence.[14][111]

### Endorsement

Further information: Unsolved:Global catastrophic risk

The thesis that AI poses an existential risk, and that this risk needs much more attention than it currently gets, has been endorsed by many computer scientists and public figures, including Alan Turing,[lower-alpha 1] the most-cited computer scientist Geoffrey Hinton,[112] Elon Musk,[11] OpenAI CEO Sam Altman,[12][113] Bill Gates, and Stephen Hawking.[113] Endorsers of the thesis sometimes express bafflement at skeptics: Gates says he does not "understand why some people are not concerned",[114] and Hawking criticized widespread indifference in his 2014 editorial:

“  | So, facing possible futures of incalculable benefits and risks, the experts are surely doing everything possible to ensure the best outcome, right? Wrong. If a superior alien civilisation sent us a message saying, 'We'll arrive in a few decades,' would we just reply, 'OK, call us when you get here—we'll leave the lights on?' Probably not—but this is more or less what is happening with AI.[31] | ”

|
|

Concern over risk from artificial intelligence has led to some high-profile donations and investments. In 2015, Peter Thiel, Amazon Web Services, and Musk and others jointly committed $1 billion to OpenAI, consisting of a for-profit corporation and the nonprofit parent company, which says it aims to champion responsible AI development.[115] Facebook co-founder Dustin Moskovitz has funded and seeded multiple labs working on AI Alignment,[116] notably $5.5 million in 2016 to launch the Centre for Human-Compatible AI led by Professor Stuart Russell.[117] In January 2015, Elon Musk donated $10 million to the Future of Life Institute to fund research on understanding AI decision making. The institute's goal is to "grow wisdom with which we manage" the growing power of technology. Musk also funds companies developing artificial intelligence such as DeepMind and Vicarious to "just keep an eye on what's going on with artificial intelligence,[118] saying "I think there is potentially a dangerous outcome there."[119][120]

In early statements on the topic, Geoffrey Hinton, a major pioneer of deep learning, noted that "there is not a good track record of less intelligent things controlling things of greater intelligence", but said he continued his research because "the prospect of discovery is too sweet".[121][122] In 2023 Hinton quit his job at Google in order to speak out about existential risk from AI. He explained that his increased concern was driven by concerns that superhuman AI might be closer than he previously believed, saying: "I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that." He also remarked, "Look at how it was five years ago and how it is now. Take the difference and propagate it forwards. That's scary."[123]

In his 2020 book The Precipice: Existential Risk and the Future of Humanity, Toby Ord, a Senior Research Fellow at Oxford University's Future of Humanity Institute, estimates the total existential risk from unaligned AI over the next 100 years at about one in ten.[66]

### Skepticism

| Parts of this unsolved (those related to the intelligence level of AI models, stated prior to significant advancements in the state of the art) need to be updated. Please update this unsolved to reflect recent events or newly available information. (June 2020)

|

The thesis that AI can pose existential risk has many detractors. Skeptics sometimes charge that the thesis is crypto-religious, with an irrational belief in the possibility of superintelligence replacing an irrational belief in an omnipotent God. Jaron Lanier argued in 2014 that the whole concept that then-current machines were in any way intelligent was "an illusion" and a "stupendous con" by the wealthy.[124][125]

Some criticism argues that AGI is unlikely in the short term. AI researcher Rodney Brooks wrote in 2014, "I think it is a mistake to be worrying about us developing malevolent AI anytime in the next few hundred years. I think the worry stems from a fundamental error in not distinguishing the difference between the very real recent advances in a particular aspect of AI and the enormity and complexity of building sentient volitional intelligence."[126] Baidu Vice President Andrew Ng said in 2015 that AI existential risk is "like worrying about overpopulation on Mars when we have not even set foot on the planet yet."[95][127] For the danger of uncontrolled advanced AI to be realized, the hypothetical AI may have to overpower or outthink any human, which some experts argue is a possibility far enough in the future to not be worth researching.[128][129]

Skeptics who believe AGI is not a short-term possibility often argue that concern about existential risk from AI is unhelpful because it could distract people from more immediate concerns about AI's impact, because it could lead to government regulation or make it more difficult to fund AI research, or because it could damage the field's reputation.[130] AI and AI ethics researchers Timnit Gebru, Emily M. Bender, Margaret Mitchell, and Angelina McMillan-Major have argued that discussion of existential risk distracts from the immediate, ongoing harms from AI taking place today, such as data theft, worker exploitation, bias, and concentration of power.[131] They further note the association between those warning of existential risk and longtermism, which they describe as a "dangerous ideology" for its unscientific and utopian nature.[132]

Wired editor Kevin Kelly argues that natural intelligence is more nuanced than AGI proponents believe, and that intelligence alone is not enough to achieve major scientific and societal breakthroughs. He argues that intelligence consists of many dimensions that are not well understood, and that conceptions of an 'intelligence ladder' are misleading. He notes the crucial role real-world experiments play in the scientific method, and that intelligence alone is no substitute for these.[133]

Meta chief AI scientist Yann LeCun says that AI can be made safe via continuous and iterative refinement, similar to what happened in the past with cars or rockets, and that AI will have no desire to take control.[134]

Several skeptics emphasize the potential near-term benefits of AI. Meta CEO Mark Zuckerberg believes AI will "unlock a huge amount of positive things", such as curing disease and increasing the safety of autonomous cars.[135] Physicist Michio Kaku, an AI risk skeptic, posits a deterministically positive outcome. In Physics of the Future he asserts that "It will take many decades for robots to ascend" up a scale of consciousness, and that in the meantime corporations such as Hanson Robotics will likely succeed in creating robots that are "capable of love and earning a place in the extended human family".[136][137]

### Popular reaction

In a 2014 article in The Atlantic, James Hamblin noted that most people do not care about artificial general intelligence, and characterized his own gut reaction to the topic as: "Get out of here. I have a hundred thousand things I am concerned about at this exact moment. Do I seriously need to add to that a technological singularity?"[124]

During a 2016 Wired interview of President Barack Obama and MIT Media Lab's Joi Ito, Ito said:

“  | There are a few people who believe that there is a fairly high-percentage chance that a generalized AI will happen in the next 10 years. But the way I look at it is that in order for that to happen, we're going to need a dozen or two different breakthroughs. So you can monitor when you think these breakthroughs will happen.  | ”

|
|

Obama added:[138][139]

“  | And you just have to have somebody close to the power cord. [Laughs.] Right when you see it about to happen, you gotta yank that electricity out of the wall, man.  | ”

|
|

Hillary Clinton wrote in What Happened:

“  | Technologists... have warned that artificial intelligence could one day pose an existential security threat. Musk has called it "the greatest risk we face as a civilization". Think about it: Have you ever seen a movie where the machines start thinking for themselves that ends well? Every time I went out to Silicon Valley during the campaign, I came home more alarmed about this. My staff lived in fear that I'd start talking about "the rise of the robots" in some Iowa town hall. Maybe I should have. In any case, policy makers need to keep up with technology as it races ahead, instead of always playing catch-up.[140] | ”

|
|

One techno-utopian viewpoint expressed in some popular fiction is that AGI may tend toward peace-building.[141]

#### Public Surveys

In 2018, a SurveyMonkey poll of the American public by USA Today found 68% thought the real current threat remains "human intelligence", but also found that 43% said superintelligent AI, if it were to happen, would result in "more harm than good", and that 38% said it would do "equal amounts of harm and good".[142]

An April 2023 YouGov poll of US adults found 46% of respondents were "somewhat concerned" or "very concerned" about "the possibility that AI will cause the end of the human race on Earth," compared with 40% who were "not very concerned" or "not at all concerned."[143]

According to an August 2023 survey by the Pew Research Centers, 52% of Americans felt more concerned than excited about new AI developments; nearly a third felt as equally concerned and excited. More Americans saw that AI would have a more helpful than hurtful impact on several areas, from healthcare and vehicle safety to product search and customer service. The main exception is privacy: 53% of Americans believe AI will lead to higher exposure of their personal information.[144]

## Mitigation

Many scholars concerned about the AGI existential risk believe that the best approach is to conduct substantial research into solving the difficult "control problem": what types of safeguards, algorithms, or architectures can programmers implement to maximize the probability that their recursively-improving AI would continue to behave in a friendly manner after it reaches superintelligence?[4][145] Social measures may mitigate the AGI existential risk;[146][147] for instance, one recommendation is for a UN-sponsored "Benevolent AGI Treaty" that would ensure only altruistic AGIs be created.[148] Similarly, an arms control approach has been suggested, as has a global peace treaty grounded in the international relations theory of conforming instrumentalism, with an artificial superintelligence potentially being a signatory.[149][150]

Researchers at Google have proposed research into general "AI safety" issues to simultaneously mitigate both short-term risks from narrow AI and long-term risks from AGI.[151][152] A 2020 estimate places global spending on AI existential risk somewhere between $10 and $50 million, compared with global spending on AI around perhaps $40 billion. Bostrom suggests that funding of protective technologies should be prioritized over potentially dangerous ones.[76] Some funders, such as Musk, propose that radical human cognitive enhancement could be such a technology, for example direct neural linking between human and machine; others argue that enhancement technologies may themselves pose an existential risk.[153][154] Researchers could closely monitor or attempt to "box in" an initial AI at a risk of becoming too powerful. A dominant superintelligent AI, if aligned with human interests, might itself take action to mitigate the risk of takeover by rival AI, although the creation of the dominant AI could itself pose an existential risk.[155]

Institutions such as the Alignment Research Center,[156] the Machine Intelligence Research Institute, the Future of Humanity Institute,[157][158] the Future of Life Institute, the Centre for the Study of Existential Risk, and the Center for Human-Compatible AI[159] are involved in research into AI risk and safety.

### Views on banning and regulation

#### Banning

Some scholars have said that even if AGI poses an existential risk, attempting to ban research into artificial intelligence is still unwise, and probably futile.[160][161][162] Skeptics argue that regulation of AI is completely valueless, as no existential risk exists. But scholars who believe in existential risk say it is difficult to depend on people from the AI industry to regulate or constrain AI research because it directly contradicts their personal interests.[163] The scholars also agree with the skeptics that banning research would be unwise, as research could be moved to countries with looser regulations or conducted covertly.[163] The latter issue is particularly relevant, as artificial intelligence research can be done on a small scale without substantial infrastructure or resources.[164][165] Two additional hypothetical difficulties with bans (or other regulation) are that technology entrepreneurs statistically tend toward general skepticism about government regulation, and that businesses could have a strong incentive to (and might well succeed at) fighting regulation and politicizing the underlying debate.[166]

#### Regulation

In March 2023, the Future of Life Institute drafted Pause Giant AI Experiments: An Open Letter, a petition calling on major AI developers to agree on a verifiable six-month pause of any systems "more powerful than GPT-4" and to use that time to institute a framework for ensuring safety; or, failing that, for governments to step in with a moratorium. The letter referred to the possibility of "a profound change in the history of life on Earth" as well as potential risks of AI-generated propaganda, loss of jobs, human obsolescence, and society-wide loss of control.[106][167] The letter was signed by prominent personalities in AI but also criticized for not focusing on current harms,[168] missing technical nuance about when to pause,[169] or not going far enough.[170]

Musk called for some sort of regulation of AI development as early as 2017. According to NPR, he is "clearly not thrilled" to be advocating government scrutiny that could impact his own industry, but believes the risks of going completely without oversight are too high: "Normally the way regulations are set up is when a bunch of bad things happen, there's a public outcry, and after many years a regulatory agency is set up to regulate that industry. It takes forever. That, in the past, has been bad but not something which represented a fundamental risk to the existence of civilisation." Musk states the first step would be for the government to gain "insight" into the actual status of current research, warning that "Once there is awareness, people will be extremely afraid... [as] they should be." In response, politicians expressed skepticism about the wisdom of regulating a technology that is still in development.[171][172][173]

In 2021 the United Nations (UN) considered banning autonomous lethal weapons, but consensus could not be reached.[174] In July 2023 the UN Security Council for the first time held a session to consider the risks and threats posed by AI to world peace and stability, along with potential benefits.[175][176] Secretary-General António Guterres advocated the creation of a global watchdog to oversee the emerging technology, saying, "Generative AI has enormous potential for good and evil at scale. Its creators themselves have warned that much bigger, potentially catastrophic and existential risks lie ahead."[17] At the council session, Russia said it believes AI risks are too poorly understood to be considered a threat to global stability. China argued against strict global regulation, saying countries should be able to develop their own rules, while also saying they opposed the use of AI to "create military hegemony or undermine the sovereignty of a country."[175]

Regulation of conscious AGIs focuses on integrating them with existing human society and can be divided into considerations of their legal standing and of their moral rights.[177] AI arms control will likely require the institutionalization of new international norms embodied in effective technical specifications combined with active monitoring and informal diplomacy by communities of experts, together with a legal and political verification process.[178][112]

In July 2023, the US government secured voluntary safety commitments from major tech companies, including OpenAI, Amazon, Google, Meta, and Microsoft. The companies agreed to implement safeguards, including third-party oversight and security testing by independent experts, to address concerns related to AI's potential risks and societal harms. The parties framed the commitments as an intermediate step while regulations are formed. Amba Kak, executive director of the AI Now Institute, said, "A closed-door deliberation with corporate actors resulting in voluntary safeguards isn't enough" and called for public deliberation and regulations of the kind that companies would not voluntarily agree to.[179][180]

In October 2023, U.S. President Joe Biden issued an executive order on the "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence".[181] Alongside other requirements, the order mandates the development of guidelines for AI models that permit the "evasion of human control."

## See also

* Appeal to probability
* AI alignment
* AI safety
* Butlerian Jihad
* Effective altruism § Long-term future and global catastrophic risks
* Gray goo
* Human Compatible
* Lethal autonomous weapon
* Paperclip maximizer
* Philosophy of artificial intelligence
* Robot ethics § In popular culture
* Statement on AI risk of extinction
* Superintelligence
* Suffering risks
* System accident

## Notes

1. ↑ 1.0 1.1 In a 1951 lecture[9] Turing argued that "It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler's Erewhon." Also in a lecture broadcast on the BBC[10] he expressed the opinion: "If a machine can think, it might think more intelligently than we do, and then where should we be? Even if we could keep the machines in a subservient position, for instance by turning off the power at strategic moments, we should, as a species, feel greatly humbled. . . . This new danger . . . is certainly something which can give us anxiety."
2. ↑ as interpreted by Seth Baum

## References

1. ↑ 1.0 1.1 1.2 1.3 1.4 1.5 1.6 Russell, Stuart; Norvig, Peter (2009). "26.3: The Ethics and Risks of Developing Artificial Intelligence". Artificial Intelligence: A Modern Approach. Prentice Hall. ISBN 978-0-13-604259-4.
2. ↑ Bostrom, Nick (2002). "Existential risks". Journal of Evolution and Technology 9 (1): 1–31.
3. ↑ 3.0 3.1 Turchin, Alexey; Denkenberger, David (2018-05-03). "Classification of global catastrophic risks connected with artificial intelligence". AI & Society 35 (1): 147–163. doi:10.1007/s00146-018-0845-5. ISSN 0951-5666. https://philarchive.org/rec/TURCOG-2.
4. ↑ 4.00 4.01 4.02 4.03 4.04 4.05 4.06 4.07 4.08 4.09 4.10 4.11 4.12 4.13 4.14 4.15 4.16 Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies (First ed.). Oxford University Press. ISBN 978-0-19-967811-2.
5. ↑ Vynck, Gerrit De (2023-05-23). "The debate over whether AI will destroy us is dividing Silicon Valley" (in en-US). Washington Post. ISSN 0190-8286. https://www.washingtonpost.com/technology/2023/05/20/ai-existential-risk-debate/.
6. ↑ Metz, Cade (2023-06-10). "How Could A.I. Destroy Humanity?" (in en-US). The New York Times. ISSN 0362-4331. https://www.nytimes.com/2023/06/10/technology/ai-humanity.html.
7. ↑ ""Godfather of artificial intelligence" weighs in on the past and potential of AI" (in en-US). 25 March 2023. https://www.cbsnews.com/news/godfather-of-artificial-intelligence-weighs-in-on-the-past-and-potential-of-artificial-intelligence/.
8. ↑ "How Rogue AIs may Arise" (in en-US). 26 May 2023. https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/.
9. ↑ Turing, Alan (1951). Intelligent machinery, a heretical theory (Speech). Lecture given to '51 Society'. Manchester: The Turing Digital Archive. Archived from the original on September 26, 2022. Retrieved 2022-07-22.
10. ↑ Turing, Alan (15 May 1951). "Can digital computers think?". Automatic Calculating Machines. Episode 2. BBC. Can digital computers think?.
11. ↑ 11.0 11.1 Parkin, Simon (14 June 2015). "Science fiction no more? Channel 4's Humans and our rogue AI obsessions" (in en). The Guardian. https://www.theguardian.com/tv-and-radio/2015/jun/14/science-fiction-no-more-humans-tv-artificial-intelligence.
12. ↑ 12.0 12.1 Jackson, Sarah. "The CEO of the company behind AI chatbot ChatGPT says the worst-case scenario for artificial intelligence is 'lights out for all of us'" (in en-US). https://www.businessinsider.com/chatgpt-openai-ceo-worst-case-ai-lights-out-for-all-2023-1.
13. ↑ "The AI Dilemma" (in en). https://www.humanetech.com/podcast/the-ai-dilemma. "50% of AI researchers believe there's a 10% or greater chance that humans go extinct from our inability to control AI."
14. ↑ 14.0 14.1 "2022 Expert Survey on Progress in AI" (in en-US). 2022-08-04. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/.
15. ↑ Roose, Kevin (2023-05-30). "A.I. Poses 'Risk of Extinction,' Industry Leaders Warn" (in en-US). The New York Times. ISSN 0362-4331. https://www.nytimes.com/2023/05/30/technology/ai-threat-warning.html.
16. ↑ Sunak, Rishi (June 14, 2023). "Rishi Sunak Wants the U.K. to Be a Key Player in Global AI Regulation". Time. https://time.com/6287253/uk-rishi-sunak-ai-regulation/.
17. ↑ 17.0 17.1 Fung, Brian (2023-07-18). "UN Secretary General embraces calls for a new UN agency on AI in the face of 'potentially catastrophic and existential risks'" (in en). CNN Business. https://www.cnn.com/2023/07/18/tech/un-ai-agency/index.html.
18. ↑ 18.0 18.1 18.2 18.3 18.4 Yudkowsky, Eliezer (2008). "Artificial Intelligence as a Positive and Negative Factor in Global Risk". Global Catastrophic Risks: 308–345. Bibcode: 2008gcr..book..303Y. https://intelligence.org/files/AIPosNegFactor.pdf. Retrieved 27 August 2018.
19. ↑ "Research Priorities for Robust and Beneficial Artificial Intelligence". AI Magazine (Association for the Advancement of Artificial Intelligence): 105–114. 2015. Bibcode: 2016arXiv160203506R. https://futureoflife.org/data/documents/research_priorities.pdf. Retrieved 10 August 2019. , cited in "AI Open Letter - Future of Life Institute". January 2015. https://futureoflife.org/ai-open-letter.
20. ↑ 20.0 20.1 20.2 Dowd, Maureen (April 2017). "Elon Musk's Billion-Dollar Crusade to Stop the A.I. Apocalypse" (in en). The Hive. https://www.vanityfair.com/news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai-space-x.
21. ↑ "AlphaGo Zero: Starting from scratch" (in en). https://www.deepmind.com/blog/alphago-zero-starting-from-scratch.
22. ↑ Breuer, Hans-Peter. 'Samuel Butler's "the Book of the Machines" and the Argument from Design.' Modern Philology, Vol. 72, No. 4 (May 1975), pp. 365–383
23. ↑ Turing, A M (1996). "Intelligent Machinery, A Heretical Theory". 1951, Reprinted Philosophia Mathematica 4 (3): 256–260. doi:10.1093/philmat/4.3.256.
24. ↑ Hilliard, Mark (2017). "The AI apocalypse: will the human race soon be terminated?" (in en). The Irish Times. https://www.irishtimes.com/business/innovation/the-ai-apocalypse-will-the-human-race-soon-be-terminated-1.3019220.
25. ↑ I.J. Good, "Speculations Concerning the First Ultraintelligent Machine" (HTML ), Advances in Computers, vol. 6, 1965.
26. ↑ Russell, Stuart J.; Norvig, Peter (2003). "Section 26.3: The Ethics and Risks of Developing Artificial Intelligence". Artificial Intelligence: A Modern Approach. Upper Saddle River, N.J.: Prentice Hall. ISBN 978-0-13-790395-5. "Similarly, Marvin Minsky once suggested that an AI program designed to solve the Riemann Hypothesis might end up taking over all the resources of Earth to build more powerful supercomputers to help achieve its goal."
27. ↑ Barrat, James (2013). Our final invention: artificial intelligence and the end of the human era (First ed.). New York: St. Martin's Press. ISBN 978-0-312-62237-4. "In the bio, playfully written in the third person, Good summarized his life's milestones, including a probably never before seen account of his work at Bletchley Park with Turing. But here's what he wrote in 1998 about the first superintelligence, and his late-in-the-game U-turn: [The paper] 'Speculations Concerning the First Ultra-intelligent Machine' (1965) . . . began: 'The survival of man depends on the early construction of an ultra-intelligent machine.' Those were his [Good's] words during the Cold War, and he now suspects that 'survival' should be replaced by 'extinction.' He thinks that, because of international competition, we cannot prevent the machines from taking over. He thinks we are lemmings. He said also that 'probably Man will construct the deus ex machina in his own image.'"
28. ↑ Anderson, Kurt (26 November 2014). "Enthusiasts and Skeptics Debate Artificial Intelligence". Vanity Fair. https://www.vanityfair.com/news/tech/2014/11/artificial-intelligence-singularity-theory.
29. ↑ Metz, Cade (9 June 2018). "Mark Zuckerberg, Elon Musk and the Feud Over Killer Robots". The New York Times. https://www.nytimes.com/2018/06/09/technology/elon-musk-mark-zuckerberg-artificial-intelligence.html.
30. ↑ Hsu, Jeremy (1 March 2012). "Control dangerous AI before it controls us, one expert says". NBC News. http://www.nbcnews.com/id/46590591/ns/technology_and_science-innovation.
31. ↑ 31.0 31.1 31.2 "Stephen Hawking: 'Transcendence looks at the implications of artificial intelligence – but are we taking AI seriously enough?'". The Independent (UK). https://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html.
32. ↑ "Stephen Hawking warns artificial intelligence could end mankind". BBC. 2 December 2014. https://www.bbc.com/news/technology-30290540.
33. ↑ Eadicicco, Lisa (28 January 2015). "Bill Gates: Elon Musk Is Right, We Should All Be Scared Of Artificial Intelligence Wiping Out Humanity". Business Insider. http://www.businessinsider.com/bill-gates-artificial-intelligence-2015-1.
34. ↑ "Research Priorities for Robust and Beneficial Artificial Intelligence: an Open Letter". Future of Life Institute. http://futureoflife.org/misc/open_letter.
35. ↑ "Anticipating artificial intelligence" (in en). Nature 532 (7600): 413. 2016. doi:10.1038/532413a. ISSN 1476-4687. PMID 27121801. Bibcode: 2016Natur.532Q.413..
36. ↑ Christian, Brian (October 6, 2020). The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company. ISBN 978-0-393-63582-9. https://brianchristian.org/the-alignment-problem/. Retrieved 5 December 2021.
37. ↑ Dignum, Virginia (26 May 2021). "AI — the people and places that make, use and manage it" (in en). Nature 593 (7860): 499–500. doi:10.1038/d41586-021-01397-x. Bibcode: 2021Natur.593..499D.
38. ↑ "Elon Musk among experts urging a halt to AI training" (in en-GB). BBC News. 2023-03-29. https://www.bbc.com/news/technology-65110030.
39. ↑ "Statement on AI Risk". https://www.safe.ai/statement-on-ai-risk#open-letter.
40. ↑ "Artificial intelligence could lead to extinction, experts warn" (in en-GB). BBC News. 2023-05-30. https://www.bbc.com/news/uk-65746524.
41. ↑ "DeepMind and Google: the battle to control artificial intelligence". The Economist. ISSN 0013-0613. https://www.economist.com/1843/2019/03/01/deepmind-and-google-the-battle-to-control-artificial-intelligence.
42. ↑ "AI timelines: What do experts in artificial intelligence expect for the future?". https://ourworldindata.org/ai-timelines.
43. ↑ De Vynck, Gerrit (20 May 2023). "The debate over whether AI will destroy us is dividing Silicon Valley". The Washington Post. https://www.washingtonpost.com/technology/2023/05/20/ai-existential-risk-debate/.
44. ↑ "'The Godfather of A.I.' just quit Google and says he regrets his life's work because it can be hard to stop 'bad actors from using it for bad things'" (in en). https://fortune.com/2023/05/01/godfather-ai-geoffrey-hinton-quit-google-regrets-lifes-work-bad-actors/.
45. ↑ "Everything you need to know about superintelligence" (in en-US). https://www.spiceworks.com/tech/artificial-intelligence/articles/everything-about-superintelligence/.
46. ↑ 46.0 46.1 46.2 "Clever cogs". The Economist. 9 August 2014. https://www.economist.com/news/books-and-arts/21611037-potential-impacts-intelligent-machines-human-life-clever-cogs. Syndicated at Business Insider
47. ↑ 47.0 47.1 Bostrom, Nick (2015-04-27), What happens when our computers get smarter than we are?, https://www.ted.com/talks/nick_bostrom_what_happens_when_our_computers_get_smarter_than_we_are, retrieved 2023-07-13
48. ↑ "Governance of superintelligence" (in en-US). https://openai.com/blog/governance-of-superintelligence.
49. ↑ "Overcoming Bias: I Still Don't Get Foom". http://www.overcomingbias.com/2014/07/30855.html.
50. ↑ Cotton-Barratt, Owen; Ord, Toby (2014-08-12). "Strategic considerations about different speeds of AI takeoff" (in en-GB). https://www.fhi.ox.ac.uk/strategic-considerations-about-different-speeds-of-ai-takeoff/.
51. ↑ Tegmark, Max (2023-04-25). "The 'Don't Look Up' Thinking That Could Doom Us With AI" (in en). Time. https://time.com/6273743/thinking-that-could-doom-us-with-ai/. Retrieved 2023-07-14. "As if losing control to Chinese minds were scarier than losing control to alien digital minds that don't care about humans. [...] it's clear by now that the space of possible alien minds is vastly larger than that.".
52. ↑ "19 - Mechanistic Interpretability with Neel Nanda" (in en). 2023-02-04. https://axrp.net/episode/2023/02/04/episode-19-mechanistic-interpretability-neel-nanda.html. "it's plausible to me that the main thing we need to get done is noticing specific circuits to do with deception and specific dangerous capabilities like that and situational awareness and internally-represented goals."
53. ↑ "Superintelligence Is Not Omniscience" (in en-US). 2023-04-07. https://aiimpacts.org/superintelligence-is-not-omniscience/.
54. ↑ 54.0 54.1 54.2 54.3 54.4 54.5 54.6 54.7 54.8 Dan Hendrycks; Mantas Mazeika; Thomas Woodside (21 June 2023). "An Overview of Catastrophic AI Risks". arXiv:2306.12001 [cs.CY].
55. ↑ Taylor, Josh; Hern, Alex (2023-05-02). "'Godfather of AI' Geoffrey Hinton quits Google and warns over dangers of misinformation" (in en-GB). The Guardian. ISSN 0261-3077. https://www.theguardian.com/technology/2023/may/02/geoffrey-hinton-godfather-of-ai-quits-google-warns-dangers-of-machine-learning.
56. ↑ "How NATO is preparing for a new era of AI cyber attacks" (in en). 2022-12-26. https://www.euronews.com/next/2022/12/26/ai-cyber-attacks-are-a-critical-threat-this-is-how-nato-is-countering-them.
57. ↑ "ChatGPT and the new AI are wreaking havoc on cybersecurity in exciting and frightening ways" (in en). https://www.zdnet.com/article/chatgpt-and-the-new-ai-are-wreaking-havoc-on-cybersecurity/.
58. ↑ Toby Shevlane; Sebastian Farquhar; Ben Garfinkel; Mary Phuong; Jess Whittlestone; Jade Leung; Daniel Kokotajlo; Nahema Marchal; Markus Anderljung; Noam Kolt; Lewis Ho; Divya Siddarth; Shahar Avin; Will Hawkins; Been Kim; Iason Gabriel; Vijay Bolina; Jack Clark; Yoshua Bengio; Paul Christiano; Allan Dafoe (24 May 2023). "Model evaluation for extreme risks". arXiv:2305.15324 [cs.AI].
59. ↑ Urbina, Fabio; Lentzos, Filippa; Invernizzi, Cédric; Ekins, Sean (7 March 2022). "Dual use of artificial-intelligence-powered drug discovery" (in en). Nature Machine Intelligence 4 (3): 189–191. doi:10.1038/s42256-022-00465-9. ISSN 2522-5839. PMID 36211133.
60. ↑ Walter, Yoshija (27 March 2023). "The rapid competitive economy of machine learning development: a discussion on the social risks and benefits". AI and Ethics: 1. doi:10.1007/s43681-023-00276-7.
61. ↑ "The AI Arms Race Is On. Start Worrying" (in en). Time. 2023-02-16. https://time.com/6255952/ai-impact-chatgpt-microsoft-google/. Retrieved 2023-07-17.
62. ↑ Brimelow, Ben. "The short film 'Slaughterbots' depicts a dystopian future of killer drones swarming the world" (in en-US). https://www.businessinsider.com/slaughterbots-short-film-depicts-killer-drone-swarms-2017-11.
63. ↑ Mecklin, John (2023-07-17). "'Artificial Escalation': Imagining the future of nuclear risk" (in en-US). https://thebulletin.org/2023/07/artificial-escalation-imagining-the-future-of-nuclear-risk/.
64. ↑ Doherty, Ben (2018-05-17). "Climate change an 'existential security risk' to Australia, Senate inquiry says" (in en-GB). The Guardian. ISSN 0261-3077. https://www.theguardian.com/environment/2018/may/18/climate-change-an-existential-security-risk-to-australia-senate-inquiry-says.
65. ↑ MacAskill, William (2022). What we owe the future. New York, NY: Basic Books. ISBN 978-1-5416-1862-6.
66. ↑ 66.0 66.1 66.2 66.3 Ord, Toby (2020). "Chapter 5: Future Risks, Unaligned Artificial Intelligence". The Precipice: Existential Risk and the Future of Humanity. Bloomsbury Publishing. ISBN 978-1-5266-0021-9.
67. ↑ Samuelsson, Paul Conrad (June–July 2019). "Artificial Consciousness: Our Greatest Ethical Challenge". Philosophy Now (132). https://philosophynow.org/issues/132/Artificial_Consciousness_Our_Greatest_Ethical_Challenge. Retrieved 2023-08-19.
68. ↑ Brian, Kateman (2023-07-24). "AI Should Be Terrified of Humans" (in en). Time. https://time.com/6296234/ai-should-be-terrified-of-humans/. Retrieved 2023-08-19.
69. ↑ Fisher, Richard. "The intelligent monster that you should let eat you" (in en). https://www.bbc.com/future/article/20201111-philosophy-of-utility-monsters-and-artificial-intelligence.
70. ↑ More, Max (2023-06-19). "Existential Risk vs. Existential Opportunity: A balanced approach to AI risk". https://maxmore.substack.com/p/existential-risk-vs-existential-opportunity.
71. ↑ Omohundro, S. M. (2008, February). The basic AI drives. In AGI (Vol. 171, pp. 483-492).
72. ↑ Carlsmith, Joseph (2022-06-16). "Is Power-Seeking AI an Existential Risk?". arXiv:2206.13353 [cs.CY].
73. ↑ "'The Godfather of A.I.' warns of 'nightmare scenario' where artificial intelligence begins to seek power" (in en). https://fortune.com/2023/05/02/godfather-ai-geoff-hinton-google-warns-artificial-intelligence-nightmare-scenario/.
74. ↑ Wakefield, Jane (15 September 2015). "Why is Facebook investing in AI?". BBC News. https://www.bbc.com/news/technology-34118481.
75. ↑ Yudkowsky, Eliezer (2011). "Complex Value Systems are Required to Realize Valuable Futures". https://intelligence.org/files/ComplexValues.pdf.
76. ↑ 76.0 76.1 Ord, Toby (2020). The Precipice: Existential Risk and the Future of Humanity. Bloomsbury Publishing Plc. ISBN 978-1-5266-0019-6.
77. ↑ Yudkowsky, E. (2011, August). Complex value systems in friendly AI. In International Conference on Artificial General Intelligence (pp. 388-393). Springer, Berlin, Heidelberg.
78. ↑ Russell, Stuart (2014). "Of Myths and Moonshine". http://edge.org/conversation/the-myth-of-ai#26015.
79. ↑ Dietterich, Thomas; Horvitz, Eric (2015). "Rise of Concerns about AI: Reflections and Directions". Communications of the ACM 58 (10): 38–40. doi:10.1145/2770869. http://research.microsoft.com/en-us/um/people/horvitz/CACM_Oct_2015-VP.pdf. Retrieved 23 October 2015.
80. ↑ 80.0 80.1 Yudkowsky, Eliezer (2023-03-29). "The Open Letter on AI Doesn't Go Far Enough" (in en). Time. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/. Retrieved 2023-07-16.
81. ↑ Bostrom, Nick (2012-05-01). "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents" (in en). Minds and Machines 22 (2): 71–85. doi:10.1007/s11023-012-9281-3. ISSN 1572-8641. "as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so.".
82. ↑ Richard Ngo; Lawrence Chan; Sören Mindermann (22 February 2023). "The alignment problem from a deep learning perspective". arXiv:2209.00626 [cs.AI].
83. ↑ "Introducing Superalignment" (in en-US). https://openai.com/blog/introducing-superalignment.
84. ↑ Tilli, Cecilia (28 April 2016). "Killer Robots? Lost Jobs?" (in en-US). Slate. http://www.slate.com/articles/technology/future_tense/2016/04/the_threats_that_artificial_intelligence_researchers_actually_worry_about.html.
85. ↑ "Norvig vs. Chomsky and the Fight for the Future of AI". 21 June 2011. http://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fight-for-the-future-of-ai/.
86. ↑ Graves, Matthew (8 November 2017). "Why We Should Be Concerned About Artificial Superintelligence". Skeptic (US magazine) 22 (2). https://www.skeptic.com/reading_room/why-we-should-be-concerned-about-artificial-superintelligence/.
87. ↑ Johnson, Phil (30 July 2015). "Houston, we have a bug: 9 famous software glitches in space" (in en). IT World. https://www.itworld.com/article/2823083/enterprise-software/88716-8-famous-software-bugs-in-space.html.
88. ↑ Yampolskiy, Roman V. (8 April 2014). "Utility function security in artificially intelligent agents". Journal of Experimental & Theoretical Artificial Intelligence 26 (3): 373–389. doi:10.1080/0952813X.2014.895114. "Nothing precludes sufficiently smart self-improving systems from optimising their reward mechanisms in order to optimisetheir current-goal achievement and in the process making a mistake leading to corruption of their reward functions.".
89. ↑ "Will artificial intelligence destroy humanity? Here are 5 reasons not to worry.". 2014-08-22. https://www.vox.com/2014/8/22/6043635/5-reasons-we-shouldnt-worry-about-super-intelligent-computers-taking.
90. ↑ Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies. Oxford, United Kingdom: Oxford University Press. p. 116. ISBN 978-0-19-967811-2.
91. ↑ Bostrom, Nick (2012). "Superintelligent Will". http://www.nickbostrom.com/superintelligentwill.pdf.
92. ↑ Armstrong, Stuart (January 1, 2013). "General Purpose Intelligence: Arguing the Orthogonality Thesis". Analysis and Metaphysics 12. https://www.questia.com/library/journal/1P3-3195465391/general-purpose-intelligence-arguing-the-orthogonality. Retrieved April 2, 2020. Full text available here .
93. ↑ 93.0 93.1 Chorost, Michael (18 April 2016). "Let Artificial Intelligence Evolve". Slate. http://www.slate.com/articles/technology/future_tense/2016/04/the_philosophical_argument_against_artificial_intelligence_killing_us_all.html. Retrieved 27 November 2017.
94. ↑ "Should humans fear the rise of the machine?". The Telegraph (UK). 1 Sep 2015. https://www.telegraph.co.uk/technology/news/11837157/Should-humans-fear-the-rise-of-the-machine.html.
95. ↑ 95.0 95.1 Shermer, Michael (1 March 2017). "Apocalypse AI" (in en). Scientific American 316 (3): 77. doi:10.1038/scientificamerican0317-77. PMID 28207698. Bibcode: 2017SciAm.316c..77S. https://www.scientificamerican.com/article/artificial-intelligence-is-not-a-threat-mdash-yet/. Retrieved 27 November 2017.
96. ↑ "Intelligent Machines: What does Facebook want with AI?". BBC News. 14 September 2015. https://www.bbc.com/news/technology-34118481.
97. ↑ Baum, Seth (2018-09-30). "Countering Superintelligence Misinformation". Information 9 (10): 244. doi:10.3390/info9100244. ISSN 2078-2489.
98. ↑ "The Myth Of AI". https://www.edge.org/conversation/jaron_lanier-the-myth-of-ai.
99. ↑ Bostrom, Nick, Superintelligence: paths, dangers, strategies, ISBN 978-1-5012-2774-5, OCLC 1061147095
100. ↑ Sotala, Kaj; Yampolskiy, Roman V (2014-12-19). "Responses to catastrophic AGI risk: a survey". Physica Scripta 90 (1): 12. doi:10.1088/0031-8949/90/1/018001. ISSN 0031-8949. Bibcode: 2015PhyS...90a8001S.
101. ↑ Pistono, Federico; Yampolskiy, Roman V. (2016-05-09). Unethical Research: How to Create a Malevolent Artificial Intelligence. OCLC 1106238048.
102. ↑ Haney, Brian Seamus (2018). "The Perils & Promises of Artificial General Intelligence". SSRN Working Paper Series. doi:10.2139/ssrn.3261254. ISSN 1556-5068.
103. ↑ Russell, Stuart (30 August 2017). "Artificial intelligence: The future is superintelligent" (in En). Nature 548 (7669): 520–521. doi:10.1038/548520a. Bibcode: 2017Natur.548..520R.
104. ↑ 104.0 104.1 104.2 Max Tegmark (2017). Life 3.0: Being Human in the Age of Artificial Intelligence (1st ed.). Mainstreaming AI Safety: Knopf. ISBN 978-0-451-48507-6.
105. ↑ Kumar, Vibhore. "Council Post: At The Dawn Of Artificial General Intelligence: Balancing Abundance With Existential Safeguards" (in en). https://www.forbes.com/sites/forbestechcouncil/2023/04/24/at-the-dawn-of-artificial-general-intelligence-balancing-abundance-with-existential-safeguards/.
106. ↑ 106.0 106.1 "Pause Giant AI Experiments: An Open Letter". https://futureoflife.org/open-letter/pause-giant-ai-experiments/.
107. ↑ "AI Principles". 11 August 2017. https://futureoflife.org/ai-principles/.
108. ↑ "Elon Musk and Stephen Hawking warn of artificial intelligence arms race" (in en). Newsweek. 31 January 2017. http://www.newsweek.com/ai-asilomar-principles-artificial-intelligence-elon-musk-550525.
109. ↑ Martin Ford (2015). "Chapter 9: Super-intelligence and the Singularity". Rise of the Robots: Technology and the Threat of a Jobless Future. Basic Books. ISBN 978-0-465-05999-7.
110. ↑ Bostrom, Nick (2016). "New Epilogue to the Paperback Edition". Superintelligence: Paths, Dangers, Strategies (Paperback ed.).
111. ↑ "Why Uncontrollable AI Looks More Likely Than Ever" (in en). Time. 27 February 2023. https://time.com/6258483/uncontrollable-ai-agi-risks/. Retrieved 30 March 2023. "It is therefore no surprise that according to the most recent AI Impacts Survey, nearly half of 731 leading AI researchers think there is at least a 10% chance that human-level AI would lead to an "extremely negative outcome," or existential risk.".
112. ↑ 112.0 112.1 Maas, Matthijs M. (2019-02-06). "How viable is international arms control for military artificial intelligence? Three lessons from nuclear weapons of mass destruction". Contemporary Security Policy 40 (3): 285–311. doi:10.1080/13523260.2019.1576464. ISSN 1352-3260.
113. ↑ 113.0 113.1 "Impressed by artificial intelligence? Experts say AGI is coming next, and it has 'existential' risks" (in en-AU). ABC News. 23 March 2023. https://www.abc.net.au/news/2023-03-24/what-is-agi-artificial-general-intelligence-ai-experts-risks/102035132.
114. ↑ Rawlinson, Kevin (29 January 2015). "Microsoft's Bill Gates insists AI is a threat". BBC News. https://www.bbc.co.uk/news/31047780.
115. ↑ Post, Washington (14 December 2015). "Tech titans like Elon Musk are spending $1 billion to save you from terminators". https://www.chicagotribune.com/bluesky/technology/ct-tech-titans-against-terminators-20151214-story.html.
116. ↑ "Doomsday to utopia: Meet AI's rival factions" (in en). Washington Post. 2023-04-09. https://www.washingtonpost.com/technology/2023/04/09/ai-safety-openai/.
117. ↑ "UC Berkeley — Center for Human-Compatible AI (2016) - Open Philanthropy" (in en-us). 2016-06-27. https://www.openphilanthropy.org/grants/uc-berkeley-center-for-human-compatible-ai-2016/.
118. ↑ "The mysterious artificial intelligence company Elon Musk invested in is developing game-changing smart computers". http://www.techinsider.io/mysterious-artificial-intelligence-company-elon-musk-investment-2015-10.
119. ↑ Clark 2015a.
120. ↑ "Elon Musk Is Donating $10M Of His Own Money To Artificial Intelligence Research". 2015-01-15. http://www.fastcompany.com/3041007/fast-feed/elon-musk-is-donating-10m-of-his-own-money-to-artificial-intelligence-research.
121. ↑ Tilli, Cecilia (28 April 2016). "Killer Robots? Lost Jobs?" (in en-US). Slate. http://www.slate.com/articles/technology/future_tense/2016/04/the_threats_that_artificial_intelligence_researchers_actually_worry_about.html.
122. ↑ Khatchadourian, Raffi (23 November 2015). "The Doomsday Invention: Will artificial intelligence bring us utopia or destruction?". The New Yorker. https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom.
123. ↑ "Warning of AI's danger, pioneer Geoffrey Hinton quits Google to speak freely". www.arstechnica.com. 2023. https://arstechnica.com/information-technology/2023/05/warning-of-ais-danger-pioneer-geoffrey-hinton-quits-google-to-speak-freely/.
124. ↑ 124.0 124.1 "But What Would the End of Humanity Mean for Me?". The Atlantic. 9 May 2014. https://www.theatlantic.com/health/archive/2014/05/but-what-does-the-end-of-humanity-mean-for-me/361931/. Retrieved 12 December 2015.
125. ↑ Andersen, Kurt (26 November 2014). "Enthusiasts and Skeptics Debate Artificial Intelligence" (in en). Vanity Fair. https://www.vanityfair.com/news/tech/2014/11/artificial-intelligence-singularity-theory. Retrieved 2020-04-20.
126. ↑ Brooks, Rodney (10 November 2014). "artificial intelligence is a tool, not a threat". http://www.rethinkrobotics.com/artificial-intelligence-tool-threat/.
127. ↑ Garling, Caleb (5 May 2015). "Andrew Ng: Why 'Deep Learning' Is a Mandate for Humans, Not Just Machines". Wired. https://www.wired.com/brandlab/2015/05/andrew-ng-deep-learning-mandate-humans-not-just-machines/. Retrieved 31 March 2023.
128. ↑ "Is artificial intelligence really an existential threat to humanity?". 2023-04-04. https://mambapost.com/2023/04/tech-news/ai-are-an-existential-threat-to-humanity/.
129. ↑ "The case against killer robots, from a guy actually working on artificial intelligence". http://fusion.net/story/54583/the-case-against-killer-robots-from-a-guy-actually-building-ai/.
130. ↑ "AI experts challenge 'doomer' narrative, including 'extinction risk' claims" (in en-US). 2023-05-31. https://venturebeat.com/ai/ai-experts-challenge-doomer-narrative-including-extinction-risk-claims/.
131. ↑ Coldewey, Devin (2023-04-01). "Ethicists fire back at 'AI Pause' letter they say 'ignores the actual harms'" (in en-US). https://techcrunch.com/2023/03/31/ethicists-fire-back-at-ai-pause-letter-they-say-ignores-the-actual-harms/.
132. ↑ "DAIR (Distributed AI Research Institute)" (in en). https://dair-institute.org/.
133. ↑ Kelly, Kevin (April 25, 2017). "The Myth of a Superhuman AI". Wired. https://www.wired.com/2017/04/the-myth-of-a-superhuman-ai/. Retrieved February 19, 2022.
134. ↑ Jindal, Siddharth (2023-07-07). "OpenAI's Pursuit of AI Alignment is Farfetched" (in en-US). https://analyticsindiamag.com/openais-farfetched-pursuit-of-ai-alignment/.
135. ↑ "Mark Zuckerberg responds to Elon Musk's paranoia about AI: 'AI is going to... help keep our communities safe.'". 25 May 2018. https://www.businessinsider.com/mark-zuckerberg-shares-thoughts-elon-musks-ai-2018-5.
136. ↑ Elliott, E. W. (2011). "Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100, by Michio Kaku". Issues in Science and Technology 27 (4): 90.
137. ↑ Kaku, Michio (2011). Physics of the future: how science will shape human destiny and our daily lives by the year 2100. New York: Doubleday. ISBN 978-0-385-53080-4. "I personally believe that the most likely path is that we will build robots to be benevolent and friendly"
138. ↑ Dadich, Scott. "Barack Obama Talks AI, Robo Cars, and the Future of the World". WIRED. https://www.wired.com/2016/10/president-obama-mit-joi-ito-interview/. Retrieved 27 November 2017.
139. ↑ Kircher, Madison Malone. "Obama on the Risks of AI: 'You Just Gotta Have Somebody Close to the Power Cord'" (in en). Select All. https://nymag.com/selectall/2016/10/barack-obama-talks-artificial-intelligence-in-wired.html.
140. ↑ Clinton, Hillary (2017). What Happened. Simon and Schuster. p. 241. ISBN 978-1-5011-7556-5. via [1]
141. ↑ LIPPENS, RONNIE (2002). "Imachinations of Peace: Scientifictions of Peace in Iain M. Banks's The Player of Games" (in en). Utopianstudies Utopian Studies 13 (1): 135–147. ISSN 1045-991X. OCLC 5542757341.
142. ↑ "Elon Musk says AI could doom human civilization. Zuckerberg disagrees. Who's right?". 5 January 2023. https://www.usatoday.com/story/tech/news/2018/01/02/artificial-intelligence-end-world-overblown-fears/985813001/.
143. ↑ "AI doomsday worries many Americans. So does apocalypse from climate change, nukes, war, and more". 14 April 2023. https://today.yougov.com/topics/technology/articles-reports/2023/04/14/ai-nuclear-weapons-world-war-humanity-poll.
144. ↑ Tyson, Alec; Kikuchi, Emma. "Growing public concern about the role of artificial intelligence in daily life" (in en-US). https://www.pewresearch.org/short-reads/2023/08/28/growing-public-concern-about-the-role-of-artificial-intelligence-in-daily-life/.
145. ↑ Kaj Sotala; Roman Yampolskiy (19 December 2014). "Responses to catastrophic AGI risk: a survey". Physica Scripta 90 (1).
146. ↑ Barrett, Anthony M.; Baum, Seth D. (2016-05-23). "A model of pathways to artificial superintelligence catastrophe for risk and decision analysis". Journal of Experimental & Theoretical Artificial Intelligence 29 (2): 397–414. doi:10.1080/0952813x.2016.1186228. ISSN 0952-813X.
147. ↑ Sotala, Kaj; Yampolskiy, Roman V (2014-12-19). "Responses to catastrophic AGI risk: a survey". Physica Scripta 90 (1): 018001. doi:10.1088/0031-8949/90/1/018001. ISSN 0031-8949. Bibcode: 2015PhyS...90a8001S.
148. ↑ Ramamoorthy, Anand; Yampolskiy, Roman (2018). "Beyond MAD? The race for artificial general intelligence". ICT Discoveries (ITU) 1 (Special Issue 1): 1–8. https://www.itu.int/pub/S-JOURNAL-ICTS.V1I1-2018-9. Retrieved 7 January 2022.
149. ↑ Carayannis, Elias G.; Draper, John (2022-01-11). "Optimising peace through a Universal Global Peace Treaty to constrain the risk of war from a militarised artificial superintelligence" (in en). AI & Society: 1–14. doi:10.1007/s00146-021-01382-y. ISSN 0951-5666. PMID 35035113.
150. ↑ Carayannis, Elias G.; Draper, John (2023-05-30), "The challenge of advanced cyberwar and the place of cyberpeace", The Elgar Companion to Digital Transformation, Artificial Intelligence and Innovation in the Economy, Society and Democracy (Edward Elgar Publishing): pp. 32–80, doi:10.4337/9781839109362.00008, ISBN 978-1-83910-936-2, https://www.elgaronline.com/edcollchap/book/9781839109362/book-part-9781839109362-8.xml, retrieved 2023-06-08
151. ↑ Vincent, James (22 June 2016). "Google's AI researchers say these are the five key problems for robot safety" (in en). The Verge. https://www.theverge.com/circuitbreaker/2016/6/22/11999664/google-robots-ai-safety-five-problems.
152. ↑ Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. "Concrete problems in AI safety." arXiv preprint arXiv:1606.06565 (2016).
153. ↑ Johnson, Alex (2019). "Elon Musk wants to hook your brain up directly to computers — starting next year" (in en). NBC News. https://www.nbcnews.com/mach/tech/elon-musk-wants-hook-your-brain-directly-computers-starting-next-ncna1030631.
154. ↑ Torres, Phil (18 September 2018). "Only Radically Enhancing Humanity Can Save Us All" (in en). Slate Magazine. https://slate.com/technology/2018/09/genetic-engineering-to-stop-doomsday.html.
155. ↑ Barrett, Anthony M.; Baum, Seth D. (23 May 2016). "A model of pathways to artificial superintelligence catastrophe for risk and decision analysis". Journal of Experimental & Theoretical Artificial Intelligence 29 (2): 397–414. doi:10.1080/0952813X.2016.1186228.
156. ↑ Piper, Kelsey (2023-03-29). "How to test what an AI model can — and shouldn't — do" (in en). https://www.vox.com/future-perfect/2023/3/29/23661633/gpt-4-openai-alignment-research-center-open-philanthropy-ai-safety.
157. ↑ Piesing, Mark (17 May 2012). "AI uprising: humans will be outsourced, not obliterated". Wired. https://www.wired.co.uk/news/archive/2012-05/17/the-dangers-of-an-ai-smarter-than-us. Retrieved 12 December 2015.
158. ↑ Coughlan, Sean (24 April 2013). "How are humans going to become extinct?". BBC News. https://www.bbc.com/news/business-22002530.
159. ↑ Bridge, Mark (10 June 2017). "Making robots less confident could prevent them taking over" (in en). The Times. https://www.thetimes.co.uk/article/making-robots-less-confident-could-prevent-them-taking-over-gnsblq7lx.
160. ↑ McGinnis, John (Summer 2010). "Accelerating AI". Northwestern University Law Review 104 (3): 1253–1270. http://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1193&context=nulr_online. Retrieved 16 July 2014. "For all these reasons, verifying a global relinquishment treaty, or even one limited to AI-related weapons development, is a nonstarter... (For different reasons from ours, the Machine Intelligence Research Institute) considers (AGI) relinquishment infeasible...".
161. ↑ Kaj Sotala; Roman Yampolskiy (19 December 2014). "Responses to catastrophic AGI risk: a survey". Physica Scripta 90 (1). "In general, most writers reject proposals for broad relinquishment... Relinquishment proposals suffer from many of the same problems as regulation proposals, but to a greater extent. There is no historical precedent of general, multi-use technology similar to AGI being successfully relinquished for good, nor do there seem to be any theoretical reasons for believing that relinquishment proposals would work in the future. Therefore we do not consider them to be a viable class of proposals.".
162. ↑ Allenby, Brad (11 April 2016). "The Wrong Cognitive Measuring Stick" (in en-US). Slate. http://www.slate.com/articles/technology/future_tense/2016/04/why_it_s_a_mistake_to_compare_a_i_with_human_intelligence.html. "It is fantasy to suggest that the accelerating development and deployment of technologies that taken together are considered to be A.I. will be stopped or limited, either by regulation or even by national legislation."
163. ↑ 163.0 163.1 Yampolskiy, Roman V. (2022). "AI Risk Skepticism". in Müller, Vincent C. (in en). Philosophy and Theory of Artificial Intelligence 2021. Studies in Applied Philosophy, Epistemology and Rational Ethics. 63. Cham: Springer International Publishing. pp. 225–248. doi:10.1007/978-3-031-09153-7_18. ISBN 978-3-031-09153-7. https://link.springer.com/chapter/10.1007/978-3-031-09153-7_18.
164. ↑ McGinnis, John (Summer 2010). "Accelerating AI". Northwestern University Law Review 104 (3): 1253–1270. http://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1193&context=nulr_online. Retrieved 16 July 2014.
165. ↑ "Why We Should Think About the Threat of Artificial Intelligence". The New Yorker. 4 October 2013. https://www.newyorker.com/tech/elements/why-we-should-think-about-the-threat-of-artificial-intelligence. Retrieved 7 February 2016. "Of course, one could try to ban super-intelligent computers altogether. But 'the competitive advantage—economic, military, even artistic—of every advance in automation is so compelling,' Vernor Vinge, the mathematician and science-fiction author, wrote, 'that passing laws, or having customs, that forbid such things merely assures that someone else will.'".
166. ↑ Baum, Seth (2018-08-22). "Superintelligence Skepticism as a Political Tool". Information 9 (9): 209. doi:10.3390/info9090209. ISSN 2078-2489.
167. ↑ "Elon Musk and other tech leaders call for pause in 'out of control' AI race" (in en). CNN. 29 March 2023. https://www.cnn.com/2023/03/29/tech/ai-letter-elon-musk-tech-leaders/index.html.
168. ↑ "Open letter calling for AI 'pause' shines light on fierce debate around risks vs. hype" (in en-US). 2023-03-29. https://venturebeat.com/ai/open-letter-calling-for-ai-pause-shines-light-on-fierce-debate-around-risks-vs-hype/.
169. ↑ Vincent, James (2023-04-14). "OpenAI's CEO confirms the company isn't training GPT-5 and "won't for some time"" (in en-US). https://www.theverge.com/2023/4/14/23683084/openai-gpt-5-rumors-training-sam-altman.
170. ↑ "The Open Letter on AI Doesn't Go Far Enough" (in en). Time. 2023-03-29. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/. Retrieved 2023-07-20.
171. ↑ Domonoske, Camila (July 17, 2017). "Elon Musk Warns Governors: Artificial Intelligence Poses 'Existential Risk'" (in en). NPR. https://www.npr.org/sections/thetwo-way/2017/07/17/537686649/elon-musk-warns-governors-artificial-intelligence-poses-existential-risk.
172. ↑ Gibbs, Samuel (17 July 2017). "Elon Musk: regulate AI to combat 'existential threat' before it's too late". The Guardian. https://www.theguardian.com/technology/2017/jul/17/elon-musk-regulation-ai-combat-existential-threat-tesla-spacex-ceo.
173. ↑ Kharpal, Arjun (7 November 2017). "A.I. is in its 'infancy' and it's too early to regulate it, Intel CEO Brian Krzanich says". CNBC. https://www.cnbc.com/2017/11/07/ai-infancy-and-too-early-to-regulate-intel-ceo-brian-krzanich-says.html.
174. ↑ Dawes, James (2021-12-20). "UN fails to agree on 'killer robot' ban as nations pour billions into autonomous weapons research" (in en). http://theconversation.com/un-fails-to-agree-on-killer-robot-ban-as-nations-pour-billions-into-autonomous-weapons-research-173616.
175. ↑ 175.0 175.1 Fassihi, Farnaz (2023-07-18). "U.N. Officials Urge Regulation of Artificial Intelligence" (in en-US). The New York Times. ISSN 0362-4331. https://www.nytimes.com/2023/07/18/world/un-security-council-ai.html.
176. ↑ "International Community Must Urgently Confront New Reality of Generative, Artificial Intelligence, Speakers Stress as Security Council Debates Risks, Rewards". https://press.un.org/en/2023/sc15359.doc.htm.
177. ↑ Sotala, Kaj; Yampolskiy, Roman V (2014-12-19). "Responses to catastrophic AGI risk: a survey". Physica Scripta 90 (1): 018001. doi:10.1088/0031-8949/90/1/018001. ISSN 0031-8949. Bibcode: 2015PhyS...90a8001S.
178. ↑ Geist, Edward Moore (2016-08-15). "It's already too late to stop the AI arms race—We must manage it instead". Bulletin of the Atomic Scientists 72 (5): 318–321. doi:10.1080/00963402.2016.1216672. ISSN 0096-3402. Bibcode: 2016BuAtS..72e.318G.
179. ↑ "Amazon, Google, Meta, Microsoft and other tech firms agree to AI safeguards set by the White House" (in en). 2023-07-21. https://apnews.com/article/artificial-intelligence-safeguards-joe-biden-kamala-harris-4caf02b94275429f764b06840897436c.
180. ↑ "Amazon, Google, Meta, Microsoft and other firms agree to AI safeguards" (in en). 2023-07-21. https://www.redditchadvertiser.co.uk/news/national/23670894.amazon-google-meta-microsoft-firms-agree-ai-safeguards/.
181. ↑ House, The White (2023-10-30). "Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence" (in en-US). https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/.

## Bibliography

* Clark, Jack (2015a). "Musk-Backed Group Probes Risks Behind Artificial Intelligence". https://www.bloomberg.com/news/articles/2015-07-01/musk-backed-group-probes-risks-behind-artificial-intelligence.

* v
* t
* e

Existential risk from artificial intelligence

Concepts|

* AI box
* AI takeover
* Control problem
* Existential risk from artificial general intelligence
* Friendly artificial intelligence
* Instrumental convergence
* Intelligence explosion
* Machine ethics
* Superintelligence
* Technological singularity

Organizations|

* Allen Institute for Artificial Intelligence
* Center for Applied Rationality
* Center for Security and Emerging Technology
* Centre for the Study of Existential Risk
* DeepMind
* Foundational Questions Institute
* Future of Humanity Institute
* Future of Life Institute
* Humanity+
* Institute for Ethics and Emerging Technologies
* Leverhulme Centre for the Future of Intelligence
* Machine Intelligence Research Institute
* OpenAI

People|

* Nick Bostrom
* Sam Harris
* Stephen Hawking
* Bill Hibbard
* Bill Joy
* Elon Musk
* Steve Omohundro
* Huw Price
* Martin Rees
* Stuart J. Russell
* Jaan Tallinn
* Max Tegmark
* Frank Wilczek
* Roman Yampolskiy
* Andrew Yang
* Eliezer Yudkowsky

Other|

* Open Letter on Artificial Intelligence
* Ethics of artificial intelligence
* Controversies and dangers of artificial general intelligence
* Artificial intelligence as a global catastrophic risk
* Superintelligence
* Our Final Invention

Category

* v
* t
* e

Global catastrophic risks

* Future of the Earth
* Ultimate fate of the universe

Technological|

* Gray goo
* Kinetic bombardment
* Mutual assured destruction
* Dead Hand
* Doomsday device
* Antimatter weapon
* Synthetic intelligence / Artificial intelligence
* Existential risk from artificial intelligence
* AI takeover
* Technological singularity
* Transhumanism
* Year 2000 problem

Sociological|

* Malthusian catastrophe
* New World Order (conspiracy theory)
* Nuclear holocaust
* winter
* famine
* cobalt
* Societal collapse
* World War III

Ecological|

| Climate change|

* Extinction risk from global warming
* Tipping points in the climate system
* Global terrestrial stilling
* Global warming
* Hypercane
* Ice age
* Ecocide
* Human impact on the environment
* Ozone depletion
* Cascade effect
* Supervolcano
* winter

|

Earth Overshoot Day|

* Overexploitation
* Overpopulation
* Human overpopulation

Biological|

| Extinction|

* Extinction event
* Human extinction
* Genetic erosion
* Genetic pollution

|

Others|

* Dysgenics
* Pandemic
* Biological agent
* Transhumanism

Astronomical|

* Big Crunch
* Big Rip
* Coronal mass ejection
* Gamma-ray burst
* Impact event
* Asteroid impact avoidance
* Potentially hazardous object
* Near-Earth supernova
* Solar flare
* Stellar collision

Mythological|

| Eschatology|

* Buddhist
* Hindu
* Last Judgment
* Christian
* Book of Revelation
* Islamic
* Jewish
* Norse
* Zoroastrian

|

Others|

* 2011 end times prediction
* 2012 phenomenon
* Apocalypse
* Armageddon
* Blood moon prophecy
* Doomsday Clock
* End time
* List of dates predicted for apocalyptic events
* Nibiru cataclysm
* Rapture
* Revelation 12 sign prophecy
* Third Temple

Fictional|

* Alien invasion
* Apocalyptic and post-apocalyptic fiction
* List of apocalyptic and post-apocalyptic fiction
* List of apocalyptic films
* Climate fiction
* Disaster films
* List of disaster films
* List of fictional doomsday devices
* Zombie apocalypse

* Categories
* Apocalypticism
* Future problems
* Hazards
* Risk analysis
* Doomsday scenarios

0.00

(0 votes)

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Unsolved:Existential_risk_from_artificial_general_intelligence&oldid=3854391"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template