I would not be surprised if, after hard takeoff and calculating the odds, a Friendly AI told us that, given the world's imminent dangers, this was the most (counterintuitively) moral action it could take. Imagine Skynet sprinkling the world with nukes, not because of a self preservation instinct, but because of perfect altruism.
I dunno. I'm not sure why you "would not be surprised". Would you be surprised if a genuinely kind person, the likes of which humanity has not yet seen, spontaneously decided to release nerve gas in a preschool, or something? I certainly would. I would also be quite surprised if that person, given self-modification ability, ended up on an enhancement trajectory that culminated in genocide. If I did happen, I would chalk that up to an technical error due to the enhancement process, or a wirehead error, or a subgoal-stomp error, or something else out of the ordinary - not the sincere will of the enhancee.
This would especially apply if I had known the enhancee beforehand, and they promised me that their intentions were entirely benevolent, and that they would proceed on the enhancement curve as safely and cautiously as possible, employing whatever precautions or safeguards it would take - whether that be slowing the enhancement rate to a crawl, spinning off comrades to help monitor one another, engaging in extra "wisdom tournaments" to model the thresholds of moral breakdown, or even reverting back to a human or humanlike being and saying "sorry, the intelligence space above humanity is too dangerous, all my models suggest that intelligence enhancement past a certain point results in genocidal thoughts."
But let's say that an initially altruistic AI did end up "logically concluding" that everyone should die. This would have further philosophical and moral implications. It would mean that some "cognitive-moral attractor" in the "memespace" of transhumanly intelligent beings possesses an *incredible* force of "drag", enough to entirely catch an altruistic AI or IAee by surprise, rapidly transforming an enthusiastically helpful, altruistic person into a genocidal one before they could get help.
Perhaps they "didn't want to get help" because killing everyone was the "logical conclusion"? If killing everyone really is the logical conclusion, then human altruists have been in "denial" throughout history, a FAI or sufficiently kind uploadee would also be in "denial", and chances are that they would self-modify into a superintelligence in "denial" as well. The alternative is that *all* beings enhanced above a certain level of intelligence go genocidal, and the second that I (or some other altruist, or a FAI) noticed that, I predict that we'd do everything in our power to prevent that from happening.
A central question that would effect this answer is "which is stronger, a transhuman's ability to rewrite and maintain its own source code, or attractors in the morality space that force convergence regardless of the will of the transhuman itself?" (Convergence in this case meaning; in 1000 different parallel universes, recursive self-improvers ranging from the extremely Friendly to the utterly oblivious are launched, and in 999 of these universes that RSIer ended up killing everyone including itself, and in only 1 did it *not* kill everyone, because that's the roll of the dice.) My guess is that "convergence" towards genocide among AIs *does* exist, because most physically possible AIs don't have goal systems complex enough to value sentient life, and an ascent in their power would probably result in the transformation of reachable material (i.e., humans and Earth) into utility-structures that the AI values; BUT, when it comes to reasonably well-programmed FAIs, I would probably rather have them spark the Singularity than me, because I'd probably consider them to be more trustworthy than me. Without rationalization or a self-opaque goal system, Friendly AIs could be transhumanly trustworthy.
"Imagine if a Friendly AI decided to destroy the world one day" is certainly a scary thought, but we can generalize it even further to yield a thought practically equivalent to the first; "what if ANY sufficiently intelligent being decided to destroy the world one day?" These are the kind of questions, of course, that we'd want to pass over to a FAI to let it consider... and if even after considering them, and taking the self-enhancement path cautiously, it STILL ends up killing everyone, then I'd probably just blame that on the fact that we live in an inherently mean universe, that seems to mysteriously force sufficiently intelligent entities to become suicidal.
I'd also like to mention a nitpick about your wording; you say "after a hard takeoff the AI might decide X", but I visualize a FAI continuing to reinforce its altruistic goal system *throughout* the hard takeoff, with transhuman strategies like wisdom tournaments, AI shadowselves; you know, CFAI-type stuff, but radically extended and improved. If the AI leaped first (by enhancing its intelligence), and neglected to *look* (through morality modeling, AI shadowselves, etc.), then I think that would fit a pretty standard class of failure. The spookiest question, that I agree is worth considering at length, is, "even if we get *everything* right, is there a chance we'll still be wiped out?" And I hope that a FAI would consider that question at length as well.
This conflicts with EY's earlier statement that he expects to succeed with a large error margin for safety. Given that EY's optimism and enthusiasm for self-modification are points upon which I would disagree the most, I am glad to hear more honest accounts of our odds for survival.
Ah, slight misunderstanding here. My *guess* of the correct interpretation of the original (mis)quote was "even if nanowar, plague, UFAI, and all the other horrible potential disasters are *"fated" not to happen*, there's *still* a small (~2%) chance that ecological or social collapse is "fated" to happen, in which case FAI could help avoid that". Seems like a pretty silly quote to me, doesn't it to you as well?
So hard to interpret, so not-said-by-Eliezer (he doesn't even like openly stating the probability for *anything*, last time I checked), so odd-attempt-at-a-shock-effect-ish.
Who said Eliezer was optimistic or enthusiastic about self-modification? Didn't you read the grim, pessimistic analyses in CFAI's policy section? *All of CFAI* is tuned towards answering the question, "what architectural features would an AI need to make it so that the potential idiocy or mistakes of the programmers does not result in critical failure?", which again, implies pessimism. Eliezer acknowledges that he is not yet complete even solving the *foundational* problems underlying Friendliness, and I would judge that to mean that he would feel doubtful about his chances of success, if, say, someone held a gun to his head over the next few years, forcing him to build an AI based on the knowledge he currently has.
But in the end, what is Eliezer's *personal* estimation of the real likelihood of success? I have no idea - he probably doesn't have one, because as the "activism vs. futurism" page says, the point is not whether the likelihood of success is 20% or 80%, but whether we can influence that probability by 5% or 10%. I'm pretty sure that he would be equally cautious, worried, and paranoid whether he estimated the likelihood of success to be 99% or 1%, simply beause the stakes are so high. I sure would be, that's for sure, yep yep. [glasses]
Oh yes, and while I have your attention (heh)... do you agree with this following statement made by Eli?
"The most critical theoretical problems in Friendliness are nonobvious, silent, catastrophic, and not inherently fun for humans to argue about; they tend to be structural properties of a computational process rather than anything analogous to human moral disputes."
