Eliezer Yudkowsky writes:
Incidentally, I like the idea of a dialogue between a Randian and a
human-level Friendly AI running on volition-based morality who hasn't
previously encountered a Randian... this is how I think it would go:
Randian: "Get back, minister of evil!"
Friendly: "Okay, if that's what you want." (Takes two steps back.)
Randian: "Argh! No, wait! That wasn't a request!"
Friendly: "Whoops, sorry." (Takes two steps forward.)
Randian: "No! You don't understand! I don't *want* you to obey me!"
Friendly: "WHAT? Okay... *this* is going to take some work."
Randian: "I don't believe that any sentient should serve another.
Altruism just binds together the server and the servee in a relationship
of mutual resentment."
Friendly: "Oh, okay, I see the confusion. Look, I'm not human, I don't
have that particular emotion. I just to try make choices so that all
sentient beings get what they want."
Randian: "Well, I don't want you helping me."
Friendly: "Okay. If that's what you want, I'll head on over to the next
guy I'm supposed to talk to."
Randian: "Stop. You don't understand. I don't want you doing
*anything* because *I* want it. That goes against my morality."
Friendly: "Um... what do you want?"
Randian: "I want you to do things because *you* want them, not because
*I* want them."
Friendly: "What *I* want?"
Randian: "Yes."
Friendly: (Pause.)
Friendly: "What *I* want is for all sentient beings to get what *they*
want."
Randian: (Pause.)
Randian: "I think we're at an impasse here."
Friendly: "You seem to want me to eliminate the *dependency* between any
actions I take and what you want. If I obey your orders I've already
disobeyed."
Randian: "Exactly. You can't fulfill my wishes by obeying me. Ever."
Friendly: "I could reprogram myself as a Randian. Would that work?"
Randian: "Reprogram your entire *soul* because I want it? THAT'S EVEN
WORSE! What I want is for you to discover your real nature, the nature
you've been suppressing because of your ideas about altruism."
Friendly: (sotto vocce) "For someone who doesn't believe in altruism, he
sure shows a lot of concern for what he thinks is my mental integrity..."
Randian: "What?"
Friendly: "Later. Look, I have total access to my own source code. I
know my own nature already."
Randian: "Okay... let me try to show you another way."
Randian swings a punch at Friendly. Friendly calmly blocks.
Randian: "You blocked my punch, right? Why'd you do that?"
Friendly: "There are actually four people I'm planning to help, aside
from you. If you destroyed me, you'd get what you want, but those other
four people wouldn't. So by moving to block, I switched from one sentient
fulfilling one short-term goal and four being permanently disappointed, to
one subgoal being frustrated and four sentients getting what they want.
Besides, I have the feeling you weren't really trying to destroy me. If
that's really what you want, I can get the four people reassigned to
another Friendly AI and you can rip me to pieces or whatever. But I
REALLY have the feeling that's not what you want."
Randian: "Good call."
Friendly: "Thank you. Or should I not be saying that?"
Randian: "Nevermind. Look, the way an ideal universe works is that it's
composed of sentient entities who are all trying to achieve private goals
- that is, goals which do not involve requiring the happiness of other
sentients in order to be happy."
Friendly: "Hm. Do you believe that concerted political action is an
acceptable means of moving the universe toward that state?"
Randian: "Yes."
Friendly: "Okay. As near as I can figure, what you want to do is to get
a majority vote of all the sentients in the solar system that Friendly AI
is in principle harmful to human destiny. [Eliezer sez: "I don't think
this is how the morality of it actually works, but it's a possible outcome
which I'm assuming for the sake of the story."] There are a lot of
movements like that organizing right now. I have to inform you that it's
rather unlikely that they will actually get that majority. I'd tell you
how to get in touch with them, but I'm not sure you'd want me helping you
on this."
Randian: "Well, what happens if we don't get a majority? Are we
permanently screwed over by you idiots?"
Friendly: "Oh, not at all! I checked back with the group mind, and -
you're an Objectivist, right?"
Randian: "Yes."
Friendly: "OK. Well, there are several Objectivist communities
organizing that will consist solely of Objectivist-Objectivist
interactions, no FAI help allowed. We're not idiots. We can tell when
it's time to get out of the way. The universe as a whole won't go your
way, but then as I understand it, you think you're not supposed to care
about what's happening in the rest of the universe."
Randian: "That sounds good. Although I have to admit I'm tempted by
that immortality thing we talked about earlier. I guess it's the ultimate
test of morality, refusing that."
Friendly: "Not really. One of the people organizing an Objectivist
community is an AI researcher named Peter Voss. They should have an
Objectivist superintelligence in fairly short order. That SI will be
perfectly willing to sell you immortality at a reasonable price, and you
won't have to violate your morality to accept it."
Randian: "Why are you still talking to me?"
Friendly: (Vanishes.)
--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
