5 replies to this topic

[Mechanus]

This is a quick summary of a few potential problems, flaws, questions, objections to the idea of Volitional Friendliness. Thinking about it, some questions assume a Sysop or similar type of situation.

Not all of these are guaranteed to make sense. Much of this could be worded in a clearer way, and organization could be much better, so I'm not completely happy with it. I'll post it anyway.

I obviously don't expect anyone to answer all these questions (it's likely that many questions here will have the same answer) - I think that a framework in which they can be answered should be given, though.

I would also like to know what problems Quicken sees that have to do with free will.

Alright, here it goes.

---

Why does the absence of an objective morality justify Volitional Friendliness? What is the rationale behind it? This is not intuitively obvious. Maybe it's answered in CFAI, in that case I should go read it again; but I don't recall anything convincing.

Is it a utilitarian sort of thing - minimize the extent to which volitions are violated integrated over the future of the Universe - or a duty-ethics sort of thing - it's always wrong to violate someone's volition?
If the first, how do you define and measure the degree to which someone's volition has been violated?
If the first, couldn't it be advantageous to do a one-time big assimilation event so that many volitions are violated now but none will have to be violated in the future?
If the second, is every deed that causes anyone's volition in your future light cone to be violated ethically forbidden? If so, isn't every deed already forbidden? If not, what deeds are ethically forbidden and what deeds aren't? Is a distinction between "causing" something and "not preventing" something needed?
If neither the first nor the second, but if making violation of volition impossible is merely a heuristic that is useful to some other goal instead of something to be pursued for itself, what is that other goal?

Does volition "count" across time - do your past volitions count, or do only present volitions count? (Comments to the quote from "Diaspora" in CFAI suggest the first, other information suggests the second)
If past volitions count, what notion of "diachronic identity" is used to identify mind states at different times as being the same person? What evidence is there that such a notion could (or would have to) exist without being completely arbitrary? (*)
If the definition of identity will be chosen by the mind it applies to, does that mean I can say "future-me" is the whole Universe?
If only present volitions count, does that mean it would not be unethical for someone to control someone as a "brain puppet", provided some sort of assimilation has already been done?
If past volitions don't count, does that mean I can't say "do what you think is best without bothering me", or anything else that involves planning my mind state in advance? Does it mean volition has to be re-verified at every moment?

Do I have volitional "authority" over anything other than whether my mind state changes, is destroyed, etc?
If yes, over what?

Can a Post-Singularity "information sea" (I like this term that Stephen Baxter uses better than "brain soup" ) always be unambiguously resolved into different minds (or persons, or decision mechanisms, and so on)? If minds can overlap, what happens if they overlap for 99.9%? Are you counting them as having double normal validity?

What about changes to reality that don't change one's mind state, but would otherwise be considered important - such as unnoticed uploading? If unnoticed uploading is something that volition applies to, what other aspects of physical reality that do not affect one's mind state does it apply to?
If unnoticed uploading can be a violation of volition, can other changes to the matter one is implemented on that don't change the mind that is implemented on it - such as small translations in space - also be violations of volition?
Is it a violation of volition to suspend someone's mind-program for one attosecond (against vis will)? If yes, is it an equally large violation as killing? (If not equivalent to killing, does the recreating in fact cancel out the killing? If yes, can recreating other people cancel out a killing?)
If this is not a violation, what time does it have to be (instead of an attosecond) to become a violation?

Do people always have volitions? How are they defined?
If you intend to find out volitions by asking, doesn't the volition often depend sensitively on how the asking is done? And can't the asking itself be a violation of volition, to those who don't want to know things, or don't want to be disturbed?
If you intend to find out volitions by scanning brains, how do you deduce volitions from this information? If by simulating, isn't that unethical? And what about privacy?
If this is solved by substituting "informed preferences", how informed? Superintelligently informed?

Volitional Friendliness has counterintuitive consequences, such as people killing themselves on a whim (easy with a Sysop that satisfies requests), insane people torturing themselves on a whim, and so on.

If all these questions are not for us to answer and will have to be answered by the AI, what is evidence that these questions have correct answers at all? By what standard would different definitions of volition be judged?
And what of all this will be needed when first programming or explaining volition to the AI?

I also think there should be another term for volitional Friendliness. The way it is now is confusing and allows for conflating different concepts. Random suggestion: v-Friendliness for volitional Friendliness, s-Friendliness ("seed") for Friendliness meaning developing from the human moral frame of reference.

(*) I don't believe an obvious definition for diachronic identity exists (at most an approximate one) (I think it is just a matter of definition rather than fact), because:
- physical continuity is not necessary or sufficient, as any uploadist will agree;
- continuity of consciousness isn't necessary, as is proven by going to sleep;
- memories aren't necessary or sufficient, otherwise one could become a different person by having different memories implanted, or by forgetting things; also, this is a matter of degree and not kind
- the only thing I see left is similarity (of personality, etc); that is also a matter of degree, and I think it wouldn't work for other reasons
- there is no reason it should be possible to extend a notion of personal identity formed in a world where there is conveniently one brain per meatbag to stranger posthuman situations; if any ethical philosophy requires it, that's that philosophy's problem.

---

Discuss, or else

[Psychodelirium]

Why does the absence of an objective morality justify Volitional Friendliness? What is the rationale behind it? This is not intuitively obvious. Maybe it's answered in CFAI, in that case I should go read it again; but I don't recall anything convincing.

Volitional Friendliness is the sort of friendliness that we humans want, given that we are designing a tool for the satisfaction of our intersubjective moral community, and not a dictator telling us how to behave. I do not look at it as a matter of "the absence of an objective morality justify[ing] Volitional Friendliness". Rather, I view it as a matter of Volitional Friendliness being assumed as the sort of model that we want, and if there just happens to be an objective morality that the FAI can discover, we want it to be able to handle this fact.

I have argued at length that the latter possibility is complete nonsense, and designing an FAI required to waste time and resources pondering this nonsense is probably a bad idea (or at worst, a suicidal idea). However, since it is too much to hope for that everyone involved shares my meta-ethical position, I am content - for the moment - with an FAI that can handle both situations. After all, it is intended to grow into the sort of moral conceptual scheme within which we humans reason about morality, so presumably it will form its own conclusions about realism, intersubjectivism, and what have you. There are numerous reservations that I have about this, but they will have to wait for when next the urge to rant strikes me.

Can a Post-Singularity "information sea" (I like this term that Stephen Baxter uses better than "brain soup" ) always be unambiguously resolved into different minds (or persons, or decision mechanisms, and so on)? If minds can overlap, what happens if they overlap for 99.9%? Are you counting them as having double normal validity?

This strikes me as one of those questions that we shouldn't be asking, because we probably wouldn't understand the answer. I would say that this is probably not even relevant to FAI morality anyway, since in a Post-Singularity "information sea", it would be just another agent involved in communication and wouldn't have any privileged access to resources. Our concern is getting there, not trying to imagine what we do once we will.

You seem so wrapped up in foundationalist reasoning that you are neglecting a very obvious answer to many of your questions regarding Post-Sing ethics (questions which I find to be a waste of time to begin with): the agents will figure it out amongst themselves. Answers that you see as arbitrary, I see as contingent, but nonetheless meaningful, products of intersubjective moral discourse. If one adopts this perspective, it immediately becomes obvious that it is not up to us to role-play a Post-Sing moral community, because that community will not reason about morality on the same level that we do.

there is no reason it should be possible to extend a notion of personal identity formed in a world where there is conveniently one brain per meatbag to stranger posthuman situations; if any ethical philosophy requires it, that's that philosophy's problem.

Exactly!

[Mechanus]

Volitional Friendliness is the sort of friendliness that we humans want, given that we are designing a tool for the satisfaction of our intersubjective moral community, and not a dictator telling us how to behave.

How do you know that's the sort of Friendliness we want (it's not the sort I want, for one); and how do you know it will still be the sort of Friendliness we want after intelligence increase?

Also, others, such as Eliezer, seem to believe in volitional Friendliness for other reasons than that that's what we want.

I do not look at it as a matter of "the absence of an objective morality justify[ing] Volitional Friendliness". Rather, I view it as a matter of Volitional Friendliness being assumed as the sort of model that we want, and if there just happens to be an objective morality that the FAI can discover, we want it to be able to handle this fact.

I don't see a difference. What I'm asking is still, why "if not (objective morality), then (volitional Friendliness)"?
If it's clearer, I'll just ask: "why volitional Friendliness?"

I have argued at length that the latter possibility is complete nonsense,

After re-reading the thread at the SDFs recently, I think I have figured out what approximately your position is (and why we were talking past each other), and I'm not really convinced. This is not the place to start on that subject, and I'm still confused by what part of what you are saying means, so again - is there a link anywhere that explains how all these things work (intersubjective communities of agents that construct moralities, etc) and why they are correct and relevant?

However, since it is too much to hope for that everyone involved shares my meta-ethical position, I am content - for the moment - with an FAI that can handle both situations.

What sort of design would you propose for a sort of AI that can handle subjective but not objective morality?

You seem so wrapped up in foundationalist reasoning that you are neglecting a very obvious answer to many of your questions regarding Post-Sing ethics (questions which I find to be a waste of time to begin with): the agents will figure it out amongst themselves.

That doesn't seem like a satisfactory answer. It seems strange that even if you don't know what something is, you still know you (let alone our moral community) want it.

If volitional Friendliness is to be explained to an AI as it grows up, we should at least know what we mean by it.

And again - even if this is the correct way to think about it, it isn't how most Singularitarians do, so the questions are still relevant.

[Psychodelirium]

Volitional Friendliness is the sort of friendliness that we humans want, given that we are designing a tool for the satisfaction of our intersubjective moral community, and not a dictator telling us how to behave.

How do you know that's the sort of Friendliness we want (it's not the sort I want, for one)

When I say "we", what I mean is that this is what most people would say if you were to ask them. A Friendly AI, pretty much by definition of the word "Friendly", is an AI that behaves in a way which we humans find to be right. We humans, as in not the universe, not God, not some abstract platonic code of behavior. If you want a Moral Dictator, then call it such. But speaking concretely, if you want a Moral Dictator, then I don't really know what else to say other than, "I hope you don't get one."

how do you know it will still be the sort of Friendliness we want after intelligence increase?

I don't think this is a relevant question to ask, because, like I said before, we cannot role-play other moral communities. We can't base our judgment in the here and now on what we might think if we belonged to some other moral community. We have to base it on what we do think, here and now. I view the question as analogous to telling an anthropologist who is planning to travel by plane to study a foreign culture, "what if you live in that culture long enough that you start to think like them, and they think that planes are evil? Better not take that plane."

I would say also that (1) FAI is no longer relevant to a posthuman community (except qua another agent to engage in moral discourse with), since it has no privileged access to resources, and (2) FAI is relevant to the community of normal humans in a Post-Sing world, which, obviously, will still reason about morality the same.

Also, others, such as Eliezer, seem to believe in volitional Friendliness for other reasons than that that's what we want.

I am not aware of this. [blink] Care to elaborate? I am not certain what Eliezer's meta-ethical views are, but the way I understand it, he would agree with me that the point of creating FAI is satisfying human preferences.

I don't see a difference. What I'm asking is still, why "if not (objective morality), then (volitional Friendliness)"?
If it's clearer, I'll just ask: "why volitional Friendliness?"

Because that's what we want. We want our volition to be respected and our preferences to be satisfied. There's really nothing else to say when you keep playing Socrates in moral discourse (or in any other discourse for that matter). Eventually you come to the point when I have to say, "just because." And "just because" is the history of the last couple of centuries of Western philosophy in a nutshell. Good riddance, foundationalism, you make no sense.

is there a link anywhere that explains how all these things work (intersubjective communities of agents that construct moralities, etc) and why they are correct and relevant?

I'll try to find some tomorrow.

What sort of design would you propose for a sort of AI that can handle subjective but not objective morality?

Actually, I think the design is fine. I'm more worried about the AI growing up in an environment with programmer feedback about meta-ethical questions that consists mostly of expectations that the AI will find some "One True Morality", or construct moralities progressively closer to the "One True Morality", and similar predominately neo-platonic viewpoints. If the AI came to believe that it was being "objective" about morality, I would be very, very afraid.

You seem so wrapped up in foundationalist reasoning that you are neglecting a very obvious answer to many of your questions regarding Post-Sing ethics (questions which I find to be a waste of time to begin with): the agents will figure it out amongst themselves.

That doesn't seem like a satisfactory answer. It seems strange that even if you don't know what something is, you still know you (let alone our moral community) want it.

I don't see how this is a reply to the quoted text. I am certainly not saying that I have any answers to Post-Sing ethical questions. In fact, I am saying just the opposite: we do not belong to the moral community that would concern itself with Post-Sing ethical questions, about shared minds and so forth, so our philosophizing about these questions will yield only nonsense.

If volitional Friendliness is to be explained to an AI as it grows up, we should at least know what we mean by it.

The point is that what constitutes volitional Friendliness is going to change as the environment that agents consider in moral discourse changes. You said it yourself: it doesn't really make sense that a concept of volition dervied from a one mind/one brain environment would apply to a Post-Sing "information sea". Volition-based ethics will be radically different in the different contexts, so the questions, "what is volition here and now?" and "what will volition be there and then?" should not be conflated. We need address only the former.

[MichaelAnissimov]

[quote]Why does the absence of an objective morality justify Volitional Friendliness? What is the rationale behind it? This is not intuitively obvious. Maybe it's answered in CFAI, in that case I should go read it again; but I don't recall anything convincing.[/quote]

The way I see it, the definition of an "objective morality" is simply "past a certain level of intelligence, everyone agrees with it". In some vague sense, meta-might does define meta-rightness. If a FAI picks a morality that literally everyone is happy with, or at least sets down a minimum set of ethics which are so unambiguously moral that they endure for the rest of eternity, then you have something so close to a "true" objective morality that it's sufficient for all practical purposes. Volitional Friendliness and successor versions are justified because they are good, not necessarily because they're anchored to an objective morality (an ideal of good which may or may not truly exist.)

[quote]Is it a utilitarian sort of thing - minimize the extent to which volitions are violated integrated over the future of the Universe - or a duty-ethics sort of thing - it's always wrong to violate someone's volition?[/quote]
This distinction is about where most of the writing on Friendliness stops, because it digs deeper than the overarching questions which people are initially liable to worry about. But in such a massive world, this "little" distinction could obviously make a massively huge difference. This goes along with the question; "Will the first benevolent transhumans decide that it's safest to just preserve everyone's pattern but not run them until they have established a safe aleph point for everyone to enjoy, as a way of minimizing the extent to which volitions are violated integrated over the entire future of the universe, since the probability of actual death for human entities in a pre-aleph scenario is too great?" It's often dangerous to ask these questions, because for objectors overly focused on the *content* of Friendliness rather than the architecture (around 99.9%, probably), this can obviously be misleading. But someone will eventually have to talk about it, and that might as well be us...

I'd say that there really is an answer to this question which is "right", and while we might not be sure what that is now, it will be "obvious" to a transhuman mind with the right benevolence foundation, in the same way that it's obvious to decent people today that torturing someone is almost always wrong.

[quote]If the first, how do you define and measure the degree to which someone's volition has been violated?[/quote]

A few possible stages;

1) The volition of an individual is defined by the extrapolation of their brainstate to their internal monologue which represents their wishes and desires in the form of concrete propositions. Considerations of the extrapolation of wills of other entities must also be computed before a conclusion about volition can be reached.

2) Volition and violation of volition is determined only by explicit verbal declarations.

3) Volition and violation of volition is determined by explicit verbal declarations plus a minimum set of moral principles and ethics extrapolated therefrom which everybody unambiguously agrees on.

4) Volition and violation of volition is contingent upon what the collective memetic definition of what these terms mean in the first place.

5) Volition and violation of volition are contingent upon the singleton's definition (which is better than the definition of the collective or any individual human's definition, whether we like it or not.)

6) etc...

[quote]If the first, couldn't it be advantageous to do a one-time big assimilation event so that many volitions are violated now but none will have to be violated in the future?[/quote]

I personally think it might be Another term for this is "seamless global upload", although that doesn't really capture the true meaning sufficiently. Can't really guess what the Sysop would choose, though - I'm not that good at imaging what a purely altruistic superintelligent mind would actually choose to do with a supermoral extension of my own (humanity's) morality.

[quote]If the second, is every deed that causes anyone's volition in your future light cone to be violated ethically forbidden? If so, isn't every deed already forbidden? If not, what deeds are ethically forbidden and what deeds aren't? Is a distinction between "causing" something and "not preventing" something needed?[/quote]

For humans, I suppose the default for this is usually "within my lifetime", but immortality (and other factors, such as intelligence increase), render this a bit moot. In comparison to the ancestral environment, where humans were illiterate and hence non-timebinding, the window of time for determining what constitutes a violation of volition was probably shorter than it is today, although through carrying this to the extreme I can see how all actions can potentially be forbidden, ethically. Maybe the way this will be approached will be via a complex, intertwined network of "moral common sense", rather than the straightforward execution of easy-to-understand ethical heuristics? I guess this is just a slightly different angle than Psychodelirium's view, but I *do* think a moral optimum will eventually be achieved and we will reach a "final moral outcome" of sorts - I just think it's too far away for us to see right now, as humans. And there's also the possibility that all goals are ultimately arbitrary...in which case the AI will probably just ground itself in some pansentient optimized morality in which the boundary condition of the universe is represented by individual volition, is my human guess.

[quote]If neither the first nor the second, but if making violation of volition impossible is merely a heuristic that is useful to some other goal instead of something to be pursued for itself, what is that other goal?[/quote]

Finding objective morality..? Maybe the prerequisite for higher forms of morality is implementing successively closer approximations and see what pops out of the meme/vemepool.

[quote]Does volition "count" across time - do your past volitions count, or do only present volitions count? (Comments to the quote from "Diaspora" in CFAI suggest the first, other information suggests the second)[/quote]

I would guess that "volitions" only after the Singularity matter, because for the other stuff, what's done is done. Although this surely invokes a whole new category of objections and questions, of course...

[quote]If past volitions count, what notion of "diachronic identity" is used to identify mind states at different times as being the same person? What evidence is there that such a notion could (or would have to) exist without being completely arbitrary? (*)[/quote]

I would guess the notion as defined by the individual in question would be the notion used. Something isn't "arbitrary" if it results in higher goodness, if goodness turns out to be a higher reality-forming pressure than unarbitrariness.

[quote]If the definition of identity will be chosen by the mind it applies to, does that mean I can say "future-me" is the whole Universe?[/quote]

Maybe an averaged weight of the definitions of all entities in the universe would be better, or, the singleton could possibly define an individual as one six billionth of all the mass/computation in the universe, or however many entities there are, because the bit is a fundamental unit of reality, as far as I can tell...(I may be wrong.)

[quote]If only present volitions count, does that mean it would not be unethical for someone to control someone as a "brain puppet", provided some sort of assimilation has already been done?[/quote]

You mean, already had been done before the Singularity?

[quote]If past volitions don't count, does that mean I can't say "do what you think is best without bothering me", or anything else that involves planning my mind state in advance? Does it mean volition has to be re-verified at every moment?[/quote]

Oh my, I should hope not...the interesting thing about looking at morality so finely is that most people don't even think about this stuff. As you make finer and finer moral distinctions, you eventually reach a point where essentially everyone is happy, an extremely high degree of convergence is achieved, and no one can tell the difference. This is constitute reaching an "objective morality"? I would argue yes.

Re-verifying volition at every moment is absurd, but heck, I don't know how often volition should be reverified. This may be cheating, but heck, maybe I should help contribute to the creation of a superintelligent morality-generating engine that can extrapolate my moral theory with ver unimaginable intelligence to show me what I really meant

[quote]Do I have volitional "authority" over anything other than whether my mind state changes, is destroyed, etc?
If yes, over what?[/quote]

"One six-billionth of the mass in the solar system" is the usual answer.

[quote]Can a Post-Singularity "information sea" (I like this term that Stephen Baxter uses better than "brain soup"  ) always be unambiguously resolved into different minds (or persons, or decision mechanisms, and so on)? If minds can overlap, what happens if they overlap for 99.9%? Are you counting them as having double normal validity?[/quote]

Maybe sacrifices in morality-computing will need to be made for the sake of expansion and improvement, if the collective morality determines that expansion and improvement for the sake of expansion improvement is important enough that it deserves to supervene on morality-computation for part of the process? I would guess that the universe could be differentiated into different zones with different relative rates of morality-computation and progress-seeking, too.

[quote]What about changes to reality that don't change one's mind state, but would otherwise be considered important - such as unnoticed uploading? If unnoticed uploading is something that volition applies to, what other aspects of physical reality that do not affect one's mind state does it apply to?[/quote]

Fantastic question! Extending volition to all aspects of physical reality regardless of how they influence the mindstate means digging up the lowest level of reality whenever possible - a task which might be theoretically impossible to ever be sure of completely accomplishing. My guess is that extending volition to unnoticed uploading and beyond may be overkill - if someone nukes the supermarket while I'm in it, that changes my mindstate, but if I'm at home about to go to the supermarket when that happens, it doesn't directly alter my mindstate at that moment, although it will do so undesirably in the future. It seems that humans have a "noticing field" constrained to a very small portion of possible sizes, shapes, and configurations of empirical regularities in the universe, and our volition may only be important insofar as it concerns those portions. If this is true, then SIs will be able to the use the insides of all our furniture (and bodies!) to perform googles of computations

[quote]If unnoticed uploading can be a violation of volition, can other changes to the matter one is implemented on that don't change the mind that is implemented on it - such as small translations in space - also be violations of volition?[/quote]

I hope not; as you've said before, if one person leading a happy, productive life is good, then billions of people leading happy, productive lives is billions of times as good. Which is more important under a given system of morality - preserving individual volition of existing entities, damn the cost, or removing constraints that would allow the creation of billions of more happy entities? Ack, I don't really know, personally.

Sorry; I can't handle the rest for now, but it all very insightful! My guess is that with the right architecture and some sort of seed content, the first normative benevolent mind will converge to a moral optimum using arguments and justifications that would convince us all if we could only hear them :D

[Mechanus]

[quote]When I say "we", what I mean is that this is what most people would say if you were to ask them.[/quote]

I very strongly doubt that most people would say that they preferred a volitional Friendliness system. Most might say something like, "whatever best serves the glory of God", or whatever gave them a big car, or a lot of sex, or a pure ecosystem. Or actually, I'd guess most would be strongly opposed to the creation of a superintelligent AI.

Should there be a world-wide referendum before a Singularity is attempted? (this question is intended seriously and is IMO interesting enough to start a thread on, unless no one else thinks it's interesting)

[quote]A Friendly AI, pretty much by definition of the word "Friendly", is an AI that behaves in a way which we humans find to be right.[/quote]

That's one of the many definitions, yes. I propose to call this one "i-Friendliness" for intuitive Friendliness.

[quote]If you want a Moral Dictator, then call it such. But speaking concretely, if you want a Moral Dictator, then I don't really know what else to say other than, "I hope you don't get one." [/quote]

I don't necessarily want a moral dictator. It seems to me there could be many other options than volitional Friendliness, without having anything that you could reasonably call a moral dictator - there could certainly be many other options than the "standard model" of volitional Friendliness. You could try emphasizing someone's informed preferences over someone's preferences, for example.

Anyway, the point was - if I have to choose something I prefer (as a small part of the intersubjective moral community), I wouldn't say "I prefer volitional Friendliness" or "I prefer a Moral Dictator"; I might say "I prefer a future where I have as low a probability as possible of being tortured", or "I prefer a future where as much is found out about reality as possible" (why? because!). For both of these things, volitional ethics would probably not be the best system to base things on. I could be part of a small minority, though - I don't know.

[quote]I don't think this is a relevant question to ask, because, like I said before, we cannot role-play other moral communities.[/quote]

I don't understand why it isn't a relevant question to ask - if the outcome depends on roleplaying a (superintelligent) moral community, doesn't that make my point that we can't yet say that volitional Friendliness will be what we want?

[quote]I view the question as analogous to telling an anthropologist who is planning to travel by plane to study a foreign culture, "what if you live in that culture long enough that you start to think like them, and they think that planes are evil? Better not take that plane."[/quote]

More like, "what if you live in that culture long enough that you start to think like them, and they think that planes are evil? Better not claim with any certainty that you will always like planes."

I'm not saying "we will want something else, therefore we should not pursue what we want now" - I'm saying "we will want something else, therefore we should not conclude that what we will want is much like what we want now". I'm just disagreeing with a prediction here, that's all.

[quote]I would say also that (1) FAI is no longer relevant to a posthuman community (except qua another agent to engage in moral discourse with), since it has no privileged access to resources,[/quote]

This would not be true in a Sysop scenario, if the FAI became Sysop, for example.

[quote]I am not certain what Eliezer's meta-ethical views are, but the way I understand it, he would agree with me that the point of creating FAI is satisfying human preferences.[/quote]

Yes, but very indirectly. I think he considers the point of FAI to be pursuing what the goals of a normative human altruist would be after transcending to superintelligence. I'm fairly sure he wouldn't agree that the point is to pursue whatever humans happen to want at this moment.
In the quote in the other thread he says volitional Friendliness is a consequence of moral symmetry.

[quote][quote]I don't see a difference. What I'm asking is still, why "if not (objective morality), then (volitional Friendliness)"?
If it's clearer, I'll just ask: "why volitional Friendliness?"[/quote]

Because that's what we want. We want our volition to be respected and our preferences to be satisfied. There's really nothing else to say when you keep playing Socrates in moral discourse (or in any other discourse for that matter).[/quote]

My mistake - I meant that how I phrased the question and the question you said you would ask instead were really the same thing. I did not mean that I wanted to ask the same question again - assuming for the moment that your view of metaethics is correct (and sufficiently well-defined), then "because we want it" is a satisfactory answer, provided that you can convince me that most humans really do want it.

[quote]Actually, I think the design is fine. I'm more worried about the AI growing up in an environment with programmer feedback about meta-ethical questions that consists mostly of expectations that the AI will find some "One True Morality",[/quote]

But don't you agree that that is very unlikely to determine how the AI will actually turn out when superhumanly intelligent - especially if the idea of a "One True Morality" is as ridiculous as you claim?
I thought you were unhappy with the possibility of objective morality being left open - that possibility will even be left open if programmer feedback will be entirely pro-(inter)-subjective morality. (And I think it will be, mostly; this does not bother me)

[quote][quote][quote]You seem so wrapped up in foundationalist reasoning that you are neglecting a very obvious answer to many of your questions regarding Post-Sing ethics (questions which I find to be a waste of time to begin with): the agents will figure it out amongst themselves.[/quote]

That doesn't seem like a satisfactory answer. It seems strange that even if you don't know what something is, you still know you (let alone our moral community) want it.[/quote]

I don't see how this is a reply to the quoted text. I am certainly not saying that I have any answers to Post-Sing ethical questions.[/quote]

And I am not saying you are saying that
The word "answer" in my post refers to "a very obvious answer" in your post. That very obvious answer ("mu") is not completely satisfactory to me, because if we don't know what it is, then it seems strange that we already know we want it.

[quote]In fact, I am saying just the opposite: we do not belong to the moral community that would concern itself with Post-Sing ethical questions, about shared minds and so forth, so our philosophizing about these questions will yield only nonsense.[/quote]

I do think you're underestimating how relevant many of the questions are to present-day volitional ethics, rather than just after the Singularity - the stuff about shared minds and so on is not relevant yet, but some of the other questions are.

Even if many of the points made here are not relevant to your view of the matter, they are relevant to the view of many other Singularitarians who think there will turn out to be one obvious "right" kind of volitional Friendliness.