Hi Gewis,
Interesting analysis. Overall, I found it a very interesting series of thought experiments. I hope you don't find my response overly critical, and please feel free to ask for clarification on anything that is unclear:
I find parts of the above pargraphs to be anthropomorphic. For example, there is no law that "more intelligent beings engage in less mob behavior". Within the *human* sphere, yes, more intelligent humans can tend to be less moblike. However, I can imagine a society of moblike superintelligences, just as easily as I can imagine a society of unmoblike ones.
I don't call post-Singularity superintelligences "AIs" because I don't think that is the best word to describe them. Remember, Artificial Intelligences automatically get a speedup factor of millions or billions relative to human beings. AIs will be self-modifying and just plain
smarter than humans. (
http://www.singinst....tro/impact.html) Recursive self-improvement is the idea of a sentient entity making design improvements to the underlying architecture it runs on, which humans would be capable of doing if we were uploaded. Also, there is the idea of brains that can make use of all the hardware available - if you have an AI that is 10^17 ops/sec big that invents nanocomputing and therefore can squeeze 10^22 ops/sec out of its current computing material through usage of that technology, then you almost instanteously get an AI that thinks with 100,000 times the computational resources it had before. It goes literally beyond our ability to comprehend.
There will be no interim period with human-equivalent AIs thinking at roughly human-equivalent speeds, engaging in power struggles in the same way that humans have with each other for millions of years. Human-equivalent AI is anthropomorphism - you start off with a slightly-dumber than human seed AI, which, through the tremendous advantages it has by virtue of its computational substrate, becomes capable of bootstrapping itself to superintelligence, at which point you get superintelligence. There is no game-theoretical similarity between superintelligence and human beings. If it is superintelligence that does not explicitly care for humanity, (or all sentient beings, or whatever) then we will be viewed as building materials, and destroyed very rapidly during the explosion of recursive self-improvement.
I think you're viewing the process of creating AI like the process of giving birth to new humans; they gradually pop up, interact with each other and the larger society for reputation and resources, and Gaussian distributions of emotions within characteristic boundaries and constraints, which coincidentally match the human boundaries and constraints. The attitudes of future AIs will be contingent on their initial design, because the pattern of all AIs after the first AI starts to recursively self-improve will reflect the goal system of the first AI. (Or, say that a few AIs come first that aren't into recursive self-improvement. Fine, but eventually a RSI AI will come into existence regardless, and unless all the AIs can sense the self-improvement of the others and begin self-improving simultaneously, the state of the world after that day will reflect the morality of the first AI that began to self-improve*.)
Humanity has a characteristic distribution of emotions and tendencies, which reflect our evolutionary past. It was never adaptive to treat your children as worthless. Humans that actually do (and they are *extremely* few) are "broken" from the viewpoint of evolution - something went critically wrong in their development or ontogenesis, like the serious deprivation of oxygen or something. All neurologically normal humans have the same set of emotional hardware for making judgements, forming internal sensations, and creating responses; they are just tuned to different activation thresholds, and conditioned by memory to be associated to slightly different things. These pieces of hardware possess the design signature of biological evolution, and are tuned to respond to ancestrally relevant cues (for example, human facial expressions) only. There are no humans born "genuinely altruistic" or "genuinely malevolent" - the machine called a human just responds differently when it is placed in different contexts or experiences the relevant sets of cues. A baby will become more malevolent, on average, if raised within a malevolent family or society. There are slight propensities to one direction or the other, but the reason why we see no
completely altruistic human beings or completely malevolent ones is because neither of these emotion-sets were adaptive. I can imagine a parallel universe where they were, though. We would be at a loss to interpret the emotions of aliens with different, or substantially more complex facial expressions than us.
AI designers will stand with respect to AIs in the same way that evolution stands with respect to us. "Malevolence", or say, "jealously" will not exist in the AI unless the programmers put it there. These complex human responses are attributable to complex underlying machinery put there by millions of years of evolution - they do not pop up spontaneously with equal frequency in blank slate minds. Just because a human designs an intelligence does not mean that intelligence will have the qualities of a human. The qualities of a human are unnecessarily complex relative to the bare-minimum engineering requirements for AI, and even the simplest of them will be outdone by what AI designers come up with. Evolution is naturally slow, blind, and constrained by a host of variables, making it a poor designer relative to intelligent engineering. For this reason, I don't think AIs will have cognitive features heavily inspired by human ones. If the
first AI does, and this first AI starts recursively self-improving, the human qualities the programmers put there will either be irreversibly changed through renormalization, reinforced and improved, or yanked out entirely. Self-improvement would take place very rapidly relative to human timescales. If open-ended self-improvement is not taking place, the AI will either be 1) getting out of its confines and accomplishing its goals, whatever they may be, 2) busy trying to get someone to let it out of its confines, or 3) expensive and complex enough to have near-human intelligence, but too stupid to conduct the improvement of its own design.
I think the first generation of independent AIs will not harbor the same worries that we do about them today, because 1) "worry", in the negative sense, is an evolved human emotion that gets in the way of sane thinking and makes the mind paranoid, and 2) there never needs to be a "second generation" of AI because continuity never needs to be broken between the first seed AI and the latest and greatest superintelligences. With regard to 1, I doubt AI programmers would ever insert "worry" in the human sense into the first AI, or at least I hope anyone smart enough to program an AI would see why human paranoid worrying is counterproductive to reaching goals. Or, in the worst case scenario, worry is renormalized into "a planning heuristic of selective attention, maximally useful for foresight, minus emotional baggage" from its earlier incarnation in humans. With regard to 2, AIs will not be physically detatched from the AIs they create in the same way that programmers would be physically detatched from the first AI to be coded. Physical attachment, in the sense of a self-improving AI creating a new being, can be arbitrarily close and arbitrarily precise; if the first AI is concerned about the state of the beings it creates, then it will spend a lot of time on tuning its child's emotions for the work of good and not evil. (I hope that doesn't imply inflexibility. Contrary to popular belief, it is possible to be morally good and simultaneously interesting.)
The following comments are slightly more speculative than the above ones, but I still believe both.
AIs won't fear their current instantation's obsolesence because they will surely not have identity theories as constraining as human ones are. In the world of humans, if someone's head turns into that of say, a rhino, then we would be worried. In the world of uploads and AIs, stuff like that will happen all the time. AIs might want self-upgrading, might not, but it would be silly of them to program themselves to fear for their identities, regardless of what happens. There should be enough computing power for every identity to have its own volition respected. (This might lead to presentient components within superintelligences combining in such a way that sentience and volition is created within the superintelligence, at which point the sentience should be offered the choice to leave the host body.) The question of "should we be this way because our creators intended it?" should not last long - it should be settled with the creation of the first AI. The first AI will acquire a morality through the cognitive content the programmers create it with, and further input from the programmers and external reality. The AI self-improves and makes changes to its own cognitive structure and external reality in order to maximally fulfill that morality. We figure that the AI's moral structure will settle into one of two major attractors; altruism, or, getting people what they want, or, egoism, getting itself what it wants. Altruistic minds that are self-modifying will be capable of making themselves entirely altruistic, and would only question their creation to the extent that the central goal content was preserved.** If these minds actually believe that
doing good is the right thing, then regardless of how they were born, they will continue to see doing good as a correct goal. Human moralities are designed to be sensitive to even small changes in external conditions; superintelligent moralities need not be like this.
I agree that the complexity of the power structure and struggle that accompanies the building of the first AI will be greater than that of the interaction of humans and machines, in the sense that you insinuate. If the first AI sees humans as building materials and not as moral agents, then there will be no struggle - humans will be swallowed by the self-improvement process of this AI. The only power structure would be the AI directing its complex and global motor affectors to gather materials for repatterning into new structures. The introduction of benevolent AI into society, if it happens, will make the idea of "humans falling behind" irrelevant - evolutionary competition as we know it would cease, and all humans would need to be offered the chance to be as smart as they desire with their share of resources. There would be no need to be "at the top" because the beings at the top would already be representing us and upholding our rights. If an SI cares about you, it is easy for it to optimize everything for your well-being, plus whatever superintelligent caveats need to be added to that action.
Thank you for the fascinating discussion!
Creating Friendly AI:
http://www.singinst.org/CFAIWhat is Friendly AI?:
http://www.kurzweila...tml?printable=1*If this AI's morality is good, then the world reflecting the AI's morality will simply display more freedom, happiness, or other qualities sentients find morally valuable.
**It sounds again like this is limiting, but building an altruistic AI of this sort seems like mankind's only safe pathway to the future. There may be no objective morality, in which case the AI will need to work with what it has and simply work to enforce the rights of as many beings as possible. History has shown a clear progression of better and fairer moralities - we have no reason to believe that this improving trend would not continue if we created AIs even more benevolent and intelligent than we are.