This discussion inspired by web-page Weaknesses of AIS.
David Sanders> AIS cannot exist (for now) without humans.
That’s not really a weakness, because time span of this weakness would be pretty short. Right now strong AI systems exist only in our dreams. :-) Within ~20 years of creating strong AI, many AIS-es would be able to survive without humans. Please, note that AIS-es would not kill humans. There would be benefits of human-AIS collaboration for all sides. This is completely different topic though. :-)
David Sanders> If they fail to understand and appreciate the human world...
If you don't understand and appreciate human world of Central Africa... would it harm you?
May be you mean "If AIS-es don't understand human world at all"? But in this case what would these AIS-es understand? And what would mean that these not-understanding systems intelligent?
David Sanders> [AIS-es] Not able to perceive like a human. They cannot hear, see, feel, taste or smell like a human.
Not true. Only first and limited versions of AIS-es wouldn’t be able to perceive like a human. Sensor devices are not too hard to implement. The major problem is implementation of Main Mind for AIS.
David Sanders> They can only feel these things like they imagine they do. Again, this makes them fundamentally incongruous with humans and I don't believe its something you can "teach around." Try to explain what "blue" is to someone who never had sight.
Have you ever seen "black hole", "conscience", or "electron"? Yet you know what they are, don't you? :-)
Blind person can understand what "blue" means: "sky is blue", "water is blue", ...
David Sanders> Until AIS have robot bodies / companions, they rely on humans for natural resources. However, once the singularity hits, that probably won't matter anymore. It is not inconceivable to think of a time in 200-500 years there are no more humans, just AIS.
Humans would probably exist long after strong AI is created. Humans just would not be the most intelligent creatures anymore :-)
David Sanders> I disagree with AIS and natural selection. I think this will happen on its own by their very nature.
AIS-es can be influenced by natural selection as much as all other living organisms. But humans had millions of years of natural selection. When would AIS-es have that much?
David Sanders> AIS will be more open about self modification as you point out. AIS will be able to make other AIS and will soon learn how to evolve themselves very quickly.
"Evolving themselves" is part of artificial selection, not natural selection.