1. Ethics for Machines J Storrs Hall

2. Stick-Built AI ● No existing AI is intelligent ● Intelligence implies the ability to learn ● Existing AIs are really “artificial skills” ● A human e.g. Grandmaster will have learned the chess-playing skills ● It's the learning that's the intelligent part ● Providing ethical constraints to stick-built AI is just a matter of competent design

3. Autogenous AI ● Truly intelligent AI would learn and grow ● Create new concepts and understand the world in new ways ● This is a problem for the ethical engineer:  Cannot know what concepts the AI will have  Can't write rules in terms of them

4. The Rational Architecture ● WM (world model) predicts the results of actions ● UF (utility function) evaluates possible worlds ● The AI evaluates the effects of its possible actions and does what it predicts will have the best results ● This is an ideal except in the case of very simplified worlds (e.g. chess)

5. Learning Rational AI ● WM is updated to use new concepts to describe the world in ● WM changes can be evaluated based on how well they predict ● But on what basis can we update the UF?

6. Vernor Vinge: A mind that stays at the same capacity cannot live forever; after a few thousand years it would look more like a repeating tape loop than a person.... To live indefinitely long, the mind itself must grow... and when it becomes great enough, and looks back... what fellow-feeling can it have with the soul that it was originally?

7. Invariants ● We must find properties that are invariant across the evolutionary process ● Base our moral designs on those

8. A Structural Invariant ● Maintaining the grasp, range, and validity of the WM is a necessary subgoal for virtually anything else the AI might want to do ● Socrates put it:  There is only one good, namely, knowledge; and only one evil, namely, ignorance.

9. Evolving AI ● Current AI only evolves like any engineered artifact  The better it works, the more likely the design is to be copied in the next generation ● Once AIs have a hand in creating new AIs themselves, there will be a strong force toward self-interest

10. The Moral Ladder ● Axelrod's “Evolution of Cooperation” ● Subsequent research expanding it  ALWAYS DEFECT is optimal in random environment  GRIM is optimal in env. of 2-state strategies  TIT-FOR-TAT in env. of human-written strategies  PAVLOV in env. cleared out by TIT-FOR-TAT  etc.

11. Open Source Honesty ● Intelligent autonomous agents are always better off if they can cooperate ● Even purely self-interested ones ● Ascending the moral evolutionary ladder requires finding others one can trust ● AIs might be able to create protocols that would guarantee their motives ● e.g. Public-key signed release of UF

12. The Horizon Effect ● Short planning horizons produce unoptimal behavior ● A planning horizon commensurate with the AI's predictive ability is evolutionarily stable ● What goes around, comes around, especially in an environment of superintelligent AIs ● Honesty really is the best policy

13. Invariant Traits ● Curious, e.g strongly motivated to increase its understanding of the world ● Self-interested ● Understands evolutionary dynamics of the moral ladder ● Capable of guranteeable trustworthiness ● Long planning horizon

14. The Moral Machine ● If we start an AI with these traits, they are unlikely to disappear in their essence, even if they get changed in their details beyond current recognition