1. Some thoughts on the future
Stuart Russell
University of California, Berkeley

6. What’s going on? Have we thought enough about what that would mean?
Eventually, AI systems will make better* decisions than humans
Taking into account more information, looking further into the future
The biggest event in human history:
Everything we have is the product of intelligence
An end to war, disease, poverty, ecological degradation
Shaping our own destiny in the universe
Have we thought enough about what that would mean?

7. Why is there a problem?
Obviously, smarter-than-human AI systems are intrinsically hard to predict and control
In particular, AI systems that are incredibly good at achieving something other than what we* really* want
AI, economics, statistics, operations research, control theory all assume utility to be exogenously specified
I.e., it’s someone else’s job to get it right!

8. Value misalignment
If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively … we had better be quite sure that the purpose put into the machine is the purpose which we really desire Norbert Wiener, 1960 King Midas, c540 BCE

9. Responses so far: It’ll never happen It’s too soon to worry about it
See Rutherford, 9/11/33, Szilard 9/12/33
It’s too soon to worry about it
2066 asteroid collision: too soon to worry?
When should we have worried about climate change?
It’s like worrying about overpopulation on Mars
We are spending billions moving humanity to Mars with no plan for what to breathe
Just don’t have explicit goals for the AI
We need to steer straight, not remove the steering wheel
Don’t worry, we’ll just have human-AI teams
Value misalignment precludes teamwork

10. Responses so far (contd.):
You can’t control research
Yes, we can and have done for genetic engineering and biosafety protocols
You’re just Luddites
Fusion researchers are Luddites if they point out the need for containment?
Alan Turing, Norbert Wiener, Bill Gates, and Elon Musk are Luddites?
Don’t worry, we can just switch it off
Sure, and we can just checkmate Deep Blue
Only humans care about self-preservation
Wrong

11. Self-preservation A natural subgoal of almost any objective
Death isn’t bad per se. It’s just hard to get the milk after you’re dead

12. Time for AI to get serious
Like physics and biology, AI can significantly impact humanity
AI must expand its concerns to ensure the best possible outcome
Value alignment research
Corrigibility research
etc.

13. Cooperative inverse reinforcement learning
Learn a multiagent value function whose Nash equilibria optimize the payoff for humans
Vast amounts of evidence for human behavior and human attitudes towards that behavior
We need value alignment even for subintelligent systems in human environments; strong economic incentives!
Humans are irrational, inconsistent, weak-willed, computationally limited, and heterogeneous

14. Wiener, contd.
[this work] requires an imaginative forward glance at history which is difficult, exacting, and only partially achievable. … We must always exert the full strength of our imagination to examine where the full use of our new modalities may lead us

15. OK, so….
Foresight on development paths and barriers for integrated general AI
Design templates for safe AI
Value function industry
Inculcating ethics/common sense in AI community (cf. civil engineering)
Ethics research contributes at systemic level
Professional societies speaking for AI and helping to develop policy