1. CMPT 310 Simon Fraser University CHAPTER 4 Oliver Schulte
Local Search
CMPT 310
Simon Fraser University
CHAPTER 4
Oliver Schulte
Show example of gradient descent for neural nets.

2. Key Concepts
Hill-Climbing
Gradient Descent
2nd-order methods.

3. Game Theory and the PEAS Model

4. Games With the Environment
Even if there is only one decision-maker agent A, the PEAS model can still be thought of as a game.
Performance: the payoffs of the single “player” A.
Environment: a “player” E who makes moves.
Payoffs:
The E player may be indifferent: always 0 payoff.
Zero-sum: E player loses whatever A gains  Worst-Case Analysis.
Actuators: moves by A.
Sensors: Information about moves by E.

5. Example: Indifferent Environment
Agent A chooses action.
E does not care what happens.
There is uncertainty: E could “choose” L or R.
think about skytrain fare check.
A might know or have learned a probability distribution for E’s choices.
Environment E
Agent A L R T
1,0
0,0 B

6. Example: Adversarial Environment
Agent A chooses action.
Zero-sum: E wants A to lose.
To win, A needs to do worst-case analysis.
Environment E
Agent A L R T
1,-1
0,0 B

7. Deterministic Certain Environment
A deterministic environment without uncertainty can be modelled by allowing E only a single action.
Without uncertainty, choice is trivial for a small finite list of actions.
Environment E
Agent A L T
1,-1 B
0,0

8. Computational Choice Under Certainty
But choice can be computationally hard if there are infinitely many actions.
The payoffs may not be given as in a look-up table but as a function that needs to be analyzed.
CMPT Blind Search

9. Environment Type Discussed In this Lecture
Fully Observable
Static Environment
yes
Deterministic
yes
Sequential no
yes
Discuss discrete variables a little bit.
Discrete no
Discrete
yes no
yes
Continuous Function Optimization
Planning, heuristic search
Control, cybernetics
Vector Search: Constraint Satisfaction
CMPT Blind Search

10. Optimization Problems

11. Optimization Problems
An optimization problem is of the form maximize f(x) subject to constraints on x where x is a vector of values.
Equivalently minimize -f(x) subject to constraints on x
If the constraints are linear (in)equalities, we have a linear programming problem.
Very large literature built up over centuries.

12. AI examples
Find the right angle of joints for pancake-flipping robots.
Find the best weights for rules for reasoning.
Any problem with continuous variables at some point needs optimization to build the best agent.

13. Hill Climbing

14. Hill-climbing search
Trivially complete because with probability 1 tries best solution.
Random-restart hill climbing overcomes local maxima.
Random Sideways move: escapes from shoulders, loops on flat maxima.
Hill-Climbing Video

15. Hill-climbing search

16. Hill-climbing search: 8-queens problem (discrete)
Put n queens on an n × n board with no two queens on the same row, column, or diagonal
h is the min-conflicts function, see text.
h = number of pairs of queens that are attacking each other, either directly or indirectly
h = 17 for the above state

17. Hill-climbing search: 8-queens problem
Demo for n-Queens Hill-Climbing
aimacode.github.io/aima-java/aima3e/aimax-osm-IntegratedAimaOsmSwingApp.jnlp
go to Apps.search.games.NQueens
use incremental
A local minimum with h = 1

18. Gradient Descent

19. Gradient Descent: Choosing a direction.
Intuition: think about trying to find street number 1000 on a block. You stop and see that you are at number 100. Which direction should you go, left or right?
You initially check every 50 houses or so where you are. What happens when you get closer to the goal 1000?
The fly and the window: the fly sees that the wall is darker, so the light gradient goes down: bad direction.
Real life: bad relationship, job is local maximum.

20. Gradient Descent In Multiple Dimensions
Fairly Simple Visualization
More complicated examples.
should have been covered in Math150
Active Math Applet

21. Gradient Descent: Example.
Try to find x,y that minimize f(x,y) = 3x + y2.
Your current location is x = 10, y = -3.
What is ? ?
Answer: the gradient vector is (3, 2y).
Evaluated at the location (10,-3), the gradient is = (3, -6).
To minimize, we move in the opposite direction -.
Letting the step size = 1, your new location is (10,-3) - (3,-6) = (7, 3).

22. Gradient Descent: Exercise
Try to find x,y that minimize f(x,y) = 3x + y2.
Your current location is x = 7, y = 3.
The gradient vector is (3, 2y).
Letting the step size = 1, what is your new location?.
Excel Demo
Answer: x = 4, y = 3 – 6 = -3.

23. Gradient Descent for Learning Neural Nets
Problem: Given input observations, how can a brain rewire the connections among its neurons to remember the observations?
Most common answer: gradient descent!
Applied in deep learning, Baidu search, Skype translation.
See AIspace demo.

24. Local Search With Second-Order Information
Using acceleration, curvature

25. Better Approach: Newton-Raphson
Uses information about second-order derivatives.
Over 300 years old.
Idea: Step size is inversely proportional to 2nd-order derivative.
Update Rule: x := x - f'(x)/f''(x).
Why does that make sense? Look at hill-climbing graph.

26. Newton-Raphson Example
Update Rule: x := x - g(x)/g'(x).
In our example with f(x,y) = 3x + y2 there is no curvature in the x-dimension, so we use NR only in the second dimension with
You can use NR with step sizes, but the method doesn’t require it.
If the acceleration is high, slow down. Otherwise speed up.
Bold vector = (x,y) pair.
So the new location is (10,-3) - (3, -6/2) = (7,0).

27. Derivation from Newton’s Method
Problem: find a root x such that g(x) = 0.
Use update rule. x := x - g(x)/g'(x).
Geometry: fit a line (tangent) to g(x), move to intersection with x-axis.
Demo in 1 Dimension

28. Newton-Raphson for Optimization
Want to find root of derivative: f'(z) = 0.
NR update rule then becomes (g = f’) z := z - f'(z)/f''(z).
You can use NR with step sizes, but the method doesn’t require it.
If the acceleration is high, slow down. Otherwise speed up.

29. Newton-Raphson Geometry
NR fits a quadratic function to the current location, then moves to the minimum of the quadratic.
For more discussion and a picture see Wikipedia

30. Optimization Problems for Local Search
As we get close to the minimum, a fixed step size can keep over/undershooting (oscillation).
See gradient descent example.
Simple solution: decrease step size with number of steps taken.
No explicit goal statement – we don’t know when minimum has been reached.
Simple solution: stop searching when “not enough” progress has been made in the “last few” iterations.
These are user-defined parameters.
Recall the vacuum robot that keeps cleaning. Gradient descent is memoryless.

31. Refinements Many more strategies are used.
Conjugate gradient descent: leave directions with 0 gradient alone.
Add probabilistic moves to avoid/escape from local minima (the fly/vacuum robot).
Random Restart.
Stochastic Gradient Descent.
Simulated Annealing (gradually reduce randomness).
Genetic Algorithms (mutations).
Try searching different locations (beam search).
Learn during searching (tabu search).
Example of beam search: auto correction on cell phone offers several possibilities, not just one

32. Summary
Greedy or Hill-Climbing Search: make local modifications to current state to find optimum.
Use gradient information to find search direction.
Main problem: get stuck in local optimum.
Other problem: can be slow in high-dimensional spaces.
Remember slow exploration of randomly moving robot.
2nd-order information speeds convergence.