CMPT 310 Simon Fraser University CHAPTER 4 Oliver Schulte Local Search CMPT 310 Simon Fraser University CHAPTER 4 Oliver Schulte Show example of gradient descent for neural nets.

Key Concepts Hill-Climbing Gradient Descent 2nd-order methods.

Game Theory and the PEAS Model

Games With the Environment Even if there is only one decision-maker agent A, the PEAS model can still be thought of as a game. Performance: the payoffs of the single “player” A. Environment: a “player” E who makes moves. Payoffs: The E player may be indifferent: always 0 payoff. Zero-sum: E player loses whatever A gains  Worst-Case Analysis. Actuators: moves by A. Sensors: Information about moves by E.

Example: Indifferent Environment Agent A chooses action. E does not care what happens. There is uncertainty: E could “choose” L or R. think about skytrain fare check. A might know or have learned a probability distribution for E’s choices. Environment E Agent A L R T 1,0 0,0 B

Example: Adversarial Environment Agent A chooses action. Zero-sum: E wants A to lose. To win, A needs to do worst-case analysis. Environment E Agent A L R T 1,-1 0,0 B

Deterministic Certain Environment A deterministic environment without uncertainty can be modelled by allowing E only a single action. Without uncertainty, choice is trivial for a small finite list of actions. Environment E Agent A L T 1,-1 B 0,0

Computational Choice Under Certainty But choice can be computationally hard if there are infinitely many actions. The payoffs may not be given as in a look-up table but as a function that needs to be analyzed. CMPT 310 - Blind Search

Environment Type Discussed In this Lecture Fully Observable Static Environment yes Deterministic yes Sequential no yes Discuss discrete variables a little bit. Discrete no Discrete yes no yes Continuous Function Optimization Planning, heuristic search Control, cybernetics Vector Search: Constraint Satisfaction CMPT 310 - Blind Search

Optimization Problems

Optimization Problems An optimization problem is of the form maximize f(x) subject to constraints on x where x is a vector of values. Equivalently minimize -f(x) subject to constraints on x If the constraints are linear (in)equalities, we have a linear programming problem. Very large literature built up over centuries.

AI examples Find the right angle of joints for pancake-flipping robots. Find the best weights for rules for reasoning. Any problem with continuous variables at some point needs optimization to build the best agent.

Hill Climbing

Multiple-Choice Question Hill-Climbing is a standard benchmark challenge for AI robots a favourite way for AI engineers to relax. like climbing Mount Everest in thick fog with amnesia a kind of tree search

Hill-climbing search Trivially complete because with probability 1 tries best solution. Random-restart hill climbing overcomes local maxima. Random Sideways move: escapes from shoulders, loops on flat maxima. Hill-Climbing Video

Hill-climbing search

Hill-climbing search: 8-queens problem (discrete) Put n queens on an n × n board with no two queens on the same row, column, or diagonal h is the min-conflicts function, see text. h = number of pairs of queens that are attacking each other, either directly or indirectly h = 17 for the above state

Hill-climbing search: 8-queens problem Demo for n-Queens Hill-Climbing aima-java.googlecode.com/svn/trunk/aima-all/release/aima-gui-NQueensApp.jnlp use complete state, not incremental A local minimum with h = 1

Gradient Descent

Gradient Descent: Choosing a direction. Intuition: think about trying to find street number 1000 on a block. You stop and see that you are at number 100. Which direction should you go, left or right? You initially check every 50 houses or so where you are. What happens when you get closer to the goal 1000? The fly and the window: the fly sees that the wall is darker, so the light gradient goes down: bad direction. Real life: bad relationship, job is local maximum.

Gradient Descent In Multiple Dimensions Basic Example More complicated examples. should have been covered in Math150 http://demo.activemath.org/ActiveMath2/main/applet.cmd;jsessionid=E1EF726B21EFBCDD2D27FC9070D7D543?itemId=mbase://OptimizationMarsu/opt_22_met_nais_grad_sp_th1/opt_22_met_nais_grad_sp_elab1&lang=en Active Math Applet http://www.onmyphd.com/?p=gradient.descent

Gradient Descent: Example. Try to find x,y that minimize f(x,y) = 3x + y2. Your current location is x = 10, y = -3. What is ? ? Answer: the gradient vector is (3, 2y). Evaluated at the location (10,-3), the gradient is = (3, -6). To minimize, we move in the opposite direction -. Letting the step size = 1, your new location is (10,-3) - (3,-6) = (7, 3).

Gradient Descent: Exercise Try to find x,y that minimize f(x,y) = 3x + y2. Your current location is x = 7, y = 3. The gradient vector is (3, 2y). Letting the step size = 1, what is your new location?. Excel Demo Answer: x = 4, y = 3 – 6 = -3.

Gradient Descent for Learning Neural Nets Problem: Given input observations, how can a brain rewire the connections among its neurons to remember the observations? Most common answer: gradient descent! Applied in deep learning, Baidu search, Skype translation. See AIspace demo.

Local Search With Second-Order Information Using acceleration, curvature

Better Approach: Newton-Raphson Uses information about second-order derivatives. Over 300 years old. Idea: Step size is inversely proportional to 2nd-order derivative. Update Rule: x := x - f'(x)/f''(x). http://en.wikipedia.org/wiki/File:NewtonIteration_Ani.gif Why does that make sense? Look at hill-climbing graph.

Newton-Raphson Example Update Rule: x := x - g(x)/g'(x). In our example with f(x,y) = 3x + y2 there is no curvature in the x-dimension, so we use NR only in the second dimension with You can use NR with step sizes, but the method doesn’t require it. If the acceleration is high, slow down. Otherwise speed up. Bold vector = (x,y) pair. So the new location is (10,-3) - (3, -6/2) = (7,0).

Derivation from Newton’s Method Problem: find a root x such that g(x) = 0. Use update rule. x := x - g(x)/g'(x). Geometry: fit a line (tangent) to g(x), move to intersection with x-axis. Demo in 1 Dimension http://en.wikipedia.org/wiki/File:NewtonIteration_Ani.gif

Newton-Raphson for Optimization Want to find root of derivative: f'(z) = 0. NR update rule then becomes (g = f’) z := z - f'(z)/f''(z). You can use NR with step sizes, but the method doesn’t require it. If the acceleration is high, slow down. Otherwise speed up.

Newton-Raphson Geometry NR fits a quadratic function to the current location, then moves to the minimum of the quadratic. For more discussion and a picture see Wikipedia

Optimization Problems for Local Search As we get close to the minimum, a fixed step size can keep over/undershooting (oscillation). See gradient descent example. Simple solution: decrease step size with number of steps taken. No explicit goal statement – we don’t know when minimum has been reached. Simple solution: stop searching when “not enough” progress has been made in the “last few” iterations. These are user-defined parameters. Recall the vacuum robot that keeps cleaning. Gradient descent is memoryless.

Refinements Many more strategies are used. Conjugate gradient descent: leave directions with 0 gradient alone. Add probabilistic moves to avoid/escape from local minima (the fly/vacuum robot). Random Restart. Stochastic Gradient Descent. Simulated Annealing (gradually reduce randomness). Genetic Algorithms (mutations). Try searching different locations (beam search). Learn during searching (tabu search). Example of beam search: auto correction on cell phone offers several possibilities, not just one

Summary Greedy or Hill-Climbing Search: make local modifications to current state to find optimum. Use gradient information to find search direction. Main problem: get stuck in local optimum. Other problem: can be slow in high-dimensional spaces. Remember slow exploration of randomly moving robot. 2nd-order information speeds convergence.