1. MAE 552 – Heuristic Optimization Lecture 3 January 28, 2002

2. Types of Optimization and Optimization Algorithms There are a many different types of optimization and even more algorithms to perform it. Why are there so many algorithms? None of the traditional algorithms are robust. For every different type of problem there is a different solution method. This can become a problem when a user only knows a limited number of algorithms and tries to apply them to problems they were not intended for.

3. Types of Optimization

4. Important Categories Methods can be classified according to: 1.The type of problem being addressed Convex vs. Multi-Modal Continuous vs. Discrete Constrained vs. Unconstrained Linear, Quadratic and Nonlinear Programming Analytical or Numerical

5. Important Categories Methods can be classified according to: 2.The type of technique being used Deterministic vs. Random Derivative-based vs. Derivative free Global vs. Local

6. Types of Problems: Convex versus Multimodal 1.Strictly Convex Problems have a single optima Local Optima = Global Optima Hessian matrix has positive eigenvalues F(x) x*x*

7. Types of Problems: Convex versus Multimodal 1.Non Convex Problems may have multiple local optima x F(x)

8. Types of Problems: Continuous vs. Discrete 1.In continuous problems x is defined over a range x l  x  x u Examples: length, weight, temperature etc. F(x) x

9. Types of Problems: Continuous vs. Discrete 1.In discrete problems x is defined only for specific values x l  x  x u where x can take on only specific values Examples: Material Type, routes taken, standardized sizing F(x) x

10. Types of Problems: Linear, Quadratic and Nonlinear F 1.Problems with Linear or Quadratic Objective Functions and constraints are much easier to solve than those with general nonlinear terms. Examples: Linear F=ax 1 + bx 2 + c Quadratic F=F=ax 1 + bx 2 + cx 2 2 +d Nonlinear F=ax 1 + bx 2 2 + cx 2 5 +x 1 x 2 1/2

11. Types of Problems:Analytical or Numerical Analytical Problems: Equations available to calculate objective function and constraints F(x)=ax 1 + bx 2 + c g 1 (x)=x 1 +x 2  0 Numerical Problems: Analysis codes used to determine objective function and constraints ANALYSIS x F(x) g(x)

12. Type of Technique : Deterministic vs. Random Deterministic Methods: With a given set of initial conditions these algorithms always follow the same path. XoXo Examples: Enumerative Search, Steepest Descent, GRG, Conjugate Gradient, Powell’s etc.

13. Type of Technique : Deterministic vs. Random Random Methods: With a given set of initial conditions these algorithms follow different paths with each run XoXo Examples: Scatter Search, Genetic Algorithms, Simulated Annealing

14. Type of Technique : Derivative-based/Derivative free Derivative Based Methods: First and Second Derivatives of the objective Function are calculated either numerically of analytically Examples: Steepest Descent, Newton’s Method, Conjugate Gradient, Varible Metric Method etc. Derivative Free Methods: Only F(X) information is used to guide the search Examples: Powell’s Search, Random Search, SA, GA, etc.

15. Type of Technique :Local or Global Optimizer Local Search Methods: Find best design within a portion of the design space usually a convex space (neighborhood). These methods include almost all of the traditional numerical techniques zero-order and derivative-based. Tend to get stuck in valleys of the design space F(x) x*x* Global Opt. Local Opt.

16. Type of Technique :Local or Global Optimizer Global Search Methods: Find best design within entire design space. Tend to be robust but inefficient and expensive to run. F(x) x*x* Global Opt. Local Opt.

17. Choosing the Right Method 1.It is very important to match the optimization method to the problem that you are solving. 2.Know the assumptions involved with each method. Example: LP requires linear objective and constraints. 3.Know the relative efficiency of each proposed method. Example: Using a Enumerative Search to solve a convex problem will be very inefficient. 4.Some problems are intractable!!! There are no methods that can guarantee a global optima in finite time. Use methods that find good solutions fast.

18. Basics of problem Solving Which algorithm is right for a problem is strongly dependent on how the problem is mathematically described. Three basic concepts common to every algorithmic approach to problem solving 1.A representation of the problem 2. The objective 3. The evaluation function

19. Basics of problem Solving- Representation The representation encodes alternative candidate solutions for manipulation and defines the size of the search space. For example in NLP typically the design variables are represented as real numbers in n dimensions. Is this the only possibility? In the TSP the representation we have used is a permutation of the natural numbers 1,2,3,….,n. The size of the search space is not determined by the problem, it is determined by your representation and the way that you handle this encoding.

20. Basics of problem Solving- Representation Your choice of representation will determine how you will be able to manipulate the potential solutions Alternative Representations for a NLP {x 1,x 2 }={5,6} {x 1,x 2 }={101,110} 0x 1 10 10 X 2 0

21. Basics of problem Solving-Objective The objective represents mathematically the task to be achieved For the TSP the objective is typically to minimize the distance covered by the salesman subject to the constraint of visiting each city once and only once and returning the starting city. min  dist (legal trip) For NLP an example might be: min F=x 1 +x 2 +x 3

22. Basics of problem Solving-Evaluation Function The evaluation function is typically a mapping from the space of possible candidate solutions to a set of numbers, where each element from the space of possible solutions is assigned a number to indicate its quality. The EF allows you to compare solutions in terms of quality. Ordinal EFs allow you to rank the candidate solutions. Current Candidate is the tenth best Numerical evaluations allow you to rank the solutions and also their degree of quality. Current Candidate is 1.3 units better that the second best candidate

23. Basics of problem Solving-Evaluation Function Since it could be computationally expensive to determine a numerical value for how good or bad a candidate it might be good enough to know approximately how good a solution is. Eval. Function Candidate 1 Candidate 2 Superior Candidate

24. Basics of problem Solving-Evaluation Function For every real world problem the evaluation problem is chosen by the designer. It should of course indicate for instance that a solution that meets the objective is better than one that does not. It should also depend on factors such as the computational complexity of the problem. Often the objective function indicates a good evaluation function. Objective - Minimize Stress -> Evaluation Function -Stress

25. Basics of problem Solving-Evaluation Function Other times you cannot derive a useful Evaluation Function from the objective: In the SAT problem the objective is to find a set of boolean (TRUE,FALSE) variables that satisfies a logical statement (makes it TRUE). All wrong candidate solutions return FALSE which does not tell you how to improve the solution.

26. Basics of problem Solving-Defining a Search Problem When you design an evaluation function you need to consider that for many problems the only solutions of interest are the subset that are Feasible (satisfy the constraints). The feasible space can be defined as F where F  S. A search problem can then be defined as: Given a search space S and its feasible part F  S find x  F such that eval(x)  eval(y) for all y  F Note that the objective does not appear at all in the formulation!! If your EF does not correspond with the objective you will searching for the answer to the wrong problem.

27. Basics of problem Solving-Defining a Search Problem A point x that satisfies the condition is called a global solution. Finding a global solution can be difficult and impossible to prove in some cases. It would be easier if we could limit the search to a smaller area of S. This fact underlies many search techniques.

28. Basics of problem Solving-Neighborhood Search If we concentrate on the area of S ‘near’ to some point in the search space we can more easily look in this ‘neighborhood’. S x N(x) N(x) of x is a set of all points in the search space that ‘close’ to the given point x. N(x) ={y  S: dist(x,y)  }

29. Basics of problem Solving-Neighborhood Search For a continuous NLP the Euclidean distance can be used to define a neighborhood. For the TSP a 2-swap neighborhood can be defined as all of the candidates that would result from swapping two cities in a given tour. A solution x (a permutation of n=5 cities) 1-2-3-4-5 has n(n-1)/2 neighbors including 1-3-2-4-5 (swapping cities 2 and 3) 5-2-3-4-1 (swapping cities 1 and 5) etc.

30. Basics of problem Solving-Neighborhood Search F=x 2 +3 xcxc Example: Quadratic Objective with no Constraints

31. Basics of problem Solving-Neighborhood Search Step 1: Define a neighborhood around point x c. N(x): x c -   x  x c +  Min F=x 2 +3 xcxc

32. Basics of problem Solving-Neighborhood Search Step 2: Sample a candidate solution from the neighborhood and evaluate it. if F(x 1 )>F(x c ) reject point and choose another. F=x 2 +3 xcxc x1x1

33. Basics of problem Solving-Neighborhood Search if F(x 1 )<F(x c ) accept point and choose replace current point x c with x 1. F=x 2 +3 xcxc x1x1

34. Basics of problem Solving-Neighborhood Search Step 3: Create new neighborhood around x c and repeat process. F=x 2 +3 xcxc

35. Basics of problem Solving-Neighborhood Search Most realistic problems are considerably more difficult than a quadratic bowl problem. The evaluation function defines a response surface that describes the topography of the search space with many hills and valleys.

36. Basics of problem Solving-Neighborhood Search Finding the best peak or the lowest valley is like trying to navigate a mountain range in the dark with only a small lamp. Your decisions must be made using local information. You can sample points in a local area and then decide where to walk next.

37. Basics of problem Solving-Neighborhood Search If you decide to always go uphill then you will reach a peak but not necessarily the highest peak. You may need to walk downhill in order to eventually reach the highest peak in the space.