Heuristic Optimization Methods Prologue Chin-Shiuh Shieh C.-S. Shieh, EC, KUAS, Taiwan

C.-S. Shieh, EC, KUAS, Taiwan Course Information 課程名稱: 啟發式最佳化方法 Heuristic Optimization Methods 授課教師: 謝欽旭 上課時間: (三)6-8 上課教室: 資405 課程網站 http://bit.nkust.edu.tw/~csshieh C.-S. Shieh, EC, KUAS, Taiwan

Course Information (cont) Objective The study of heuristic optimization methods Course Outline Introduction to Optimization Calculus, Optimization, and Search Linear Programming, Combinatorial Optimization Heuristic Approaches to Optimization Genetic Algorithm Ant Colony System Simulated Annealing Particle Swarm Optimization … C.-S. Shieh, EC, KUAS, Taiwan

Course Information (cont) Readings Kwang Y. Lee and Mohamed A. El-Sharkawi, Eds., Modern Heuristic Optimization Techniques, John Wiley & Sons, 2008. Johann Dr´eo, Patrick Siarry, Alain P´etrowski, and Eric Taillard, Metaheuristics for Hard Optimization, Springer, 2006. Xin-She Yang, Nature-Inspired Optimization Algorithms, Elsevier, 2014 C.-S. Shieh, EC, KUAS, Taiwan

C.-S. Shieh, EC, KUAS, Taiwan What Is Optimization? Optimization defined in Wikipedia In mathematics, statistics, empirical sciences, computer science, or management science, mathematical optimization (alternatively, optimization or mathematical programming) is the selection of a best element (with regard to some criteria) from some set of available alternatives. C.-S. Shieh, EC, KUAS, Taiwan

What Is Optimization? (cont) A mathematical formulation Given: a function f : AR from some set A to the real numbers Sought: an element x0 in A such that f(x0) ≤ f(x) for all x in A ("minimization") or such that f(x0) ≥ f(x) for all x in A ("maximization"). C.-S. Shieh, EC, KUAS, Taiwan

C.-S. Shieh, EC, KUAS, Taiwan Challenges Object function High dimensionalities Local optimum Huge search space C.-S. Shieh, EC, KUAS, Taiwan

Example Optimization Problems Minimize the costs of shipping from production facilities to warehouses Maximize the probability of detecting an incoming warhead (vs. decoy) in a missile defense system Place sensors in manner to maximize useful information Determine the times to administer a sequence of drugs for maximum therapeutic effect Find the best red-yellow-green signal timings in an urban traffic network Determine the best schedule for use of laboratory facilities to serve an organization’s overall interests C.-S. Shieh, EC, KUAS, Taiwan

C.-S. Shieh, EC, KUAS, Taiwan Major Subfields Linear Programming, a type of convex programming, studies the case in which the objective function f is linear and the set of constraints is specified using only linear equalities and inequalities. Nonlinear Programming studies the general case in which the objective function or the constraints or both contain nonlinear parts. C.-S. Shieh, EC, KUAS, Taiwan

Major Subfields (cont) Integer Programming studies linear programs in which some or all variables are constrained to take on integer values. This is not convex, and in general much more difficult than regular linear programming. Stochastic Programming studies the case in which some of the constraints or parameters depend on random variables. C.-S. Shieh, EC, KUAS, Taiwan

Major Subfields (cont) Calculus of Variations seeks to optimize an objective defined over many points in time, by considering how the objective function changes if there is a small change in the choice path. Combinatorial Optimization is concerned with problems where the set of feasible solutions is discrete or can be reduced to a discrete one. C.-S. Shieh, EC, KUAS, Taiwan

Major Subfields (cont) Heuristics and Meta-heuristics make few or no assumptions about the problem being optimized. Usually, heuristics do not guarantee that any optimal solution need be found. On the other hand, heuristics are used to find approximate solutions for many complicated optimization problems. C.-S. Shieh, EC, KUAS, Taiwan

C.-S. Shieh, EC, KUAS, Taiwan Optimization Methods (Exact) Algorithms An algorithm is sometimes described as a set of instructions that will result in the solution to a problem when followed correctly. Unless otherwise stated, an algorithm is assumed to give the optimal solution to an optimization problem. That is, not just a good solution, but the best solution. C.-S. Shieh, EC, KUAS, Taiwan

Optimization Methods (cont) Approximation Algorithms Approximation algorithms (as opposed to exact algorithms) do not guarantee to find the optimal solution. However, there is a bound on the quality, e.g., for a maximization problem, the algorithm can guarantee to find a solution whose value is at least half that of the optimal value. We will not see many approximation algorithms here, but mention them as a contrast to heuristic algorithms. C.-S. Shieh, EC, KUAS, Taiwan

Optimization Methods (cont) Heuristic Algorithms Heuristic algorithms do not guarantee to find the optimal solution. Heuristic algorithms do not even necessarily have a bound on how bad they can perform. However, in practice, heuristic algorithms (heuristics for short) have proven successful. Near optimal solutions in reasonable time. Most of this course will focus on this type of heuristic solution method. C.-S. Shieh, EC, KUAS, Taiwan

C.-S. Shieh, EC, KUAS, Taiwan What Is Heuristic? Heuristic defined in Wikipedia Heuristic (Greek: “Εὑρίσκω", "find" or "discover") refers to experience-based techniques for problem solving, learning, and discovery. Where the exhaustive search is impractical, heuristic methods are used to speed up the process of finding a satisfactory solution; mental short cuts to ease the cognitive load of making a decision. 16 C.-S. Shieh, EC, KUAS, Taiwan 16

Why Not Always Exact Methods? The running time of the algorithm For reasons explained soon, the running time of an algorithm may render it useless on the problem you want to solve. The link between the real-world problem and the formal problem is weak Sometimes you cannot properly formulate a COP/IP that captures all aspects of the real-world problem. If the problem you solve is not the right problem, it might be just as useful to have one (or more) heuristic solutions, rather than the optimal solution of the formal problem. 17 C.-S. Shieh, EC, KUAS, Taiwan 17

Heuristic Optimization Methods Random Walk Hill-climbing Genetic Algorithms Particle Swarm Optimization Ant Colony Optimization Simulated Annealing Tabu Search Memetic Algorithm C.-S. Shieh, EC, KUAS, Taiwan

C.-S. Shieh, EC, KUAS, Taiwan A Challenge Maximize F1(x,y) C.-S. Shieh, EC, KUAS, Taiwan