1. Heuristic Optimization Methods Prologue
Chin-Shiuh Shieh
C.-S. Shieh, EC, KUAS, Taiwan

2. C.-S. Shieh, EC, KUAS, Taiwan
Course Information
課程名稱: 啟發式最佳化方法
Heuristic Optimization Methods
授課教師: 謝欽旭
上課時間: (三)6-8
上課教室: 資405
課程網站
C.-S. Shieh, EC, KUAS, Taiwan

3. Course Information (cont)
Objective
The study of heuristic optimization methods
Course Outline
Introduction to Optimization
Calculus, Optimization, and Search
Linear Programming, Combinatorial Optimization
Heuristic Approaches to Optimization
Genetic Algorithm
Ant Colony System
Simulated Annealing
Particle Swarm Optimization …
C.-S. Shieh, EC, KUAS, Taiwan

4. Course Information (cont)
Readings
Kwang Y. Lee and Mohamed A. El-Sharkawi, Eds., Modern Heuristic Optimization Techniques, John Wiley & Sons, 2008.
Johann Dr´eo, Patrick Siarry, Alain P´etrowski, and Eric Taillard, Metaheuristics for Hard Optimization, Springer, 2006.
Xin-She Yang, Nature-Inspired Optimization Algorithms, Elsevier, 2014
C.-S. Shieh, EC, KUAS, Taiwan

5. C.-S. Shieh, EC, KUAS, Taiwan
What Is Optimization?
Optimization defined in Wikipedia
In mathematics, statistics, empirical sciences, computer science, or management science, mathematical optimization (alternatively, optimization or mathematical programming) is the selection of a best element (with regard to some criteria) from some set of available alternatives.
C.-S. Shieh, EC, KUAS, Taiwan

6. What Is Optimization? (cont)
A mathematical formulation
Given: a function f : AR from some set A to the real numbers
Sought: an element x0 in A such that f(x0) ≤ f(x) for all x in A ("minimization") or such that f(x0) ≥ f(x) for all x in A ("maximization").
C.-S. Shieh, EC, KUAS, Taiwan

7. C.-S. Shieh, EC, KUAS, Taiwan
Challenges
Object function
High dimensionalities
Local optimum
Huge search space
C.-S. Shieh, EC, KUAS, Taiwan

8. Example Optimization Problems
Minimize the costs of shipping from production facilities to warehouses
Maximize the probability of detecting an incoming warhead (vs. decoy) in a missile defense system
Place sensors in manner to maximize useful information
Determine the times to administer a sequence of drugs for maximum therapeutic effect
Find the best red-yellow-green signal timings in an urban traffic network
Determine the best schedule for use of laboratory facilities to serve an organization’s overall interests
C.-S. Shieh, EC, KUAS, Taiwan

9. C.-S. Shieh, EC, KUAS, Taiwan
Major Subfields
Linear Programming, a type of convex programming, studies the case in which the objective function f is linear and the set of constraints is specified using only linear equalities and inequalities.
Nonlinear Programming studies the general case in which the objective function or the constraints or both contain nonlinear parts.
C.-S. Shieh, EC, KUAS, Taiwan

10. Major Subfields (cont)
Integer Programming studies linear programs in which some or all variables are constrained to take on integer values. This is not convex, and in general much more difficult than regular linear programming.
Stochastic Programming studies the case in which some of the constraints or parameters depend on random variables.
C.-S. Shieh, EC, KUAS, Taiwan

11. Major Subfields (cont)
Calculus of Variations seeks to optimize an objective defined over many points in time, by considering how the objective function changes if there is a small change in the choice path.
Combinatorial Optimization is concerned with problems where the set of feasible solutions is discrete or can be reduced to a discrete one.
C.-S. Shieh, EC, KUAS, Taiwan

12. Major Subfields (cont)
Heuristics and Meta-heuristics make few or no assumptions about the problem being optimized. Usually, heuristics do not guarantee that any optimal solution need be found. On the other hand, heuristics are used to find approximate solutions for many complicated optimization problems.
C.-S. Shieh, EC, KUAS, Taiwan

13. C.-S. Shieh, EC, KUAS, Taiwan
Optimization Methods
(Exact) Algorithms
An algorithm is sometimes described as a set of instructions that will result in the solution to a problem when followed correctly.
Unless otherwise stated, an algorithm is assumed to give the optimal solution to an optimization problem.
That is, not just a good solution, but the best solution.
C.-S. Shieh, EC, KUAS, Taiwan

14. Optimization Methods (cont)
Approximation Algorithms
Approximation algorithms (as opposed to exact algorithms) do not guarantee to find the optimal solution.
However, there is a bound on the quality, e.g., for a maximization problem, the algorithm can guarantee to find a solution whose value is at least half that of the optimal value.
We will not see many approximation algorithms here, but mention them as a contrast to heuristic algorithms.
C.-S. Shieh, EC, KUAS, Taiwan

15. Optimization Methods (cont)
Heuristic Algorithms
Heuristic algorithms do not guarantee to find the optimal solution.
Heuristic algorithms do not even necessarily have a bound on how bad they can perform.
However, in practice, heuristic algorithms (heuristics for short) have proven successful.
Near optimal solutions in reasonable time.
Most of this course will focus on this type of heuristic solution method.
C.-S. Shieh, EC, KUAS, Taiwan

16. C.-S. Shieh, EC, KUAS, Taiwan
What Is Heuristic?
Heuristic defined in Wikipedia
Heuristic (Greek: “Εὑρίσκω", "find" or "discover") refers to experience-based techniques for problem solving, learning, and discovery. Where the exhaustive search is impractical, heuristic methods are used to speed up the process of finding a satisfactory solution; mental short cuts to ease the cognitive load of making a decision. 16
C.-S. Shieh, EC, KUAS, Taiwan 16

17. Why Not Always Exact Methods?
The running time of the algorithm
For reasons explained soon, the running time of an algorithm may render it useless on the problem you want to solve.
The link between the real-world problem and the formal problem is weak
Sometimes you cannot properly formulate a COP/IP that captures all aspects of the real-world problem. If the problem you solve is not the right problem, it might be just as useful to have one (or more) heuristic solutions, rather than the optimal solution of the formal problem. 17
C.-S. Shieh, EC, KUAS, Taiwan 17

18. Heuristic Optimization Methods
Random Walk
Hill-climbing
Genetic Algorithms
Particle Swarm Optimization
Ant Colony Optimization
Simulated Annealing
Tabu Search
Memetic Algorithm
C.-S. Shieh, EC, KUAS, Taiwan

19. C.-S. Shieh, EC, KUAS, Taiwan
A Challenge
Maximize F1(x,y)
C.-S. Shieh, EC, KUAS, Taiwan