Mathematical Modeling and Optimization: Summary of “Big Ideas”

A schematic view of modeling/optimization process assumptions, abstraction,data,simplifications Real-world problem Mathematical model makes sense? change the model, assumptions? optimization algorithm Solution to real-world problem Solution to model interpretation

Mathematical models in Optimization The general form of an optimization model: min or max f(x1,…,xn) (objective function) subject to gi(x1,…,xn) ≥ 0 (functional constraints) x1,…,xn  S (set constraints) x1,…,xn are called decision variables In words, the goal is to find x1,…,xn that satisfy the constraints; achieve min (max) objective function value.

Types of Optimization Models Stochastic (probabilistic information on data) Deterministic (data are certain) Discrete, Integer (S = Zn) Continuous (S = Rn) Linear (f and g are linear) Nonlinear (f and g are nonlinear)

What is Discrete Optimization? is a field of applied mathematics, combining techniques from combinatorics and graph theory, linear programming, theory of algorithms, to solve optimization problems over discrete structures.

Solution Methods for Discrete Optimization Problems Integer Programming Network Algorithms Dynamic Programming Approximation Algorithms

Integer Programming Programming = Planning in this context In a survey of Fortune 500 firms, 85% of those responding said that they had used linear or integer programming. Why is it so popular? Many different real-life situations can be modeled as IPs. There are efficient algorithms to solve IPs.

Topics in this class about Integer Programming Modeling real-life situations as integer programs Applications of integer programming Solution methods (algorithms) for integer programs (maybe) Using software (called AMPL) to solve integer programs

IP modeling techniques Using binary variables Restrictions on number of options Contingent decisions Variables (functions) with k possible values Either-Or Constraints Big M method Balance constraints Fixed Charges Making choices with non-binary variables Piecewise linear functions

IP applications Facility Location Problem Knapsack Problem Multi-period production planning Inventory management Fair representation in electoral systems Consultant hiring Bin Packing Problem Pairing Problem Traveling Salesman Problem

Difficulties of real-life modeling The problems that you will encounter in the real world are always a lot messier than the ``clean'' problems we looked at in this class; there are always side constraints that complicate getting even a feasible solution. Most real world problems have multiple objectives, and it is hard to choose from among them. In the real world you must gain experience with how to adapt the idealized models of academia to each new problem you are asked to solve.

Utilizing the relationship between problems Important modeling skill: Suppose you know how to model Problems A1,…,Ap; You need to solve Problem B; Notice the similarities between Problems Ai and B; Build a model for Problem B, using the model for Problem Ai as a prototype. Examples: The version of the facility location problem as a special case of the knapsack problem Transshipment and Assignment problems as special cases of Maximum Flow Problem

Complexity of Solving Discrete Optimization Problems Two classes: Class 1 problems have polynomial-time algorithms for solving the problems optimally. Examples: Min. Spanning Tree Problem, Maximum Flow Problem For Class 2 problems (NP-hard problems) No polynomial-time algorithm is known; And more likely there is no one. Example: Traveling Salesman Problem Most discrete optimization problems are in the second class.

Three main directions to solve NP-hard discrete optimization problems: Integer programming techniques Approximation algorithms Heuristics Important Observation: Any solution method suggests a tradeoff between time and accuracy. On time-accuracy tradeoff schedule: Integer programming Approximation algorithms Heuristics Brute force Most accuracy Least accuracy Worst time Best time

Solving Integer Programs (IP) vs solving Linear Programs (LP) LP algorithms Simplex Method Interior-point methods IP algorithms use the above-mentioned LP algorithms as subroutines. The algorithms for solving LPs are much more time-efficient than the algorithms for IPs. Important modeling consideration: Whenever possible avoid integer variables in your model.

LP-relaxation-based solution methods for Integer Programs Branch-and-Bound Technique Cutting Plane Algorithms

Basic Concepts of Branch-and-Bound The basic concept underlying the branch-and-bound technique is to divide and conquer. Since the original “large” problem is hard to solve directly, it is divided into smaller and smaller subproblems until these subproblems can be conquered. The dividing (branching) is done by partitioning the entire set of feasible solutions into smaller and smaller subsets. The conquering (fathoming) is done partially by (i) giving a bound for the best solution in the subset; (ii) discarding the subset if the bound indicates that it can’t contain an optimal solution.

Summary of branch-and-bound Steps for each iteration: Branching: Among the unfathomed subproblems, select the one that was created most recently. (Break ties according to which has larger LP value.) Choose a variable xi which has a noninteger value xi* in the LP solution of the subproblem. Create two new subproblems by adding the respective constraints xi   xi* and xi ≥  xi* . Bounding: Solve the new subproblems, record their LP solutions. Based on the LP values, update the incumbent, and the lower and upper bounds for OPT(IP) if necessary. Fathoming: For each new subproblem, apply the three fathoming tests. Discard the subproblems that are fathomed. Optimality test: If there are no unfathomed subproblems left then return the current incumbent as optimal solution (if there is no incumbent then IP is infeasible.) Otherwise, perform another iteration.

Importance of tight lower and upper bounds in branch-and-bound Having tight lower and upper bounds on the IP optimal value might significantly reduce the number of branch-and-bound iterations. For maximization problem, A lower bound can be found by applying a fast heuristic algorithm to the problem. An upper bound can be found by solving a relaxation of the problem (e.g., LP-relaxation). If the current lower and upper bounds are close enough, we can stop the branch-and-bound algorithm and return the current incumbent solution (it can’t be too far from the optimum).

General Idea of Cutting Plane Technique Add new constraints (cutting planes) to the problem such that (i) the set of feasible integer solutions remains the same, i.e., we still have the same integer program. (ii) the new constraints cut off some of the fractional solutions making the feasible region of the LP-relaxation smaller. Smaller feasible region might result in a better LP value (i.e., closer to the IP value), thus making the search for the optimal IP solution more efficient. Each integer program might have many different formulations. Important modeling skill: Give as tight formulation as possible. How? Find cutting planes that make the formulation of the original IP tighter.

Methods of getting Cutting Planes Exploit the special structure of the problem to get cutting planes (e.g., bin packing problem, pairing problem) Often can be hard to get Topic of intensive research More general methods are also available Can be used automatically for many problems (e.g., knapsack-type constraints)

Branch-and-cut algorithms Integer programs are rarely solved based solely on cutting plane method. More often cutting planes and branch-and-bound are combined to provide a powerful algorithmic approach for solving integer programs. Cutting planes are added to the subproblems created in branch-and-bound to achieve tighter bounds and thus to accelerate the solution process. This kind of methods are known as branch-and-cut algorithms.

Network Models Minimum Spanning Tree Problem Traveling Salesman Problem Min. Spanning Tree is in Class 1 (has polynomial-time algorithms for solving the problem optimally). Traveling Salesman Problem is in Class 2 (NP-hard problem).

Methods for solving NP-hard problems Three main directions to solve NP-hard discrete optimization problems: Integer programming techniques Heuristics Approximation algorithms We gave examples of all three methods for TSP. 2-approximation algorithm for TSP was given and analyzed in details.

Some Characteristics of Approximation Algorithms Time-efficient (sometimes not as efficient as heuristics) Don’t guarantee optimal solution Guarantee good solution within some factor of the optimum Rigorous mathematical analysis to prove the approximation guarantee Often use algorithms for related problems as subroutines The 2-approximation algorithm for TSP used the algorithm of finding a minimum spanning tree as subroutine.

Performance of TSP algorithms in practice A more sophisticated algorithm (which again uses the MST algorithm as a subroutine) guarantees a solution within factor of 1.5 of the optimum (Christofides). For many discrete optimization problems, there are benchmarks of instances on which algorithms are tested. For TSP, such a benchmark is TSPLIB. On TSPLIB instances, the Christofides’ algorithm outputs solutions which are on average 1.09 times the optimum. For comparison, the nearest neighbor algorithm outputs solutions which are on average 1.26 times the optimum. A good approximation factor often leads to good performance in practice.

Dynamic Programming Dynamic programming is a widely-used mathematical technique for solving problems that can be divided into stages and where decisions are required in each stage. The goal of dynamic programming is to find a combination of decisions that optimizes a certain amount associated with a system.

General characteristics of Dynamic Programming The problem structure is divided into stages Each stage has a number of states associated with it Making decisions at one stage transforms one state of the current stage into a state in the next stage. Given the current state, the optimal decision for each of the remaining states does not depend on the previous states or decisions. This is known as the principle of optimality for dynamic programming. The principle of optimality allows to solve the problem stage by stage recursively.

Problems that can be solved by Dynamic Programming Shortest path problems Multi-period production planning Resource allocation problems

Advantages of Dynamic Programming More time-efficient compared to integer programming (but large-scale real-life problems might require lots of states) Can solve a variety of problems (however integer programming is a more universal technique) Can handle stochastic data (for example, stochastic demand in multi-period production planning)