1. UMass Lowell Computer Science 91.503 Analysis of Algorithms Prof. Karen Daniels Spring, 2010 Lecture 2 Tuesday, 2/2/10 Design Patterns for Optimization Problems Greedy Algorithms

2. Algorithmic Paradigm Context Subproblem solution order Make choice, then solve subproblem(s) Solve subproblem(s), then make choice

3. Greedy Algorithms

4. What is a Greedy Algorithm? Solves an optimization problemSolves an optimization problem Optimal Substructure:Optimal Substructure: optimal solution contains in it optimal solutions to subproblemsoptimal solution contains in it optimal solutions to subproblems Greedy Strategy:Greedy Strategy: At each decision point, do what looks best “locally”At each decision point, do what looks best “locally” Choice does not depend on evaluating potential future choices or presolving overlapping subproblemsChoice does not depend on evaluating potential future choices or presolving overlapping subproblems Top-down algorithmic structureTop-down algorithmic structure With each step, reduce problem to a smaller problemWith each step, reduce problem to a smaller problem Greedy Choice Property:Greedy Choice Property: “locally best” = globally best“locally best” = globally best

5. A Greedy Strategy Approach 1. Determine the optimal substructure of the problem. 2. Develop a recursive solution. 3. Prove that, at any stage of the recursion, one of the optimal choices is the greedy choice. 4. Show that all but one of the subproblems caused by making the greedy choice are empty. 5. Develop a recursive greedy algorithm. 6. Convert it to an iterative algorithm. source: 91.503 textbook Cormen, et al. With experience, it is also possible to directly devise a (possibly iterative) greedy strategy.

6. Examples of Greedy Algorithms Activity Selection Activity Selection Minimum Spanning Tree Minimum Spanning Tree Dijkstra Shortest Path Dijkstra Shortest Path Huffman Codes Huffman Codes Fractional Knapsack Fractional Knapsack

7. Activity Selection

8. Activity Selection Optimization Problem Problem Instance:Problem Instance: Set S = {a 1, a 2,..., a n } of n activitiesSet S = {a 1, a 2,..., a n } of n activities Each activity a i has:Each activity a i has: start time: s istart time: s i finish time: f ifinish time: f i Activities i, j are compatible iff non-overlapping:Activities i, j are compatible iff non-overlapping: Objective:Objective: select a maximum-sized set of mutually compatible activitiesselect a maximum-sized set of mutually compatible activities source: 91.503 textbook Cormen, et al. and and

9. Activity Selection 12 3 4 5 6 7 8 9 10 12 11 13 1415 16 1 2 3 4 8 7 6 5 Activity Time Duration Activity Number

10. Algorithmic Progression “Brute-Force” “Brute-Force” Check all possible solutions Check all possible solutions Exponential number of subproblems! Exponential number of subproblems! Dynamic Programming Dynamic Programming Quadratic number of subproblems Quadratic number of subproblems Greedy Algorithm Greedy Algorithm Linear number of subproblems Linear number of subproblems

11. Activity Selection: Dynamic Programming Formulation Solution to S ij including a k produces 2 subproblems: 1) S ik (start after a i finishes; finish before a k starts) 2) S kj (start after a k finishes; finish before a j starts) source: 91.404 textbook Cormen, et al. c[i,j]=size of maximum-size subset of mutually compatible activities in S ij.

12. source: 91.503 textbook Cormen, et al. (3 rd edition) High-level call: REC-ACTIVITY-SELECTOR( s, f,0, n ) Returns an optimal solution for Activity Selection: Recursive Greedy Algorithm Activity Selection: Recursive Greedy Algorithm Only 1 subproblem to solve at each “level”.

13. source: web site accompanying 91.503 textbook Cormen, et al.

14. Running time? Activity Selection: Iterative Greedy Algorithm: Algorithm: Presort activities in S by non-decreasing finish time and renumber. Presort activities in S by non-decreasing finish time and renumber. source: 91.503 textbook Cormen, et al. source: 91.503 textbook Cormen, et al. (3 rd edition)

15. Streamlined Greedy Strategy Approach 1. View optimization problem as one in which making choice leaves one subproblem to solve. 2. Prove there always exists an optimal solution that makes the greedy choice. 3. Show that greedy choice + optimal solution to subproblem optimal solution to problem. source: 91.503 textbook Cormen, et al. Greedy Choice Property: “locally best” = globally best

16. Minimum Spanning Tree

17. A B C D E F G 2 2 1 1 3 4 4 5 6 6 8 7 source: 91.503 textbook Cormen et al. (Ch 23) for Undirected, Connected, Weighted Graph G=(V,E) Produces minimum weight tree of edges that includes every vertex. Invariant: Minimum weight spanning forest Becomes single tree at end Invariant: Minimum weight tree Spans all vertices at end Time: O(|E|lg|E|) given fast FIND-SET, UNION Time: O(|E|lg|V|) = O(|E|lg|E|) slightly faster with fast priority queue Where are the greedy choices made?

18. Dijkstra Shortest Path

19. Single Source Shortest Paths: Dijkstra’s Algorithm for (nonnegative) weighted, directed graph G=(V,E) A B C D E F G 2 2 1 1 3 4 4 5 6 6 8 7 source: 91.503 textbook Cormen et al. (Ch 24) We will revisit this algorithm when we study shortest paths.

20. Huffman Codes

21. source: web site accompanying 91.503 textbook Cormen, et al. Huffman Code Motivation Prefix Code: No code is a prefix of any other.

22. Huffman Code Tree Example source: 91.503 textbook Cormen, et al. Goal: Minimize number of bits required to encode a file. length of codeword for character c

23. source: web site accompanying 91.503 textbook Cormen, et al. Code Tree Comparison Fixed-length code: not optimal Huffman prefix code: optimal

24. source: web site accompanying 91.503 textbook Cormen, et al. Example of Huffman Code Steps

25. source: web site accompanying 91.503 textbook Cormen, et al. Huffman Code Greedy Algorithm Running Time?

26. source: web site accompanying 91.503 textbook Cormen, et al. Huffman Correctness: A Key Idea Lemma 16.2: Let C be an alphabet in which each character c in C has frequency c.freq. Let x and y be 2 characters in C with lowest frequencies. Then there exists an optimal prefix code for C in which codewords x and y have the same length and differ only in the last bit. ( T represents arbitrary optimal prefix code.) (Claim: T’’ also has optimal cost.)

27. Fractional Knapsack

28. Knapsack 50 10 20 30 item1item2item3 “knapsack” Value: $60 $100$120 Each item has value and weight. Goal: maximize total value of items chosen, subject to weight limit. 0-1: take all or none of an item fractional: can take part of an item

29. source: web site accompanying 91.503 textbook Cormen, et al.

30. Greedy Heuristic If optimization problem does not have “greedy choice property”, greedy approach may still be useful as a heuristic in bounding the optimal solution If optimization problem does not have “greedy choice property”, greedy approach may still be useful as a heuristic in bounding the optimal solution Example: minimization problem Example: minimization problem Optimal (unknown value) Upper Bound (heuristic) Lower Bound Solution Values