Dr. Shahriar Bijani Shahed University Feb. 2014

 Kleinberg and Tardos, Algorithm Design, CSE 5311, M Kumar, Spring  Chapter 4, Computer Algorithms, Horowitz,…,  Chapter 16, Introduction to Algorithms, CLRS, Algorithms Design, Greedy Algorithms

3  The problem: We are required to find a feasible solution that either maximizes or minimizes a given objective solution.  It is easy to determine a feasible solution but not necessarily an optimal solution.  The greedy method solves this problem in stages, at each stage, a decision is made considering inputs in an order determined by the selection procedure which may be based on an optimization measure. Algorithms Design, Greedy Algorithms

4  An optimization problem is one in which you want to find, not just a solution, but the best solution  A “greedy algorithm” sometimes works well for optimization problems  A greedy algorithm works in phases. At each phase: ◦ You take the best you can get right now, without regard for future consequences ◦ You hope that by choosing a local optimum at each step, you will end up at a global optimum Algorithms Design, Greedy Algorithms

5  Suppose you want to count out a certain amount of money, using the fewest possible bills and coins  A greedy algorithm would do this would be: At each step, take the largest possible bill or coin that does not overshoot ◦ Example: To make $6.39, you can choose:  a $5 bill  a $1 bill, to make $6  a 25¢ coin, to make $6.25  A 10¢ coin, to make $6.35  four 1¢ coins, to make $6.39  For US money, the greedy algorithm always gives the optimum solution Algorithms Design, Greedy Algorithms

6  In some (fictional) monetary system, “Rial” come in 1 Rial, 7 Rial, and 10 Rial coins  Using a greedy algorithm to count out 15 Rials, you would get ◦ A 10 Rial piece ◦ Five 1 Rial pieces, for a total of 15 Rials ◦ This requires six coins  A better solution would be to use two 7 Rial pieces and one 1 Rial piece ◦ This only requires three coins  This greedy algorithm results in a solution, but not in an optimal solution Algorithms Design, Greedy Algorithms

7  You have to run nine jobs, with running times of 3, 5, 6, 10, 11, 14, 15, 18, and 20 minutes  You have three processors on which you can run these jobs  You decide to do the longest-running jobs first, on whatever processor is available P1 P2 P3  Time to completion: = 35 minutes  This solution isn’t bad, but we might be able to do better Algorithms Design, Greedy Algorithms

8  What would be the result if you ran the shortest job first?  Again, the running times are 3, 5, 6, 10, 11, 14, 15, 18, and 20 minutes  That wasn’t such a good idea; time to completion is now = 40 minutes  Note, however, that the greedy algorithm itself is fast ◦ All we had to do at each stage was pick the minimum or maximum P1 P2 P3 Algorithms Design, Greedy Algorithms

9  This solution is clearly optimal (why?)  Clearly, there are other optimal solutions (why?)  How do we find such a solution? ◦ One way: Try all possible assignments of jobs to processors ◦ Unfortunately, this approach can take exponential time  Better solutions do exist: P1 P2 P3 Algorithms Design, Greedy Algorithms

10  The Huffman encoding algorithm is a greedy algorithm  You always pick the two smallest numbers to combine  Average bits/char: 0.22* * * * * *4 = 2.42  The Huffman algorithm finds an optimal solution A B C D E F A=00 B=100 C=01 D=1010 E=11 F=1011 Algorithms Design, Greedy Algorithms

5/16/2015 CSE 5311 Spring 2007 M Kumar11 Procedure Huffman_Encoding(S,f); Input : S (a string of characters) and f (an array of frequencies). Output : T (the Huffman tree for S) 1.insert all characters into a heap H according to their frequencies; 2.while H is not empty do 3. if H contains only one character x then 4.x  root (T); 5. else 6.z  ALLOCATE_NODE(); 7.x  left[T,z]  EXTRACT_MIN(H); 8.y  right[T,z]  EXTRACT_MIN(H); 9.f z  f x + f y ; 10.INSERT(H,z);

5/16/2015 CSE 5311 Spring 2007 M Kumar12 Complexity of Huffman algorithm Building a heap in step 1 takes O(n) time Insertions (steps 7 and 8) and deletions (step 10) on H take O (log n) time each Therefore Steps 2 through 10 take O(n logn) time Thus the overall complexity of the algorithm is O( n logn ).

 Assume a thief is robbing a store with various valued and weighted objects. Her bag can hold only W pounds.  The 0-1 knapsack problem: must take all or nothing  The fractional knapsack problem: can take all or part of the item Algorithms Design, Greedy Algorithms13

=$220 =$160 =$180 =$240 $60 $100 $ knapsack

 Given n items of sizes w 1, w 2, …, w n and values p 1, p 2, …, p n and size of knapsack of C, the problem is to find x 1, x 2, …, x n  that maximize subject to

 Consider y i = p i / w i  What is y i ?  What is the solution?  fractional problem has greedy-choice property, 0-1 does not.

17 Activity On Edge (AOE) Networks:  Tasks (activities) : a0, a1, …  Events : v0,v1, … V0 V1 V2 V3 V4 V6 V7 V8 V5 finish a0 = 6 start a1 = 4 a2 = 5 a4 = 1 a3 = 1 a5 = 2 a6 = 9 a7 = 7 a8 = 4 a10 = 4 a9 = 2 Some definition: Predecessor Successor Immediate predecessor Immediate successor

18  A critical path is a path that has the longest length. (v0, v1, v4, v7, v8) V0 V1 V2 V3 V4 V6 V7 V8 V5 a0 = 6 a1 = 4 a2 = 5 a4 = 1 a3 = 1 a5 = 2 a6 = 9 a7 = 7 a8 = 4 a10 = 4 a9 = 2 startfinish = 18 (Max)

19  The earliest time of an activity, a i, can occur is the length of the longest path from the start vertex v 0 to a i ’ s start vertex. （ Ex: the earliest time of activity a 7 can occur is 7. ）  We denote this time as early(i) for activity a i. ∴ early(6) = early(7) = 7. V0 V1 V2 V3 V4 V6 V7 V8 V5 finish a0 = 6 start a1 = 4 a2 = 5 a4 = 1 a3 = 1 a5 = 2 a6 = 9 a7 = 7 a8 = 4 a10 = 4 a9 = 2 6/? 0/? 7/? 16/? 0/? 5/? 7/? 14/?7/? 4/? 0/? 18

20  The latest time, late(i), of activity, a i, is defined to be the latest time the activity may start without increasing the project duration.  Ex: early(5) = 5 & late(5) = 8; early(7) = 7 & late(7) = 7 V0 V1 V2 V3 V4 V6 V7 V8 V5 finish a0 = 6 start a1 = 4 a2 = 5 a4 = 1 a3 = 1 a5 = 2 a6 = 9 a7 = 7 a8 = 4 a10 = 4 a9 = 2 late(5) = 18 – 4 – = 8 late(7) = 18 – 4 – 7 = 7 6/6 0/1 7/7 16/16 0/3 5/8 7/10 14/147/7 4/5 0/0

21  A critical activity is an activity for which early(i) = late(i).  The difference between late(i) and early(i) is a measure of how critical an activity is. Calculation of Latest Times Calculation of Earliest Times Finding Critical path(s) To solve AOE Problem

 Let activity a i is represented by edge (u, v). ◦ early (i) = earliest [u] ◦ late (i) = latest [v] – duration of activity a i  We compute the times in two stages: a forward stage and a backward stage.  The forward stage: ◦ Step 1: earliest [0] = 0 ◦ Step 2: earliest [j] = max {earliest [i] + duration of (i, j)} i is in P(j) P(j) is the set of immediate predecessors of j.

23  The backward stage: ◦ Step 1: latest[n-1] = earliest[n-1] ◦ Step 2: latest [j] = min {latest [i] - duration of (j, i)} i is in S(j) S(j) is the set of vertices adjacent from vertex j. latest[8] = earliest[8] = 18 latest[6] = min{earliest[8] - 2} = 16 latest[7] = min{earliest[8] - 4} = 14 latest[4] = min{earliest[6] – 9; earliest[7] – 7} = 7 latest[1] = min{earliest[4] - 1} = 6 latest[2] = min{earliest[4] - 1} = 6 latest[5] = min{earliest[7] - 4} = 10 latest[3] = min{earliest[5] - 2} = 8 latest[0] = min{earliest[1] – 6; earliest[2] – 4; earliest[3] – 5} = 0

24 V0 V1 V2 V3 V4 V6 V7 V8 V5 finish a0 start a1 a2 a4 a3 a5 a6 a7 a8 a10 a9 V0 V1 V4 V6 V7 V8 finish a0 start a3 a6 a7a10 a9 ActivityEarlyLateL - ECritical a0a0 000Yes a1a1 022No a2a2 033 a3a3 660Yes a4a4 462No a5a5 583 a6a6 770Yes a7a7 770 a8a8 7103No a9a9 16 0Yes a Yes

 The longest path(critical path) problem can be solved by the critical path method(CPM) : Step 1:Find a topological ordering. Step 2: Find the critical path.

26  A salesman must visit every city (starting from city A), and wants to cover the least possible distance ◦ He can revisit a city (and reuse a road) if necessary  He does this by using a greedy algorithm: He goes to the next nearest city from wherever he is  From A he goes to B  From B he goes to D  This is not going to result in a shortest path!  The best result he can get now will be ABDBCE, at a cost of 16  An actual least-cost path from A is ADBCE, at a cost of 14 E ABC D Algorithms Design, Greedy Algorithms

27  A greedy algorithm typically makes (approximately) n choices for a problem of size n ◦ (The first or last choice may be forced)  Hence the expected running time is: O(n * O(choice(n))), where choice(n) is making a choice among n objects ◦ Counting: Must find largest useable coin from among k sizes of coin ( k is a constant), an O(k)=O(1) operation;  Therefore, coin counting is (n) ◦ Huffman: Must sort n values before making n choices  Therefore, Huffman is O(n log n) + O(n) = O(n log n) Algorithms Design, Greedy Algorithms

28  Dijkstra’s algorithm for finding the shortest path in a graph (single source) ◦ Always takes the shortest edge connecting a known node to an unknown node  Prim’s algorithm for finding a minimum-cost spanning tree ◦ Always takes the lowest-cost edge between nodes in the spanning tree and nodes not yet in the spanning tree  Kruskal’s algorithm for finding a minimum- cost spanning tree ◦ Always tries the lowest-cost remaining edge

Algorithms Design, Greedy Algorithms29  In a shortest-paths problem, we are given a weighted, directed graph G = (V,E), with weights assigned to each edge in the graph. The weight of the path p = (v0, v1, v2, …, vk) is the sum of the weights of its edges:  v0  v1  v2...  vk-1  vk  The shortest-path from u to v is given by d(u,v) = min {weight (p)} : if there are one or more paths from u to v =  otherwise

Algorithms Design, Greedy Algorithms30 Given G (V,E), find the shortest path from a given vertex u  V to every vertex v  V ( u  v). For each vertex v  V in the weighted directed graph, d[v] represents the distance from u to v. Initially, d[v] = 0 when u = v. d[v] =  if (u,v) is not an edge d[v] = weight of edge (u,v) if (u,v) exists. Dijkstra's Algorithm : At every step of the algorithm, we compute, d[y] = min {d[y], d[x] + w(x,y)}, where x,y  V. Dijkstra's algorithm is based on the greedy principle because at every step we pick the path of least weight.

Algorithms Design, Greedy Algorithms31

Algorithms Design, Greedy Algorithms32 Step # Vertex to be marked Distance to vertexUnmarked vertices uabcdefgh 0u015  9  a,b,c,d,e,f,g,h 1a01539  b,c,d,e,f,g,h 2c01537  12  b,d,e,f,g,h 3b  d,e,f,g,h 4d  e,f,g,h 5e f,g,h 6h g,h 7g h 8f

Algorithms Design, Greedy Algorithms33 Procedure Dijkstra's Single-source shortest path_G(V,E,u) Input: G =(V,E), the weighted directed graph and v the source vertex Output: for each vertex, v, d[v] is the length of the shortest path from u to v. 1. mark vertex u; 2. d[u]  0; 3. for each unmarked vertex v  V do 4. if edge (u,v) exists d [v]  weight (u,v); 5. else d[v]   ; 6. while there exists an unmarked vertex do 7. let v be an unmarked vertex such that d[v] is minimal; 8. mark vertex v; 9. for all edges (v,x) such that x is unmarked do 10. if d[x] > d[v] + weight[v,x] then 11. d[x]  d[v] + weight[v,x]

Algorithms Design, Greedy Algorithms34 Complexity of Dijkstra's algorithm:  Steps 1 and 2 take  (1) time  Steps 3 to 5 take O(  V  ) time  The vertices are arranged in a heap in order of their paths from u  Updating the length of a path takes O(log V) time.  There are  V  iterations, and at most  E  updates  Therefore the algorithm takes O((  E  +  V  ) log  V  ) time.

 Spanning tree of a connected graph G: a connected acyclic subgraph of G that includes all of G’s vertices  Minimum spanning tree of a weighted, connected graph G: a spanning tree of G of the minimum total weight  # MST = n n-2 (n = numbers of nodes) Example: c d b a c d b a c d b a 6 4 1

 Start with tree T 1 consisting of one (any) vertex and “grow” tree one vertex at a time to produce MST through a series of expanding subtrees T 1, T 2, …, T n  On each iteration, construct T i+1 from T i by adding vertex not in T i that is closest to those already in T i (this is a “greedy” step!)  Stop when all vertices are included

c d b a c d b a c d b a c d b a c d b a

 Proof by induction that this construction actually yields an MST (CLRS, Ch. 23.1).  Efficiency ◦ O(n 2 ) for weight matrix representation of graph and array implementation of priority queue ◦ O(m log n) for adjacency list representation of graph with n vertices and m edges and min-heap implementation of the priority queue

5/16/  The algorithm maintains a collection VS of disjoint sets of vertices  Each set W in VS represents a connected set of vertices forming a spanning tree  Initially, each vertex is in a set by itself in VS  Edges are chosen from E in order of increasing cost, we consider each edge (v, w) in turn; v, w  V.  If v and w are already in the same set (say W) of VS, we discard the edge  If v and w are in distinct sets W1 and W2 (meaning v and/or w not in T) we merge W1 with W2 and add (v, w) to T.

5/16/  Procedure MCST_G(V,E)  (Kruskal's Algorithm)  Input: An undirected graph G(V,E) with a cost function c on the edges  Output: T the minimum cost spanning tree for G 1. T  0; 2. VS  0; 3. for each vertex v  V do 4. VS = VS  {v}; 5. sort the edges of E in nondecreasing order of weight 6. while  VS  > 1 do 7. choose (v,w) an edge E of lowest cost; 8. delete (v,w) from E; 9. if v and w are in different sets W1 and W2 in VS do 10. W1 = W1  W2; 11. VS = VS - W2; 12. T  T  (v,w); 13. return T

5/16/ Consider the example graph shown earlier, The edges in nondecreasing order [(A,D),1],[(C,D),1],[(C,F),2],[(E,F),2],[(A,F),3],[(A,B),3], [(B,E),4],[(D,E),5],[(B,C),6] EdgeActionSets in VS Spanning Tree, T =[{A},{B},{C},{D},{E},{F}] {0}(A,D) merge [{A,D}, {B},{C}, {E}, {F}] {(A,D)} (C,D) merge [{A,C,D}, {B}, {E}, {F}] {(A,D), (C,D)} (C,F) merge [{A,C,D,F},{B},{E}]{ (A,D),(C,D), (C,F)} (E,F) merge [{A,C,D,E,F},{B}] {(A,D),(C,D), (C,F),(E,F)}(A,F) reject [{A,C,D,E,F},{B}] {(A,D),(C,D), (C,F), (E,F)}(A,B) merge [{A,B,C,D,E,F}] {(A,D),(A,B),(C,D), (C,F),(E,F)}(B,E) reject (D,E) reject (B,C) reject

5/16/  Steps 1 thru 4 take time O (V)  Step 5 sorts the edges in nondecreasing order in O (E log E ) time  Steps 6 through 13 take O (E) time  The total time for the algorithm is therefore given by O (E log E)  The edges can be maintained in a heap data structure with the property, E[PARENT(i)]  E[i]  This property can be used to sort data elements in nonincreasing order.  Construct a heap of the edge weights, the edge with lowest cost is at the root During each step of edge removal, delete the root (minimum element) from the heap and rearrange the heap.  The use of heap data structure reduces the time taken because at every step we are only picking up the minimum or root element rather than sorting the edge weights.

5/16/ Consider a network of computers connected through bidirectional links. Each link is associated with a positive cost: the cost of sending a message on each link. This network can be represented by an undirected graph with positive costs on each edge. In bidirectional networks we can assume that the cost of sending a message on a link does not depend on the direction. Suppose we want to broadcast a message to all the computers from an arbitrary computer. The cost of the broadcast is the sum of the costs of links used to forward the message.

5/16/201544

 Problem Formulation ◦ Given a set of n activities, S = {a 1, a 2,..., a n } that require exclusive use of a common resource, find the largest possible set of nonoverlapping activities (also called mutually compatible).  For example, scheduling the use of a classroom. ◦ Assume that a i needs the resource during period [s i, f i ), which is a half-open interval, where  s i = start time of the activity, and  f i = finish time of the activity.  Note: Could have many other objectives: ◦ Schedule room for longest time. ◦ Maximize income rental fees.

 Assume the following set S of activities that are sorted by their finish time, find a maximum-size mutually compatible set. i sisi fifi

i sisi fifi

 Define S i,j = {a k  S : f i  s k < f k  s j } ◦ activities that start after a i finishes and finish before a j starts  Activities in S i,j are compatible with: ◦ a i and a j ◦ a w that finishes not later than a i and start not earlier than a j  Add the following [imaginary] activities a 0 = [– , 0) and a n+1 = [ ,  +1)  Hence, S = S 0,n+1 and the range of S i,j is 0  i,j  n+1

 Assume that activities are sorted by monotonically increasing finish time: ◦ i.e., f 0  f 1  f 2 ...  f n < f n+1  Then, S i,j =  for i  j. Proof:  Therefore, we only need to worry about S i,j where 0  i < j  n+1

 Suppose that a solution to S i,j includes a k. We have 2 sub-problems: ◦ S i,k (start after a i finishes, finish before a k starts) ◦ S k,j (start after a k finishes, finish before a j starts)  The Solution to S i,j is (solution to S i,k )  {a k }  (solution to S k,j )  Since a k is in neither sub-problem, and the subproblems are disjoint, |solution to S| = |solution to S i,k | |solution to S k,j |

 Let A i,j = optimal solution to S i,j. So A i,j = A i,k  {a k }  A k,j, assuming: ◦ S i,j is nonempty, and ◦ we know a k.  Hence,  c[i,j]: number of activities in a max subset of mutually compatible activities

 Theorem: Let S i,j  , and let a m be the activity in S i,j with the earliest finish time: f m = min { f k : a k  S i,j }. Then: 1.a m is used in some maximum-size subset of mutually compatible activities of S i,j 2.S im = , so that choosing a m leaves S m,j as the only nonempty subproblem.

Algorithms Design, Greedy Algorithms54

ɵ (Sorting)+ ɵ (n) = ɵ (n log n)

56  A board has a certain number of coins on it  A robot starts in the upper-left corner, and walks to the bottom left-hand corner ◦ The robot can only move in two directions: right and down ◦ The robot collects coins as it goes  You want to collect all the coins using the minimum number of robots  Example:  Do you see a greedy algorithm for doing this?  Does the algorithm guarantee an optimal solution? ◦ Can you prove it? ◦ Can you find a counterexample?