1. Optimization problems, Greedy Algorithms, Optimal Substructure and Greedy choice Learning & Development Team http://academy.telerik.com Telerik Software Academy

2. 1. Optimization Problems 2. Greedy Algorithms and Failure Cases 3. Optimal Greedy Algorithms  Optimal Substructure and Greedy Choice  Demo: Proving Optimality of a Greedy Solution 4. The Set Cover Problem 5. Notable Greedy algorithms 2

3. Not "just" Looking for a Solution

4.  Finding the best solution of a problem  From all solution candidates  i.e. most optimal solution candidate  Not just any solution  Two categories – based on variables  Continuous – variable values are continuous  Discrete – a.k.a. Combinatorial 4

5.  Importance of optimization problems  Optimal solutions reduce cost  Optimal solutions are more likely to be realistically possible  E.g. finding any route between to a city  In theory enables getting there  In practice it might be too long to travel  Finding the shortest route is much better

6. Picking Locally Best Solution

7.  Greedy algorithms are a category of algorithms  Can solve some optimization problems  Usually more efficient than all other algorithms  For the same problems  Greedy algorithms pick  The best solution  From their current position & point of view  i.e. they make local solutions

8.  Playing against someone, alternating turns  Per turn, you can take up to three coins  Your goal is to have as much coins as possible

9.  Things to notice in the way you played to win Always take the max number of coins i.e. make the current optimal solution You don't consider what the other player does You don't consider your actions' consequences The greedy algorithm works optimally here

10.  A greedy algorithm solves a problem in steps  At each step  Algorithm picks the best action available  Best action is determined regardless of future actions or states  i.e. greedy algorithms assume  Always choosing a local optimum  Leads to the global optimum 10

11.  Main components of a Greedy algorithm  A candidate set  A selection function  A feasibility function  An objective function and a solution function  Basically, a greedy algorithms have  A set of possible actions  A way to pick the best action at a time  A way to determine the solution is reached

12.  Greedy algorithms exploit problem structure  E.g. where the solution is the sum of the optimal solutions to subproblems  E.g. where greedy choices don't lead to bad overall positions  Many currency systems' denomination units are suited to greedy processing  Ironic, isn't it?

13.  Consider the US currency denominations  Problem: gather a sum of money, using the least possible number of bills/coins  Suppose you have infinite supplies of each denomination Sum to make: 1.29$

14.  Greedy algorithm to make a sum  With minimum number of coins  Given the US currency system  This will work for most 1-2-5 series based systems We want to achieve the sum S We start with the sum Q = 0 We have a set of coins C = { 1, 5, 10 … } At each step pick the largest value V from C such that Q + V less than or equal to S pick the largest value V from C such that Q + V less than or equal to S Increase Q by V //i.e. add a coin of value V Increase Q by V //i.e. add a coin of value V Repeat until Q == S //the number of repetitions is the number of needed coins

15. Live Demo

16.  Greedy algorithms are often not optimal  Even can reach the unique worst possible solutions for some problems  Example: Largest sum path (starting at top)  Example: Coin denominations 4, 10, 25 ; Greedy will fail to make the sum 41 which is 25 + 4 * 4

17. Optimal Substructure, Greedy Choice Property, Proving Optimality of a Greedy Approach

18.  Suitable problems for greedy algorithms often have these properties:  Greedy choice property  Optimal substructure  Any problem having the above properties  Guaranteed to have an optimal greedy solution  Matroids – way to prove greedy optimality Matroids  If a problem has the properties of a matroid, it is guaranteed to have an optimal greedy solution

19.  Greedy choice property  An optimal solution to the problem begins with a greedy choice  Subproblems that arise can be solved by consequent choices  Also enforced by optimal substructure

20.  Optimal substructure  After each greedy choice  The problem remains an optimization problem  Of the same form as the original problem  i.e. the optimal solution to the problem contains optimal solutions to the subproblems

21. Greedy for the Activity Selection Problem and Proving its Optimality

22.  The Activity Selection Problem (a.k.a. Conference Scheduling Problem)  Given a set of activities S = {a 1, a 2, … a n }  Each with a start & finish time: a i = (s i, f i )  Activities are "compatible" if they don't overlap  i.e. their intervals do not intersect  What is the maximum-size subset of compatible activities?  i.e. which is the largest list of compatible activities that can be scheduled

23.  The Activity Selection Problem  Can have several optimal solutions  In the following case {a 1, a 4, a 8, a 11 } is optimal  Another optimal is {a 2, a 4, a 9, a 11 } Index1234567891011 Start (s i )130535688212 Finish (f i )4567891011121314

24.  A greedy algorithm for the task:  Greedy characteristic of above algorithm  Taking the earliest finish activity gives more time for other activities  i.e. choose the "maximum remaining time" Select activity with the earliest finish from S Select activity with the earliest finish from S Remove activities in S conflicting with selected Remove activities in S conflicting with selected i.e. non-compatible activities are removed i.e. non-compatible activities are removed Repeat the until no activities remain in S Repeat the until no activities remain in S

25.  To prove the discussed greedy is optimal  Need to prove the problem has a greedy choice property  Need to prove the problem has optimal substructure  Observations 1.If there exists an optimal solution, it is a subset of activities from S 2.In any solution, the first activity to start is the first to finish

26.  Let A be an optimal solution (subset of S)  Sort activities in A by finish time. Let k be the index of the earliest activity in A  If k = 1 => A begins with a greedy choice  If k != 1 =>  Let B = (A – {k}) + {1}. Prove B is optimal:  f 1 1 ) => f 1 doesn't overlap any activity in B, so B is a solution  B has the same size (number of activities) as A  Hence, B is also optimal

27.  So far we proved that:  A solution starting with a greedy choice exists  The greedy choice solution is also optimal  Hence we proved the problem exhibits the greedy choice property  There exists an optimal solution, starting with a greedy choice  Now, we need to prove the problem has optimal substructure

28.  We have selected activity 1 (greedy 1 st choice)  Thus, we reduced to the same problem form  Without activities in S which intersect activity 1  If A is optimal to S, then:  A' = A – {1} is optimal to S' = all activities of S which start after f 1  Can A' be non-optimal to S' ?  If exists B' with more activities than A' (from S' )  Adding activity 1 to B' gives B with more activities (from S ) than A -> contradiction

29.  We just proved the problem has optimal substructure  Each greedy choice leads us to a problem of the same form  The new problem's solution is a subset of the initial problem's solution  i.e. all local solutions joined form the global optimal solution  We have proven both properties, so our greedy algorithm is optimal

30. Live Demo

31. Using Greedy for Approximation

32.  Greedy algorithms can sometimes find optimal solutions  This is not their only application  There exist problems, for which  An optimal solution is to complex to find  i.e. calculating an optimal solution is infeasible  Sometimes greedy algorithms provide good approximations of the optimal result

33.  The Set Cover Problem (SCP) is such a problem  SCP formulation:  Given a set {1,2,…,m} called "the Universe" ( U )  And a set S{{…},…} of n sets whose union = U  Find the smallest subset of S, the union of which = U (if it exists)  So, we have a target set, and a set of sets with repeating elements (i.e. redundant elements)  How do we find the smallest number of sets, which in union make the target set

34.  The SCP turns out very complex  The optimal solution is NP-complete  i.e. infeasible for calculations (unless P = NP) P = NPP = NP  However, relatively good solutions can be achieved through a greedy approach:  At each step, pick the set containing the largest number of uncovered elements

35. Several Common Greedy Algorithms

36.  Dijkstra's algorithm for finding the shortest path between two vertices  in a weighted graph (with no negative cycles)  At each step, of all reached edges, pick:  the one that, along with the path to it, constitutes a minimal sum  The algorithms is proven optimal  Immediately after it reaches a vertex, the path generated to it is guaranteed to be optimal  i.e. no need to traverse all vertices

37.  Prim and Kruskal's algorithms for a minimum spanning tree (MST) are greedy algorithms  Prim:  pick the smallest edge, not in the MST so far  Kruskal:  pick the smallest edge, connecting two vertices, not in the same tree  Both algorithms have the same complexity

38.  The prefix tree generation algorithm in Huffman coding is greedy  Greedy: pick the two smallest-value leaves/nodes and combine them  Left move: 0, Right move: 1 46 15 27 54 100 0 0 0 0 0 1 1 1 1 1 A: 00 B: 100 C: 01 D: 1010 E: 11 F: 1011 Frequency %2212246279 SymbolABCDEF

39. Live Demo

40. форум програмиране, форум уеб дизайн курсове и уроци по програмиране, уеб дизайн – безплатно програмиране за деца – безплатни курсове и уроци безплатен SEO курс - оптимизация за търсачки уроци по уеб дизайн, HTML, CSS, JavaScript, Photoshop уроци по програмиране и уеб дизайн за ученици ASP.NET MVC курс – HTML, SQL, C#,.NET, ASP.NET MVC безплатен курс "Разработка на софтуер в cloud среда" BG Coder - онлайн състезателна система - online judge курсове и уроци по програмиране, книги – безплатно от Наков безплатен курс "Качествен програмен код" алго академия – състезателно програмиране, състезания ASP.NET курс - уеб програмиране, бази данни, C#,.NET, ASP.NET курсове и уроци по програмиране – Телерик академия курс мобилни приложения с iPhone, Android, WP7, PhoneGap free C# book, безплатна книга C#, книга Java, книга C# Николай Костов - блог за програмиране http://algoacademy.telerik.com

41.  The following materials were very helpful in the making of this lecture and we recommend them for further reading:  Lecture @ Boston University (Shang-Hua Teng)  http://www.cs.bu.edu/~steng/teaching/Fall2003/lectures/l ecture7.ppt http://www.cs.bu.edu/~steng/teaching/Fall2003/lectures/l ecture7.ppt http://www.cs.bu.edu/~steng/teaching/Fall2003/lectures/l ecture7.ppt  Lecture @ University of Pennsylvania  http://www.cis.upenn.edu/~matuszek/cit594- 2005/Lectures/36-greedy.ppt http://www.cis.upenn.edu/~matuszek/cit594- 2005/Lectures/36-greedy.ppt http://www.cis.upenn.edu/~matuszek/cit594- 2005/Lectures/36-greedy.ppt  Book on Algorithms @ Berkeley University  http://www.cs.berkeley.edu/~vazirani/algorithms/ch ap5.pdf http://www.cs.berkeley.edu/~vazirani/algorithms/ch ap5.pdf http://www.cs.berkeley.edu/~vazirani/algorithms/ch ap5.pdf