1. Introduction to Algorithms`
Chapter 16: Greedy Algorithms

2. Overview
Like dynamic programming, used to solve optimization problems.
Problems exhibit optimal substructure (like DP).
Problems also exhibit the greedy-choice property.
When we have a choice to make, make the one that looks best right now.
Make a locally optimal choice in hope of getting a globally optimal solution.
Comp 122, Fall 2003

3. Greedy Strategy
The choice that seems best at the moment is the one we go with.
Prove that when there is a choice to make, one of the optimal choices is the greedy choice. Therefore, it’s always safe to make the greedy choice.
Show that all but one of the subproblems resulting from the greedy choice are empty.
Comp 122, Fall 2003

4. Activity-selection Problem
Input: Set S of n activities, a1, a2, …, an.
si = start time of activity i.
fi = finish time of activity i.
Output: Subset A of maximum number of compatible activities.
Two activities are compatible, if their intervals don’t overlap.
Example:
Activities in each line
are compatible.
Comp 122, Fall 2003

5. Optimal Substructure Assume activities are sorted by finishing times.
f1  f2  …  fn.
Suppose an optimal solution includes activity ak.
This generates two subproblems.
Selecting from a1, …, ak-1, activities compatible with one another, and that finish before ak starts (compatible with ak).
Selecting from ak+1, …, an, activities compatible with one another, and that start after ak finishes.
The solutions to the two subproblems must be optimal.
Prove using the cut-and-paste approach.
Comp 122, Fall 2003

6. Recursive Solution
Let Sij = subset of activities in S that start after ai finishes and finish before aj starts.
Subproblems: Selecting maximum number of mutually compatible activities from Sij.
Let c[i, j] = size of maximum-size subset of mutually compatible activities in Sij.
Recursive
Solution:
Comp 122, Fall 2003

7. Greedy-choice Property
The problem also exhibits the greedy-choice property.
There is an optimal solution to the subproblem Sij, that includes the activity with the smallest finish time in set Sij.
Can be proved easily.
Hence, there is an optimal solution to S that includes a1.
Therefore, make this greedy choice without solving subproblems first and evaluating them.
Solve the subproblem that ensues as a result of making this greedy choice.
Combine the greedy choice and the solution to the subproblem.
Comp 122, Fall 2003

8. Recursive Algorithm Recursive-Activity-Selector (s, f, i, j) m  i+1
while m < j and sm < fi
do m  m+1
if m < j
then return {am} 
Recursive-Activity-Selector(s, f, m, j)
else return 
Initial Call: Recursive-Activity-Selector (s, f, 0, n+1)
Complexity: (n)
Straightforward to convert the algorithm to an iterative one.
See the text.
Comp 122, Fall 2003

9. Typical Steps
Cast the optimization problem as one in which we make a choice and are left with one subproblem to solve.
Prove that there’s always an optimal solution that makes the greedy choice, so that the greedy choice is always safe.
Show that greedy choice and optimal solution to subproblem  optimal solution to the problem.
Make the greedy choice and solve top-down.
May have to preprocess input to put it into greedy order.
Example: Sorting activities by finish time.
Comp 122, Fall 2003

10. Elements of Greedy Algorithms
Greedy-choice Property.
A globally optimal solution can be arrived at by making a locally optimal (greedy) choice.
Optimal Substructure.
Comp 122, Fall 2003

11. Minimum Spanning Trees
Comp 122, Spring 2004

12. Carolina Challenge
Comp 122, Fall 2003

13. Minimum Spanning Trees
Given: Connected, undirected, weighted graph, G
Find: Minimum - weight spanning tree, T
Example: 7 b c 5
Acyclic subset of edges(E) that connects
all vertices of G. a 1 3 -3 11 d e f 2 b c 5 a 1 3 -3 d e f
Comp 122, Fall 2003

14. Generic Algorithm “Grows” a set A. A is subset of some MST.
Edge is “safe” if it can be added to A without destroying this
invariant.
A := ;
while A not complete tree do
find a safe edge (u, v);
A := A  {(u, v)} od
Comp 122, Fall 2003

15. Definitions no edge in the set crosses the cut
cut respects the edge set {(a, b), (b, c)}
a light edge crossing cut
(could be more than one) 5 7 a b c 1 -3 11 3
cut partitions vertices into
disjoint sets, S and V – S. d e f 2
this edge crosses the cut
one endpoint is in S and the other is in V – S.
Comp 122, Fall 2003

16. Theorem 23.1 edge in A (x, y) crosses cut.
Theorem 23.1: Let (S, V-S) be any cut that respects A, and let (u, v)
be a light edge crossing (S, V-S). Then, (u, v) is safe for A.
Proof:
Let T be a MST that includes A.
Case: (u, v) in T. We’re done.
Case: (u, v) not in T. We have the following:
edge in A
(x, y) crosses cut.
Let T´ = T - {(x, y)}  {(u, v)}.
Because (u, v) is light for cut,
w(u, v)  w(x, y). Thus,
w(T´) = w(T) - w(x, y) + w(u, v)  w(T).
Hence, T´ is also a MST.
So, (u, v) is safe for A. x
cut y u
shows edges
in T v
Comp 122, Fall 2003

17. Corollary In general, A will consist of several connected components.
Corollary: If (u, v) is a light edge connecting one CC in (V, A)
to another CC in (V, A), then (u, v) is safe for A.
Comp 122, Fall 2003

18. Kruskal’s Algorithm Starts with each vertex in its own component.
Repeatedly merges two components into one by choosing a light edge that connects them (i.e., a light edge crossing the cut between them).
Scans the set of edges in monotonically increasing order by weight.
Uses a disjoint-set data structure to determine whether an edge connects vertices in different components.
Comp 122, Fall 2003

19. Prim’s Algorithm Builds one tree, so A is always a tree.
Starts from an arbitrary “root” r .
At each step, adds a light edge crossing cut (VA, V - VA) to A.
VA = vertices that A is incident on.
Comp 122, Fall 2003

20. Prim’s Algorithm Uses a priority queue Q to find a light edge quickly.
Each object in Q is a vertex in V - VA.
Key of v is minimum weight of any edge (u, v), where u  VA.
Then the vertex returned by Extract-Min is v such that there exists u  VA and (u, v) is light edge crossing (VA, V - VA).
Key of v is  if v is not adjacent to any vertex in VA.
Comp 122, Fall 2003

21. Prim’s Algorithm Q := V[G]; for each u  Q do key[u] := 
od;
key[r] := 0;
[r] := NIL;
while Q   do
u := Extract - Min(Q);
for each v  Adj[u] do
if v  Q  w(u, v) < key[v] then
[v] := u;
key[v] := w(u, v) fi od
Complexity:
Using binary heaps: O(E lg V).
Initialization – O(V).
Building initial queue – O(V).
V Extract-Min’s – O(V lgV).
E Decrease-Key’s – O(E lg V).
Using Fibonacci heaps: O(E + V lg V).
(see book)
 decrease-key operation
Note: A = {(v, [v]) : v  v - {r} - Q}.
Comp 122, Fall 2003

22. Example of Prim’s Algorithm
Not in tree 5 7
a/0
b/
c/ 1
Q = a b c d e f
0   -3 11 3
d/
e/
f/ 2
Comp 122, Fall 2003

23. Example of Prim’s Algorithm5 7
a/0
b/5
c/ 1
Q = b d c e f
5 11   -3 11 3
d/11
e/
f/ 2
Comp 122, Fall 2003

24. Example of Prim’s Algorithm5 7
a/0
b/5
c/7 1
Q = e c d f  -3 11 3
d/11
e/3
f/ 2
Comp 122, Fall 2003

25. Example of Prim’s Algorithm5 7
a/0
b/5
c/1 1
Q = d c f -3 11 3
d/0
e/3
f/2 2
Comp 122, Fall 2003

26. Example of Prim’s Algorithm5 7
a/0
b/5
c/1 1
Q = c f
1 2 -3 11 3
d/0
e/3
f/2 2
Comp 122, Fall 2003

27. Example of Prim’s Algorithm5 7
a/0
b/5
c/1 1
Q = f -3 -3 11 3
d/0
e/3
f/-3 2
Comp 122, Fall 2003

28. Example of Prim’s Algorithm5 7
a/0
b/5
c/1 1
Q =  -3 11 3
d/0
e/3
f/-3 2
Comp 122, Fall 2003

29. Example of Prim’s Algorithm5
a/0
b/5
c/1 1 3 -3
d/0
e/3
f/-3
Comp 122, Fall 2003

30. Greedy Algorithms Similar to dynamic programming, but simpler approach
Also used for optimization problems
Idea: When we have a choice to make, make the one that looks best right now
Make a locally optimal choice in hope of getting a globally optimal solution
Greedy algorithms don’t always yield an optimal solution
Makes the choice that looks best at the moment in order to get optimal solution.

31. Fractional Knapsack Problem
Knapsack capacity: W
There are n items: the i-th item has value vi and weight wi
Goal:
find xi such that for all 0  xi  1, i = 1, 2, .., n
 wixi  W and
 xivi is maximum

32. Fractional Knapsack - Example
E.g.: 50 20
--- 30 50
$80 +
Item 3 30 20
Item 2
$100 + 20
Item 1 10 10
$60
$60
$100
$120
$240
$6/pound
$5/pound
$4/pound

33. Fractional Knapsack Problem
Greedy strategy 1:
Pick the item with the maximum value
E.g.:
W = 1
w1 = 100, v1 = 2
w2 = 1, v2 = 1
Taking from the item with the maximum value:
Total value taken = v1/w1 = 2/100
Smaller than what the thief can take if choosing the other item
Total value (choose item 2) = v2/w2 = 1

34. Fractional Knapsack Problem
Greedy strategy 2:
Pick the item with the maximum value per pound vi/wi
If the supply of that element is exhausted and the thief can carry more: take as much as possible from the item with the next greatest value per pound
It is good to order items based on their value per pound

35. Fractional Knapsack Problem
Alg.: Fractional-Knapsack (W, v[n], w[n])
While w > 0 and as long as there are items remaining
pick item with maximum vi/wi
xi  min (1, w/wi)
remove item i from list
w  w – xiwi
w – the amount of space remaining in the knapsack (w = W)
Running time: (n) if items already ordered; else (nlgn)

36. Huffman Code Problem
Huffman’s algorithm achieves data compression by finding the best variable length binary encoding scheme for the symbols that occur in the file to be compressed.

37. Huffman Code Problem
The more frequent a symbol occurs, the shorter should be the Huffman binary word representing it.
The Huffman code is a prefix-free code.
No prefix of a code word is equal to another codeword.

38. Overview Huffman codes: compressing data (savings of 20% to 90%)
Huffman’s greedy algorithm uses a table of the frequencies of occurrence of each character to build up an optimal way of representing each character as a binary string
C: Alphabet

39. Example
Assume we are given a data file that contains only 6 symbols, namely a, b, c, d, e, f With the following frequency table:
Find a variable length prefix-free encoding scheme that compresses this data file as much as possible?

40. Huffman Code Problem
Left tree represents a fixed length encoding scheme
Right tree represents a Huffman encoding scheme

41. Example

42. Constructing A Huffman Code
// C is a set of n characters
// Q is implemented as a binary min-heap
O(n)
Total computation time = O(n lg n)
O(lg n)
O(lg n)
O(lg n)

43. Cost of a Tree T For each character c in the alphabet C
let f(c) be the frequency of c in the file
let dT(c) be the depth of c in the tree
It is also the length of the codeword. Why?
Let B(T) be the number of bits required to encode the file (called the cost of T)

44. Huffman Code Problem In the pseudocode that follows:
we assume that C is a set of n characters and that each character c €C is an object with a defined frequency f [c].
The algorithm builds the tree T corresponding to the optimal code
A min-priority queue Q, is used to identify the two least-frequent objects to merge together.
The result of the merger of two objects is a new object whose frequency is the sum of the frequencies of the two objects that were merged.

45. Running time of Huffman's algorithm
The running time of Huffman's algorithm assumes that Q is implemented as a binary min-heap.
For a set C of n characters, the initialization of Q in line 2 can be performed in O(n) time using the BUILD-MINHEAP
The for loop in lines 3-8 is executed exactly n - 1 times, and since each heap operation requires time O(lg n), the loop contributes O(n lg n) to the running time. Thus, the total running time of HUFFMAN on a set of n characters is O(n lg n).

46. Prefix Code
Prefix(-free) code: no codeword is also a prefix of some other codewords (Un-ambiguous)
An optimal data compression achievable by a character code can always be achieved with a prefix code
Simplify the encoding (compression) and decoding
Encoding: abc  =
Decoding: =  aabe
Use binary tree to represent prefix codes for easy decoding
An optimal code is always represented by a full binary tree, in which every non-leaf node has two children
|C| leaves and |C|-1 internal nodes Cost:
Depth of c (length of the codeword)
Frequency of c

47. Huffman Code Reduce size of data by 20%-90% in general
If no characters occur more frequently than others, then no advantage over ASCII
Encoding:
Given the characters and their frequencies, perform the algorithm and generate a code. Write the characters using the code
Decoding:
Given the Huffman tree, figure out what each character is (possible because of prefix property)

48. Application on Huffman code
Both the .mp3 and .jpg file formats use Huffman coding at one stage of the compression

49. Dynamic Programming vs. Greedy Algorithms
We make a choice at each step
The choice depends on solutions to subproblems
Bottom up solution, from smaller to larger subproblems
Greedy algorithm
Make the greedy choice and THEN
Solve the subproblem arising after the choice is made
The choice we make may depend on previous choices, but not on solutions to subproblems
Top down solution, problems decrease in size