1. Approaches to Problem Solving greedy algorithms dynamic programming backtracking divide-and-conquer

2. Problem: Huffman Coding Def: binary character code = assignment of binary strings to characters e.g. ASCII code A = 01000001 B = 01000010 C = 01000011 … fixed-length code How to decode: ? 01000001010000100100001101000001

3. Problem: Huffman Coding e.g. code A = 0 B = 10 C = 11 … variable-length code How to decode: ? 0101001111 Def: binary character code = assignment of binary strings to characters

4. Problem: Huffman Coding e.g. code A = 0 B = 10 C = 11 … How to decode: ? 0101001111 Def: A code is prefix-free if no codeword is a prefix of another codeword. variable-length code Def: binary character code = assignment of binary strings to characters

5. Problem: Huffman Coding Def: A code is prefix-free if no codeword is a prefix of another codeword. variable-length code Def: binary character code = assignment of binary strings to characters e.g. another code A = 1 B = 10 C = 11 … How to decode: ? 10101111

6. Problem: Huffman Coding Def: Huffman coding is an optimal prefix-free code. E11.1607% A8.4966% R7.5809% I7.5448% O7.1635% T6.9509% N6.6544% S5.7351% L5.4893% C4.5388% U3.6308% D3.3844% P3.1671% M3.0129% H3.0034% G2.4705% B2.0720% F1.8121% Y1.7779% W1.2899% K1.1016% V1.0074% X0.2902% Z0.2722% J0.1965% Q0.1962% Optimization problems - Input: - Output: - Objective:

7. an alphabet with frequencies a prefix-free code minimize expected number of bits per character Problem: Huffman Coding Def: Huffman coding is an optimal prefix-free code. E11.1607% A8.4966% R7.5809% I7.5448% O7.1635% T6.9509% N6.6544% S5.7351% L5.4893% C4.5388% U3.6308% D3.3844% P3.1671% M3.0129% H3.0034% G2.4705% B2.0720% F1.8121% Y1.7779% W1.2899% K1.1016% V1.0074% X0.2902% Z0.2722% J0.1965% Q0.1962% Huffman coding - Input: - Output: - Objective:

8. Problem: Huffman Coding A60% B20% C10% D10% an alphabet with frequencies a prefix-free code minimize expected number of bits per character Huffman coding - Input: - Output: - Objective: Example:Is fixed-width coding optimal ?

9. Problem: Huffman Coding A60% B20% C10% D10% an alphabet with frequencies a prefix-free code minimize expected number of bits per character Huffman coding - Input: - Output: - Objective: Example:Is fixed-width coding optimal ? NO, exists a prefix-free code using 1.6 bits per character !

10. Problem: Huffman Coding an alphabet with frequencies a prefix-free code minimize expected number of bits per character Huffman coding - Input: - Output: - Objective: Huffman ( [a 1,f 1 ],[a 2,f 2 ],…,[a n,f n ]) 1. if n=1 then 2. code[a 1 ]  “” 3. else 4. let f i,f j be the 2 smallest f’s 5. Huffman ( [a i,f i +f j ],[a 1,f 1 ],…,[a n,f n ] ) omits a i,a j 6. code[a j ]  code[a i ] + “0” 7. code[a i ]  code[a i ] + “1”

11. Problem: Huffman Coding Let x,y be the symbols with frequencies f x > f y. Then in an optimal prefix code length(C x )  length(C y ). Lemma 1:

12. Problem: Huffman Coding Let x,y be the symbols with frequencies f x > f y. Then in an optimal prefix code length(C x )  length(C y ). Lemma 1: If w is a longest codeword in an optimal code then there exists another codeword of the same length. Lemma 2:

13. Problem: Huffman Coding Let x,y be the symbols with frequencies f x > f y. Then in an optimal prefix code length(C x )  length(C y ). Lemma 1: Let x,y be the symbols with the smallest frequencies. Then there exists an optimal prefix code such that the codewords for x and y differ only in the last bit. Lemma 3: If w is a longest codeword in an optimal code then there exists another codeword of the same length. Lemma 2:

14. Problem: Huffman Coding Let x,y be the symbols with frequencies f x > f y. Then in an optimal prefix code length(C x )  length(C y ). Lemma 1: Let x,y be the symbols with the smallest frequencies. Then there exists an optimal prefix code such that the codewords for x and y differ only in the last bit. Lemma 3: The prefix code output by the Huffman algorithm is optimal. Theorem: If w is a longest codeword in an optimal code then there exists another codeword of the same length. Lemma 2:

15. Greedy Algorithms - make decisions “greedily”, previous decisions are never reconsidered - e.g. Huffman coding - another example: the divisible version of KNAPSACK Input: Output: Objective: a number W and a set of n items, the i-th item has a weight w i and a cost c i a subset of items with total weight · W maximize cost Version 1: Items are divisible. KNAPSACK

16. KNAPSACK – divisible: a greedy solution KNAPSACK-DIVISIBLE(n,c,w,W) 1. sort items in decreasing order of c i /w i 2. i = 1 3. currentW = 0 4. while (currentW + w i < W) { 5. take item of weight w i and cost c i 6. currentW += w i 7. i++ 8. } 9. take W-currentW portion of item i Correctness: Running time:

17. KNAPSACK – indivisible Version 2: Items are indivisible. Does previous algorithm work for this version of KNAPSACK?

18. BackTracking - “brute force”: try all possibilities - running time?

19. Dynamic Programming - idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, …

20. Dynamic Programming - idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … A recursive solution: Fib(n) 1. if n < 0 then RETURN “undefined” 2. if n · 1 then RETURN n 3. RETURN Fib(n-1) + Fib(n-2)

21. Dynamic Programming - idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … A recursive solution: Fib(n) 1. if n < 0 then RETURN “undefined” 2. if n · 1 then RETURN n 3. RETURN Fib(n-1) + Fib(n-2) Can do better (faster) ?

22. Dynamic Programming - idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … A recursive solution: Fib(n) 1. if n < 0 then RETURN “undefined” 2. if n · 1 then RETURN n 3. RETURN Fib(n-1) + Fib(n-2) Can do better (faster) ? YES ! Just store Fib[k] in an array !

23. FibDynProg(n) 1. Fib[0] = 0 2. Fib[1] = 1 3. for i=2 to n do 4. Fib[i] = Fib[i-1] + Fib[i-2] 5. RETURN Fib[n] Dynamic Programming - idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … Can do better (faster) ? YES ! Just store Fib[k] in an array ! The heart of the solution: Fib[k] = the k-th Fibonacci number

24. KNAPSACK – indivisible: a dyn-prog solution A recursive approach (returns only the best cost, not the set of items): KNAP-IND-REC(n,c,w,W) 1. if n · 0 and W<0 2. return 0 3. if W<w n : 4. withLastItem = -1 // undefined 5. else: 6. withLastItem =c n +KNAP-IND-REC(n-1,c,w,W-w n ) 7. withoutLastItem = KNAP-IND-REC(n-1,c,w,W) 8. return max{withLastItem,withoutLastItem}

25. KNAPSACK – indivisible: a dyn-prog solution The heart of the algorithm: S[k][v] =

26. KNAPSACK – indivisible: a dyn-prog solution maximum cost of a subset of the first k items, where the weight of the subset is at most v The heart of the algorithm: S[k][v] =

27. KNAPSACK – indivisible: a dyn-prog solution maximum cost of a subset of the first k items, where the weight of the subset is at most v The heart of the algorithm: S[k][v] = KNAPSACK-INDIVISIBLE(n,c,w,W) 1. init S[0][v]=0 for every v=0,…,W 2. init S[k][0]=0 for every k=0,…,n 3. for v=1 to W do 4. for k=1 to n do 5. S[k][v] = S[k-1][v] 6. if (w k · v) and (S[k-1][v-w k ]+c k > S[k][v]) then 7. S[k][v] = S[k-1][v-w k ]+c k 8. RETURN S[n][W]

28. KNAPSACK – indivisible: a dyn-prog solution maximum cost of a subset of the first k items, where the weight of the subset is at most v The heart of the algorithm: S[k][v] = Running time: KNAPSACK-INDIVISIBLE(n,c,w,W) 1. init S[0][v]=0 for every v=0,…,W 2. init S[k][0]=0 for every k=0,…,n 3. for v=1 to W do 4. for k=1 to n do 5. S[k][v] = S[k-1][v] 6. if (w k · v) and (S[k-1][v-w k ]+c k > S[k][v]) then 7. S[k][v] = S[k-1][v-w k ]+c k 8. RETURN S[n][W]

29. KNAPSACK – indivisible: a dyn-prog solution maximum cost of a subset of the first k items, where the weight of the subset is at most v The heart of the algorithm: S[k][v] = How to output a solution ? KNAPSACK-INDIVISIBLE(n,c,w,W) 1. init S[0][v]=0 for every v=0,…,W 2. init S[k][0]=0 for every k=0,…,n 3. for v=1 to W do 4. for k=1 to n do 5. S[k][v] = S[k-1][v] 6. if (w k · v) and (S[k-1][v-w k ]+c k > S[k][v]) then 7. S[k][v] = S[k-1][v-w k ]+c k 8. RETURN S[n][W]