1. 1 Foundations of Software Design Lecture 26: Text Processing, Tries, and Dynamic Programming Marti Hearst & Fredrik Wallenberg Fall 2002

2. 2 Problem: String Search Determine if, and where, a substring occurs within a string

3. 3 Approaches/Algorithms: Brute Force Rabin-Karp Tries Dynamic Programming

4. 4 “Brute Force” Algorithm

5. 5 Worst-case Complexity

6. 6 Best-case Complexity, String Found

7. 7 Best-case Complexity, String Not Found

8. 8 Rabin-Karp Algorithm Calculate a hash value for –The pattern being searched for (length M), and –Each M-character subsequence in the text Start with the first M-character sequence –Hash it –Compare the hashed search term against it –If they match, then look at the letters directly Why do we need this step? –Else go to the next M-character sequence (Note 1: Karp is a Turing-award winning prof. in CS here!) (Note 2: CS theory is a good field to be in because they name things after you!)

9. 9 Karp-Rabin: Looking for 31415 31415 mod 13 = 7 Thus compute each 5-char substring mod 13 looking for 7 2 3 5 9 0 2 3 1 4 1 5 2 6 7 3 9 9 2 1 ----------- 8 2 3 5 9 0 2 3 1 4 1 5 2 6 7 3 9 9 2 1 ----------- 9 2 3 5 9 0 2 3 1 4 1 5 2 6 7 3 9 9 2 1 ----------- 3 2 3 5 9 0 2 3 1 4 1 5 2 6 7 3 9 9 2 1 ----------- 7 Found 7! Now check the digits

10. 10 Rabin-Karp Algorithm Worst case time? –N is length of the string –O(N) if the hash function is chosen well http://orca.st.usm.edu/~suzi/stringmatch/rk_alg.html http://www.mills.edu/ACAD_INFO/MCS/CS/S00MCS125/String.Matching.Algorithms /animations.htmlhttp://www.mills.edu/ACAD_INFO/MCS/CS/S00MCS125/String.Matching.Algorithms /animations.html (Note 1: Karp is a Turing-award winning prof. in CS here!) (Note 2: CS theory is a good field to be in because they name things after you!)

11. 11 Tries A tree-based data structure for storing strings in order to make pattern matching faster Main idea: –Store all the strings from the document, one letter at a time, in a tree structure –Two strings with the same prefix are in the same subtree Useful for IR prefix queries –Search for the longest prefix of a query string Q that matches a prefix of some string in the trie –The name comes from Information Retrieval

12. 12 Trie Example The standard trie over the alphabet {a,b} for the set {aabab, abaab, babbb, bbaaa, bbbab}

13. 13 A Simple Incremental Algorithm To build the trie, simple add one string at a time Check to see if the current character matches the current node. If so, move to the next character If not, make a new branch labeled with the mismatched character, and then move to the next character Repeat

14. 14 Trie-growing Algorithm a r i d e l l c k t o p h e a r s e e u b y l l l l buy bell hear see bid bear stop bull sell stock

15. 15 Tries, more formally The path from the root of T to any node represents a prefix that is equal to the concatenation of the characters encountered while traversing the path. –An internal node can have from 1 to d children where d is the size of the alphabet. The previous example is a binary tree because the alphabet had only 2 letters –A path from the root of T to an internal node i corresponds to an i-character prefix of a string S –The height of the tree is the length of the longest string –If there are S unique strings, T has S leaf nodes –Looking up a string of length M is O(M)

16. 16 Compressed Tries Compression is done after the trie has been built up; can’t add more items.

17. 17 Compressed Tries Also known as PATRICIA Trie –Practical Algorithm To Retrieve Information Coded In Alphanumeric –D. Morrison, Journal of the ACM 15 (1968). Improves a space inefficiency of Tries Tries to remove nodes with only one child (pardon the pun) The number of nodes is proportional to the number of strings, not to their total length –But this just makes the node labels longer –So this only helps if an auxiliary data structure is used to actually store the strings –The trie only stores triplets of numbers indicating where in the auxiliary data structure to look

18. 18 Compressed Trie s hear$ar e b llidll u y to pckll e e

19. 19 Suffix Tries Regular tries can only be used to find whole words. What if we want to search on suffixes? –build*, mini* Solution: use suffix tries where each possible suffix is stored in the trie Example: minimize nimize mi i zemize ze e nimize Find:imi i m i

20. 20 Dynamic Programming Used primarily for optimization problems. –Not just a good solution, but an optimal one. Brute force algorithms –Try every possibility –Guarantee finding the optimal solution –But inefficient DP requires a certain amount of structure, namely: –Simple Subproblems (and simple break-down) –Global optimum is a composition of subproblem optimums –Subproblem Overlap: optimal solutions to unrelated problems can contain subproblems in common. –In other words, can re-use the results of solving the subproblem

21. 21 Longest Common Subsequence LCS: find the longest string S that is a subsequence of both X and Y, where X is of length n Y is of length m Example: what is the LCS of supergalactic galaxy (The characters do not have to be contiguous)

22. 22 * Longest Common Subsequence Dynamic Programming Applied to LCS Problem Let’s compare: X = [GTG] X[0…i] Y = [CGATG] Y[0…j] We represent the longest subsequence as L[i,j]

23. 23 Dynamic Programming for LCS Note that the longest string of X and Y (L[i,j]) must be equal to the longest string of... X[0…i-1] = [GT] (removing the last G) Y[0…j-1] = [CGAT] (removing the last G) … plus 1, since the matching Gs at X i,Y j will increase the length by one.

24. 24 Dynamic Programming for LCS If X i,Y j had NOT matched, L[i,j] would have to be equal to the longest string in L[i-1,j] or L[i,j-1]. If this is true for L[i,j], it must be true for all L. We know that L[-1,-1] = 0 (since both strings are empty) Finally we know that L[i,j] cannot be larger than max(i,j)+1

25. 25 Dynamic Programming for LCS L[0,1] = 1 (X 0,Y 1 does match… L[-1,0] + 1) 1 L[0,0] = 0 (X 0,Y 0 doesn’t match… max of L[-1,0] and L[0,-1]) 0 0 0 111 1 2 2 3 21 11 For each position, take the max of L[i-1,j] or L[i,j-1] Add 1 when a new match is found.

26. 26 Dynamic Programming Running Time/Space: –Strings of length m and n –O(mn) –Brute force algorithm: 2 m subsequences of x to check against n elements of y: O(n 2 m )

27. 27 Dynamic Programming vs. Greedy Algorithms Sometimes they are the same. Sometimes not What makes an algorithm greedy? –Globally optimal solution can be obtained by making locally optimal choices Dynamic Programming –Solves subproblems, that can be re-used –Trickier to think of –More work to program

28. 28 From www.cs.virginia.edu/~luebke/cs332.fall00/lecture23.ppt edu/~bodik/cs536.html Greedy Vs. Dynamic Programming: The famous knapsack problem: –A thief breaks into a museum. Fabulous paintings, sculptures, and jewels are everywhere. The thief has a good eye for the value of these objects, and knows that each will fetch hundreds or thousands of dollars on the clandestine art collector’s market. But, the thief has only brought a single knapsack to the scene of the robbery, and can take away only what he can carry. What items should the thief take to maximize the haul?

29. 29 From www.cs.virginia.edu/~luebke/cs332.fall00/lecture23.ppt edu/~bodik/cs536.html The Knapsack Problem More formally, the 0-1 knapsack problem: –The thief must choose among n items, where the ith item worth v i dollars and weighs w i pounds –Carrying at most W pounds, want to maximize value Note: assume v i, w i, and W are all integers “0-1” b/c each item must be taken or left in entirety A variation, the fractional knapsack problem: –Thief can take fractions of items –Think of items in 0-1 problem as gold ingots, in fractional problem as buckets of gold dust

30. 30 From www.cs.virginia.edu/~luebke/cs332.fall00/lecture23.ppt edu/~bodik/cs536.html The Knapsack Problem: Optimal Substructure Both variations exhibit optimal substructure To show this for the 0-1 problem, consider the most valuable load weighing at most W pounds –If we remove item j from the load, what do we know about the remaining load? –The remainder must be the most valuable load weighing at most W - w j that the thief could take from museum, excluding item j

31. 31 From www.cs.virginia.edu/~luebke/cs332.fall00/lecture23.ppt edu/~bodik/cs536.html Solving The Knapsack Problem The optimal solution to the fractional knapsack problem can be found with a greedy algorithm The optimal solution to the 0-1 problem cannot be found with the same greedy strategy –Greedy strategy: take in order of dollars/pound –Example: 3 items weighing 10, 20, and 30 pounds, knapsack can hold 50 pounds Suppose item 2 is worth $100. Assign values to the other items so that the greedy strategy will fail

32. 32 From www.cs.virginia.edu/~luebke/cs332.fall00/lecture23.ppt edu/~bodik/cs536.html The Knapsack Problem: Greedy Vs. Dynamic The fractional problem can be solved greedily The 0-1 problem cannot be solved with a greedy approach –It can, however, be solved with dynamic programming