1 A Fast Algorithm for Multi-Pattern Searching Sun Wu, Udi Manber Tech. Rep. TR94-17,Department of Computer Science, University of Arizona, May 1994

2 Outline Introduction Boyer-Moore algorithm review Fast algorithm for Multi-Pattern Search Preprocessing Stage Scanning Stage Performance Experiments Conclusion

3 Introduction Given a algorithm to find all occurrences of all the pattern of P in T. P={p 1, p 2,......, p k } be the ser of patterns, which are strings of characters from a fixed alphabet Σ. T = t 1, t 2,...., t N be a large text, consisting of character from Σ.

4 Boyer-Moore algorithm review Symbol used: Σ : the set of alphabets patlen : the length of pattern m : the last m characters of pattern matched char : the mismatched character m ……… string pattern char

5 Bad Character Heuristic Observation 1: If the char doesn’t occur in pat: Pattern Shift : j character String pointer shift: patlen character Example:......a c d a b b a c d e a f e c a text string ptr a b c e pat

6 Bad Character Heuristic (cont.) Observation 2: If the char occur in the pattern The rightmost char in pattern in position δ 1 [char] and the pointer to the pattern is in j If j < δ 1 [char] we shift the pattern right by 1 If j > δ 1 [char] we shift the pattern right by j- δ 1 [char] We say δ 1 is SHIFT table

7 Bad Character Heuristic (cont.) Example: j < δ 1 [char]......A C F D B A D A E C A D A E text j  δ 1 [char]......A C F D B A D A E C A D A E text string ptr δ1[A] = 7 and j = 4 shift pattern right by 1 j D A E C E C A string ptr D A E C E C j δ1[A] = 2 and j = 4 shift pattern right by 2

8 Multi-Pattern Searching Instead looking at character from text one by one, we consider them in blocks of size B. A good value of B is in the order of log c 2M. In practice, we use either B=2 or B=3. M is the total size of all patterns. c is the size of the alphabet. text size = B

9 Multi-Pattern Searching (cont.) Preprocessing Stage built three tables for the set of patterns: SHIFT table : like Boyer-Moore’s Shift table with little different. HASH table and PREFIX table: used when the shift value = 0.

10 Preprocessing Stage First Compute the minimum length m of a pattern, and consider first m character of each pattern. SHIFT table contains all possible string of size B Table size is c B We can use hash function to compress table.

11 SHIFT table Let X = x 1 x x B be the B characters in the text, and X is mapped into i’th entry of SHIFT table. Case 1: X doesn’t appear as a substring in P, we shift text m-B+1 characters. BAABDACBAD text ADBA m =4, B =2 so we shift pattern m-B+1

12 SHIFT table (cont.) Case 2: X appears in some patterns:To find the rightmost occurrence of X in any of the patterns. X ends at position q of Pj, and q is the largest in all possible patterns. We shift text m-j characters-> SHIFT[i] = m-j. CAAB ACAD DBECDACBAG text

13 SHIFT table (cont.) The value of SHIFT table are the largest possible safe value for shifts. To do pre-scan all of the patterns, set SHIFT value min(current value, m-j) Initial value is m-B+1 We can map several different strings into the same entry.

14 HASH table When SHIFT[i] = 0, we match some patterns. HASH[i] records the pointer PAT_POINT which point to the patterns. …….. list of PAT_POINT patterns which sorted by the hash value of the last B characters of each pattern.

15 HASH table (cont.) HASH[i] = p, point to the beginning of the list of patterns whose hash value mapped to h. To find the end of this list, we keep incrementing this pointer until it’s value equal to the value in HASH[i+1]

16 PREFIX table Nature language isn’t random. The suffix “ion”, “ing” is common in English Text. It may appear in several of the patterns. We use PREFIX table to speed up this process. Mapping the first B’ characters of all patterns into Prefix function. It can filter patterns whose suffix is the same but whose prefix is different.

17 Scanning Stage while (text <= textend) { h = Huchfunct(B); /* The hash function (we use Hbits=5) */ shift = SHIFT[h]; if (shift == 0) { text_prefix = (*(text-m+1)<<8) + *(text-m+2); p = HASH[h]; p_end = HASH[h+1]; while (p++ < p_end) { if(text_prefix != PREFIX[p]) continue; px = PAT_POINT[p]; qx = text-m+1; while (*(px++) == *(qx++)); if (*(px-1) == 0) { /* 0 indicates the end of a string */ report a match } shift = 1; } text += shift; } 1.Compute the hash value h based on the B character from the text Text possible shift is zero. Some match happened. Check for each p HASH[i] <= p < HASH[i+1] where PREFIX[p] = text_prefix.

18 Performance The SHIFT table is constructed in O(M) M = m * P B = log c 2M c B = c  logc2M   2Mc

19 Performance (cont.) Lemma: The probability of random string of size B leads to a shift value of i, is <=1/2m Prof: 1. P = M/m strings lead to shift value of i 2. the number of possible strings of size B is 2M at least

20 Performance (cont.) Lemma implies that the expected value of shift is >= m/2 total amount of non-zero shift is O(BN/m) shift = 0, the amount of cost is O(m) * O(1/2m) The total amount is O(BN/m)

21 Experiment

22 Experiment (cont.)

23 Conclusion This algorithm use three table : SHIFT, HASH, Prefix, to save scanning time. Preprocessing stage cost is low. It can use in many application, such as file search in database,