1. Hokkaido University Lecture on Information Knowledge Network "Information retrieval and pattern matching" Laboratory of Information Knowledge Network, Division of Computer Science, Graduate School of Information Science and Technology, Hokkaido University Takuya KIDA 1 Lecture on Information knowledge network2011/11/22

2. The 6th Pattern matching on compression text About data compression Motivation and aim of this study Pattern matching on Huffman encoded text Pattern matching on LZW compressed text Unified framework: Collage system Aspect of speeding-up of pattern matching by text compression: BPE compression 2011/11/22 Lecture on Information knowledge network 2

3. Hokkaido University 3 Lecture on Information knowledge network 2011/11/22 About data compression Lossless compressionLossy compression LZ77 Sequitur LZ78 BPELZW JPEG MPEG MP3 used for image and voice data Entropy encoding Huffman encoding Arithmetic encoding Non-universal encoding Run- length BWT Universal encoding Dictionary-based sort-based Grammar-based PPM Statistical reference Managing Gigabytes: Compressing and Indexing Documents and Images, I. H. Witten, A. Moffat, T. C. Bell, Morgan Kaufmann Pub, 1999.

4. Hokkaido University 4 Lecture on Information knowledge network 2011/11/22 Compressed text Original text decompress Ordinal pattern matching machine Pattern matching machine for compressed texts Compressed text Aim of this study Original text Ordinal pattern matching machine

5. Hokkaido University 5 Lecture on Information knowledge network 2011/11/22 Example of application E-mails Directories Schedule tables E2J/J2E dictionaries Business cards Short memos E-books KOJIEN Personal databases We want to pack a lot of data into a small computer such as a mobile phone and PDA as much as possible! Because of small amount of memory, to construct an extra index structure isnt good solution! However, we want to retrieve at high speed! sharp mi110 V601T

6. Hokkaido University 6 Lecture on Information knowledge network 2011/11/22 Difficulty of PM on compressed texts There might hardly be "To decrease capacity, the text data is preserved by compressing it" in the category that personally uses the computer today when the capacity of the hard disk and the memory has grown enough. I have not used this function though the function to reduce capacity putting compression on Windows in each folder is provided. It will be seemed as an advantage none to compress the text data because there are 100 harms though preserving it by compressing it if it is a multimedia data like the image and the voice data, etc. is natural. However, the good policy doing the compression preservation deleting neither for instance a large amount of log file nor past mail data, etc.In a word Document files Compressed document files 01111000011110011111110101101000 1010101001111010001011100110101 11101100011101111110100110101111 1001101001110011011000001111110 1011010111111111000001010010010 10011010 1.The starting position of each codeword is invisible 2.Representation of each string is not unique

7. Hokkaido University 7 Lecture on Information knowledge network 2011/11/22 Search-without-decompress method Search-on-the-fly method Decompress-then-search method Our goal Goal Do pattern matching faster than the above!

8. Hokkaido University 8 Lecture on Information knowledge network 2011/11/22 Lempel-Ziv-Welch (LZW) compression a b ab ab ba b c aba bc abab 1234569 11 42 Text T: Compressed text E(T): LZW is used for UNIX compress command, GIF image format, and so on. T. A. Welch: A technique for high performance data compression, IEEE Comput., Vol.17, pp.8-19, 1984. |D| = O(compressed text length) Let D be the set of strings entered in the dictionary trie D = {a, b, c, ab, ba, bc, ca, aba, abb, bab, bca, abab} Dictionary trie 0 123 c a b 4 b 5 a 9 c 10 a 6 a 7 b 8 b 12 a 11 b D is constructed adaptively 0 123 45910 67812 11 Dictionary trie c a b b a c a a bb a b

9. Hokkaido University 9 Lecture on Information knowledge network 2011/11/22 Move of Aho-Corasick PM machine AC machine for pattern set Π= {aba, ababb, abca, bb} a 01234 5 6 7 9 8 b b a b c a b b {bb} {abca} {aba} {ababb, bb} : goto function : failure function { } : output abababba 012343 45 1 aba Output aba bb ababb Text Current state

10. Hokkaido University 10 Lecture on Information knowledge network 2011/11/22 12343 45 1 Idea for doing pattern matching on LZW texts To simulate the move of AC machine on LZW compressed texts Comp. text a 01234 5 6 7 9 8 b b a b c a b b {bb} {abca} {aba} {ababb, bb} abababba 012 aba Output aba bb ababb Text Current state 12445 4 4 1 T. Kida, M. Takeda, A. Shinohara, M. Miyazaki, and S. Arikawa: Multiple pattern matching in LZW compressed text, Proc. Data Compression Conference, pp. 103-112, IEEE Computer Society, Mar. 1998. : goto function : failure function { } : output

11. Hokkaido University 11 Lecture on Information knowledge network 2011/11/22 Core functions Jump & Output Can we compute two functions Jump and Output well? –function Jump(q, u) simulates the consecutive transitions caused by string u in O(1) time. The domain is Q×D. returns the state number of AC machine –function Output(q, u) reports the occurrences within the string obtained by concatenating the string corresponding to state q and string u in O(r) time. The domain is Q×D. returns the set of pattern IDs It needs O(m|D|) space by a naïve way. It can be realized in O(m 2 +|D|) space! It needs O(m|D|) space by a naïve way. It can be realized in O(m 2 +|D|) space!

12. Hokkaido University 12 Lecture on Information knowledge network 2011/11/22 function Jump δ(q, u) δ(ε, u) if u is a factor of some pattern, otherwise. Jump(q, u) = O(m 3 ) space O(|D|) space Ancestor(N' 1 (q, u'), |u'| |u|) δ(ε, u) if u is a factor of some pattern, otherwise. Jump(q, u) = O(m 2 ) space O(|D|) space O(m 2 ) space Let δ(q,u) be the (extended) state transition function of the AC machine. δ(q,u) returns the state position after making transition from the state q by string u. u is the string corresponding to the nearest ancestor node of u that is also explicit on the generalized suffix trie for P.

13. Hokkaido University 13 Lecture on Information knowledge network 2011/11/22 function Output u ~ the longest prefix of u that is also a suffix of a pattern. A(u) = i,p | p Π, |u|< i <|u|, |p|< i, and u[i |p|+1...i ]=p ~ Output(q, u) = Output(q, u) A(u) ~ q u p1p1p1p1 p2p2p2p2 u~ p1p1p1p1 p2p2p2p2 O(|D|) space O(m 2 ) space Note that state q corresponds to a prefix of some pattern

14. Hokkaido University 14 Lecture on Information knowledge network 2011/11/22 Pseudo code of Kida, et al.[1998]s algorithm PMonLZW (E(T) = u 1 u 2 …u n, Π: pattern set) 1 Construct AC machine and generalized suffix trie for Π; 2 Initialize the dictionary trie for E(T); 3 Preprocess Jump(q,u) and Output(q,u) for any q and u {a pattern π Π factor} 4 l 0; 5 q q 0 ; 6 for i 1…n do 7 for each d,π Output(q, u i ) do 8 report pattern π occurs at position l+d; 9 q Jump(q, u i ); 10 l l + |u i |; 11 Update the dictionary trie; /* enter the string for node u i+1 into D */ 12 Update variables for Jump(q, u i+1 ) and Output(q, u i+1 ); /* compute δ(ε,u i+1 ), A(u i+1 ), u i+1, and |u i+1 | by using its parent info. */ 13 end of for 14 end of for

15. Hokkaido University 15 Lecture on Information knowledge network 2011/11/22 The result of Kida, et al. [1998] The original idea is from –A. Amir, G. Benson, and M. Farach: Let sleeping files lie: Pattern matching in Z-compressed files, J. Computer and System Sciences, Vol.52, pp.299-307, 1996. It simulates KMP on LZW compressed texts By simulating Aho-Corasick AC pattern matching machine, we can do multiple pattern matching. It takes O(m 2 +|D|) time and space for preprocessing. It scans compressed texts in O(n+r) time with O(m 2 +|D|) space for multiple patterns, and reports all the occurrences. This firstly appears in T. Kida, M. Takeda, A. Shinohara, M. Miyazaki, and S. Arikawa: Multiple pattern matching in LZW compressed text, Proc. Data Compression Conference, pp. 103-112, IEEE Computer Society, Mar. 1998. Its Journal ed. Appears in T. Kida, M. Takeda, A. Shinohara, M. Miyazaki, and S. Arikawa: Multiple Pattern Matching in LZW Compressed Text, Journal of Discrete Algorithms, 1(1), pp. 133-158, Hermes Science Publishing, Dec. 2000.

16. Hokkaido University 16 Lecture on Information knowledge network 2011/11/22 Idea for applying bit-parallel technique 1 0 0 0 0 aabaacaabacab abc Pattern P:= a b a c a Text T:= aabac 1 1 0 0 0 0 0 1 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 1 0 0 0 0 0 0 1 Mask bits aab a a a c aa b a c Jump! T. Kida, M. Takeda, A. Shinohara, and S. Arikawa: Shift-And approach to pattern matching in LZW compressed text, Proc. CPM'99, LNCS1645, pp. 1-13, Springer-Verlag, Jul. 1999.

17. Hokkaido University 17 Lecture on Information knowledge network 2011/11/22 Extended state updating function f For any a Σ, u Σ *, S {1,…, m}, we define as follows. – M(a) = { 1< i < m | P[i] = a } – f(S, a) = ((S 1) {1}) M(a) – f(S,ε) = S and f(S, ua) = f( f(S, u), a) – M(u) = f({1,, m}, u) Then, for any u Σ *, S {1,, m}, we define as –f(S, u) = ((S |u|) {1,, |u|}) M(u) O(1) time O(|D|) time and space

18. Hokkaido University 18 Lecture on Information knowledge network 2011/11/22 function Output (Bit-parallel type) Definition –Output(S, u) = { 1 < j < |u| | m S } –U(u) = {1 < j < |u| | i <m and u[1..i] =P[m-i+1..m] } –A(u) = {1 < j < |u| | m < i and u[1-m+1..i]=P } –Output(S, u) =((m S)U(u)) A(u) O(|D|) time and space q u PP (m S)U(u) A(u)

19. Hokkaido University 19 Lecture on Information knowledge network 2011/11/22 The result of Kida, et al. [1999] applied the bit-parallel technique based on Shift-And method to processing of functions Jump and Output to speed up. It uses O(m+|Σ|) time and space for preprocessing. For a given pattern, it scans a given compressed text in O(n+r) time and O(m+|D|) space, and it reports all the occurrences. It excels in the extensibility as well as Shift-And method. –pattern matching for a generalized pattern –pattern matching with allowing k mismatches –multiple pattern matching

20. Hokkaido University 20 Lecture on Information knowledge network 2011/11/22 Achievement of our aim! 0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 51015202530 Pattern length CPU time sec. compress(LZW) + KMP AlphaStation XP1000 (Alpha21264: 667MHz) Tru64 UNIX V4.0F Genbank DNA base sequence 17.1Mbyte T. Kida, et al.[1998] gunzip(LZ77)+ KMP Speeding-up by bit-parallelism[1999] Search-on-the-fly method Search-without-decompress method

21. Hokkaido University Take a breath 21 Lecture on Information knowledge network 2011/11/22 2010.12.24 RG Gundam1/1 @Higashi-Shizuoka Park

22. Hokkaido University 22 Lecture on Information knowledge network 2011/11/22 If … The time for doing pattern matching on the original text The time for doing compressed pattern matching Why do you need compressed PM? Goal 2 A new goal! We have enough storage space now. Why do you compress small data like text documents? × × × ×

23. Hokkaido University 23 Lecture on Information knowledge network 2011/11/22 A new goal! Goal 2 0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 51015202530 Pattern length CPU time sec. Matching by KMP on the original text Overwhelmingly faster! compress(LZW) + KMP AlphaStation XP1000 (Alpha21264: 667MHz) Tru64 UNIX V4.0F Genbank DNA base sequence 17.1Mbyte T. Kida, et al.[1998] gunzip(LZ77)+ KMP Speeding-up by bit-parallelism[1999] Search-on-the-fly method Search-without-decompress method

24. Hokkaido University 24 Lecture on Information knowledge network 2011/11/22 After substitutions Text ABABCDEBDEFABDEABC GGCHBHFGHGC GIHBHFGHI GGCDEBDEFGDEGC G H I 9 G H I AB DE GCGC dictionary 18 Size = 1 byte Byte Pair Encoding (BPE) method

25. Hokkaido University 25 Lecture on Information knowledge network 2011/11/22 Achievement of Goal 2 AlphaStation XP1000 (Alpha21264: 667MHz) Tru64 UNIX V4.0F Medline English text 60.3Mbyte 51015202530 Pattern length 0.0 0.3 0.4 0.5 0.8 0.1 0.2 0.6 0.7 CPU time sec. Matching by KMP on the original text Compressed PM on BPE (KMP type) Search-without-decompress method Agrep on the original text Compressed PM on BPE (BM type) Shibata, et al. (2000) Search-without-decompress method The fastest in the previous

26. Hokkaido University 26 Lecture on Information knowledge network 2011/11/22 Text compressed by BPE Text compressed by LZSS ordinal Text compressed by LZW The original uncompressed text for LZSSfor BPEfor LZW GOAL 1342 Low compression Medium compression High compression …but its the most suitable for PM! Summarize the above…

27. Hokkaido University 27 Lecture on Information knowledge network 2011/11/22 Paradigm shift 1 Develop pattern matching algorithms for each compression methods Choosing a suitable compression enables us to accelerate pattern matching! Develop a novel compression method which is suitable for pattern matching!

28. Hokkaido University Data compression methods for PM Dense coding type –[ETDC] Nieves R. Brisaboa, Eva Lorenzo Iglesias, Gonzalo Navarro, and Jose R. Parama: An efficient compression code for text databases, In ECIR2003, pp. 468-481, 2003. –[SCDC] Nieves R. Brisaboa, Antonio Farina, Gonzalo Navarro, and Maria F. Esteller: (s, c)-dense coding: An optimized compression code for natural language text databases, In SPIRE2003, pp. 122-136, 2003. –[FibC] Shmuel Tomi Klein and Miri Kopel Ben-Nissan: Using fibonacci compression codes as alternatives to dense codes, In DCC2008, pp. 472-481, 2008. –[SVVC] Nieves R. Brisaboa, Antonio Farina, Juan-Ramon Lopez, Gonzalo Navarro, and Eduardo R. Lopez: A new searchable variable-to-variable compressor, In DCC2010, pp. 199-208, 2010. VF coding type (including grammar-based compressions) –[BPEX] Shirou Maruyama, Yohei Tanaka, Hiroshi Sakamoto, and Masayuki Takeda: Context-sensitive grammar transform: Compression and pattern matching, In SPIRE2008, LNCS5280, pp. 27-38, Nov. 2008. –[DynC] Shmuel T. Klein and Dana Shapira: Improved variable-to-fixed length codes, In SPIRE2008, pp. 39-50, 2009. –[STVF] Takashi Uemura, Satoshi Yoshida, Takuya Kida, Tatsuya Asai, and Seishi Okamoto: Training parse trees for efficient VF coding, In SPIRE2010, pp. 179-184, 2010. 28 Lecture on Information knowledge network 2011/11/22

29. Hokkaido University 29 Lecture on Information knowledge network 2011/11/22 Paradigm shift 2 We use the data compression technology to reduce the cost for storing and transferring the data. We can speed up pattern matching by compressing the data. Break difficulties of various processing by using the compression technology!

30. Hokkaido University 30 Lecture on Information knowledge network 2011/11/22 Doing something by using compression Speeding up the calculation of similarity between two long strings by compression technique. –A Sub-quadratic Sequence Alignment Algorithm for Unrestricted Cost Matrices, M. Crochemore, G. M. Landau, and M. Ziv-Ukelson, Proceeding of 13th Symposium on Discrete Algorithm, pp.679-688, 2002 Processing a very huge graph structure on memory at high speed by compression technique. –Shinichi Nakano Gunma University Graph compression with query support Their method can represent a triangulated planar graph in 2m+o(n) bit and moreover can support some queries on it. Speeding up the query processing for XML data by compression technique. – Tetsuya Maita and Hiroshi Sakamoto Kyushu Institute of Technology

31. Hokkaido University 31 Lecture on Information knowledge network 2011/11/22 The 6th summary Pattern matching algorithms on compressed texts –Pattern matching on Huffman encoded text automaton with synchronization –Pattern matching on LZW compressed text simulating the move of KMP(AC) on the compressed text Unified framework: Collage system –A formal system to represent a text compressed by lexicographical compression method –We have clarified what kind of compression methods are suitable for pattern matching. Aspect of speed-up pattern matching by compression –BPE compression: it has low compression ratio, but it can speed up pattern matching –Our experimental results showed that we could do pattern matching faster than doing on the original texts A big paradigm shift caused –The data compression technology can be used in the other purposes rather than reducing the data size The next theme (which is the final topic of "Information retrieval and pattern matching) –Various topics I didnt mention about