1. Online Sequential Prediction via Incremental Parsing: The Active LeZi Algorithm Source: IEEE Intelligent Systems, 22(1), pp. 52-58, 2007 Authors: Karthik Gopalratnam and Diane J. Cook Reporter: Yuen-Kuei Hsueh

2. December 16, 20092 Outline  Introduction  Background  LZ78 and LeZi Update  Active LeZi  Probability assignment  Evaluations

3. December 16, 20093 Introduction  The concept of predictor time0123tt+1 Unknown symbolObserved symbols Predict unknown symbol on the basis of the observed symbols symbol a b c a b ?

4. December 16, 20094 Introduction  Scenario clock 192021222324 Observed events event Light on TV on nop TV off nop Light off 12 nop Dec 1, 2009 clock 192021222324 Observed events event Light on TV on nop TV off nop Light off 12 nop Dec 2, 2009 clock 192021222324 Observed events event Light on TV on nop TV off nop Light off 12 nop Dec 3, 2009 clock 192021222324 Observed events event Light on TV on nop TV off nop Light off 12 nop Dec 4, 2009 clock 192021222324 Unknown eventObserved events event Light on TV on nop TV off nop ? 12 Dec 5, 2009

5. December 16, 20095 Background  Good text compression algorithms are also good predictors LZ78 LeZi Update Active LeZi  According to information theory, a predictor model with an order that grows at a rate approximating the source’s entropy rate is an optimal predictor

6. December 16, 20096 LZ78 - Pseudocode loop wait for next symbol v if ((w.v) in dictionary): w = w.v else add (w.v) to dictionary w = null increment frequency for every possible prefix of phrase forever

7. December 16, 20097 LZ78 - Example w = null loop wait for next symbol v=a if ((w.v) in dictionary): w = w.v else add (w.v=a) to dictionary w = null increment frequency for every possible prefix of phrase forever aa ab Input: dictionary a1 w = null loop wait for next symbol v=a if ((w.v=a) in dictionary): w = w.v= a else add (w.v) to dictionary w = null increment frequency for every possible prefix of phrase forever w = null loop wait for next symbol v=a if ((w.v) in dictionary): w = w.v else add (w.v=aa) to dictionary w = null increment frequency for every possible prefix of phrase {a} forever dictionary a2 aa1 w = null loop wait for next symbol v=b if ((w.v) in dictionary): w = w.v else add (w.v=b) to dictionary w = null increment frequency for every possible prefix of phrase forever dictionary a2 aa1 b1 ab w = null loop wait for next symbol v=a if ((w.v=a) in dictionary): w = w.v=a else add (w.v=b) to dictionary w = null increment frequency for every possible prefix of phrase forever w = null loop wait for next symbol v=b if ((w.v) in dictionary): w = w.v else add (w.v=ab) to dictionary w = null increment frequency for every possible prefix of phrase {a} forever dictionary a3 aa1 b1 ab1

8. December 16, 20098 LZ78 - Example  Trie (order-1 Markov model) ^ a(3) b(1) a(1) 0-order 1-order b(1)

9. December 16, 20099 LZ78 and LeZi Update - drawback  Slow convergence e.g. In above example string (aaabab), 4 th symbol (b) and 5 th and 6 th (ab) does not form phrase bab  LeZi Update Overcome LZ78 slow convergence problem, but attempt to recapture information lost across phrase boundaries

10. December 16, 200910 Active LeZi initialize Max_LZ_length = 0 phrases = 0 loop wait for next symbol v if ((w.v) in dictionary): w := w.v else add (w.v) to dictionary if(length(w.v) > Max_LZ_length) // update Max_LZ_length if necessary Max_LZ_length = length(w.v) w := null add v to window if (length(window) > Max_LZ_length) delete window[0] Update frequencies of all possible contexts within window that includes v forever

11. December 16, 200911 Active LeZi  Window length equals to the length of and LZ78 parsing’s longest phrase  e.g. aaababbbbbaabccddcbaaaa Window length = 0 Window = {a, a, a} The window will update the following phrases’ frequency {aaa, aa, a}

12. December 16, 200912 LeZi Update vs. Active LeZi  Convergence Test A pattern consisting of 200 random symbols 100 symbols Repeated pattern 100 times Embedded the data with noise by replacing 20% ex: A pattern consisting of 5 random symbols 3 symbols {a, b, c} Repeated 6 times =>ababc ababc ababc ababc ababc ababc

13. December 16, 200913 LeZi Update vs. Active LeZi

14. December 16, 200914 Probability assignment  Use Prediction by Partial Match (PPM)  Example: Input = aaababbbbbaabccddcbaaaa Window = {a, a, a} Using all suffix of window, except window itself = {aa, a, null} Predict the probability of symbol a

15. December 16, 200915 Probability assignment for aa phrase a(10)a(5) -> a(2) a(10)a(5) -> b(1) a(10)a(5) -> null(2) 2/5 ^(2) 2/5+2/5

16. December 16, 200916 Probability assignment for a phrase a(10) -> a(5) a(10) -> b(3) a(10) -> null(2) 5/10 2/10 ^(2) 2/5+2/5*(5/10+2/10)

17. December 16, 200917 Probability assignment for null phrase null(23) ->a (10) 10/23 2/5+2/5*(5/10+2/10*(10/23))

18. December 16, 200918 Probability assignment  The probability that the next symbol is a: 2/5+2/5*(5/10+2/10*(10/23))=0.59 b: 1/5+2/5*(3/10+2/10*(8/23))=0.37 c: 0/5+2/5*(0/10+2/10*(3/23))=0.03 d: 0/5+2/5*(0/10+2/10*(2/23))=0.02

19. December 16, 200919 Evaluations DatasetDevice numberAccuracy(%) Top five predictions Accuracy (%) MavLab647%89% MavHome502%20%

20. December 16, 200920 Conclusion  This article is a good start point for persons who want to gain the knowledge about universal predictor (without domain knowledge)