1. Information Distance More Applications

2. 1. Information Distance from a Question to an Answer

3. Question & Answer  Practical concerns: Partial matching does not satisfy triangle inequality. When x is very popular, and y is not, x contains a lot of irrelevant information w.r.t. y, then C(x|y) << C(y|x), and d(x,y) prefers y. d max does not satisfy universality. Neighborhood density -- there are answers that are much more popular than others. Nothing to compress: a question and an answer.

4. Partial matching Triangle inequality does not hold: d(man,horse) ≥ d(man, centaur) + d(centaur, horse)

5. Separate Irrelevant Information  In max theory, we wanted smallest p, converting x,y:  Now let’s remove redundant information from p:  We now wish to minimize q+s+t. st x py x qy

6. The Min Theory (Li, Int’l J. TCS, 2007, Zhang et al, KDD’2007)  E min (x,y) = smallest program p needed to convert between x and y, but keeping irrelevant information out from p.  Formally: E min (x,y) =min{|p| : U(x,p,r)=y, U(y,p,q)=x, |p|+|q|+|r| ≤ E(x,y) }  All other development similar to E(x,y). Define: min {C(x|y), C(y|x) } d min (x,y) = ----------------------- min {C(x),C(y)} Fundamental Theorem II: E min (x,y) = min { C(x|y), C(y|x) }

7. Other properties Theorem 1. d min (x,y) ≤ d max (x,y) Theorem 2. d min (x,y) is universal, does not satisfy triangle inequality is symmetric has required density properties: good guys have more neighbors.

8. How to approximate d max (x,y), d min (x,y)  Each term C(x|y) may be approximated by one of the following: 1. Compression. 2. Shannon-Fano code (Cilibrasi, Vitanyi): an object with probability p may be encoded by –logp + 1 bits. 3. Mixed usage of (1) and (2) – in question and answer application. This is especially useful for Q&A systems.

9. Shannon-Fano Code  Consider n symbols 1,2, …, N, with decreasing probabilities: p 1 ≥ p 2 ≥, … ≥ p n. Let P r =∑ i=1..r p i. The binary code E(r) for r is obtained by truncating the binary expansion of P r at length |E(r)| such that - log p r ≤ |E(r)| < -log p r +1  Highly probably symbols are mapped to shorter codes, and 2 -|E(r)| ≤ p r < 2 -|E(r)|+1  Near optimal: Let H = -∑ r p r logp r --- the average number of bits needed to encode 1…N. Then we have - ∑ r p r logp r ≤ H < ∑ r (-log p r +1)p r = 1 - ∑ r p r logp r

10. Query-Answer System X. Zhang, Y. Hao, X. Zhu, M. Li, KDD’2007  Adding conditions to normalized information distance, we built a Query-Answer system.  The information distance naturally measures Good pattern matches – via compression Frequently occurring items – via Shannon-Fano code Mixed usage of the above two.

11. Some comparisons 1.Benchmark is based on a common factoid QA test set of 109 questions that are provided by Text Retrieval Conference (TREC) sponsored by National Institute of Standard and Technology (NIST). 2.MRAR = (∑C(i)/i)/N = (C1 + C2*0.5 + C3*0.33 + C4*0.25 + C5*0.2)/109

12. 2. Parameter-Free Data Mining (Keogh, Lonadi, Ratanamahatana, KDD’04)  Most data mining algorithms require setting many input parameters. Parameter-laden methods have 2 dangers: Wrong parameter setting leads to errors. Over fitting causes more problems.  Data mining algorithms should have ideally no parameters – thus, no prejudices, expectations, or presumptions.  Compared (a variant of) information distance with every time serious distance (51 of them) appeared in SIGKDD, SIGMOD, ICDM, ICDE, VLDB, ICML, SSDB, PKDD, PADDD during the previous decade.

13. Experiment on 18 pairs of time series (length 1000 each): Q = correct /total Inf. Dist. Q=1 ¾ of measures, Q=0 Best of them, HMM, Q=0.33 Data cover: finance, science, medicine, industry Data: all in SAX format

14. Anomaly detection (KLR, KDD’04) – algorithm: use divide and conquer to find a region that have large inf. dist. from other parts. * All other methods produced wrong results – different from cardiologists.

15. Anomaly Detection (KLR KDD’04)

16. 3. Identifying Multiword Expressions (Fan Bu, Tsinghua University)  Multiword expressions (MWEs) appear frequently and ungrammatically in English.  An MWE is a sequence of neighboring words whose meaning cannot be derived from the meaning of its components. Example: Kolmogorov complexity  Automatically identifying MWEs is a major challenge in computational linguistics.

17. Distance from an n-gram to its semantics  Given an n-gram, let me define the “semantics” of the n-gram to be the set of all web pages containing all the words in the n-gram.  The (plain) information distance simplifies to: D max (n-gram g, its semantics) = log (#pages(g)/#pages(g’s semantics))

18. Experiment on 1529 idioms, 1798 compositional phrases

19. Complex name entity extraction:

20. 4. Texture classification (Campana & Keogh, 2010) Why are we interested in this

21. Their method  Used, as information distance: d(x,y) = [K(x|y) + K(y|x) ] / K(xy)  MPEG-1 format for images x, y. To do K(x|y), they created a movie of a pair of frames x and y. They used MPEG video compression (lossy).

22. Note that the algorithm has no access to color or shape information, this clustering is based only on texture. Dictionnaire D'Histoire Naturelle by Charles Orbigny. 1849

23. Ornaments from the Hand-Press period. The particularity of this period is the use of block of wood to print the ornaments on the books. The specialist historians want to record the individual instances of ornament occurrence and to identify the individual blocks. This identification work is very useful to date the books and to authenticate outputs from some printing-houses and authors. Indeed, numerous editions published in the past centuries do not reveal on the title page their true origin. The blocks could be re-used to print several books, be exchanged between the printing-houses or duplicated in the case of damage. Mathieu Delalandre The algorithm can handle very subtle differences.

24. Clovis Egyptian

27. 5. Gene Expression Dynamics (Nykter et al, PNAS, 2008)  Macrophage (white blood cells)  94 microarrays, 9941 differentially expressed genes.  Computed Normalized information distance between two time-point measurements.  Observed the underlying dynamical network of the macrophage exhibits criticality.  Somebody figure out what this is about – and present in class.

28. 6. Tree from metabolic network (Nykter et al, Phy. Rev. Lett. 2008)  Metabolic networks of 107 organisms from KEGG.  Normalized information distance is computed between each pair and a tree was generated.

29. Red – Bacteria Blue – archaea Green -- eukaryotes

30. Phylogenetic Compression (Ane-Sanderson, Syst. Biol. 2005)  Several interesting questions from bioinformatics: Is the parsimony tree the best tree? DNA sequence compression by sequence only can be optimal?  The authors propose a scheme to encode the tree and data simultaneously to minimize the descriptive complexity of tree(s) plus data. Better compression More economical than the parsimony tree.