1. Hypernetwork Models of Memory Hypernetwork Models of Memory Byoung-Tak Zhang Biointelligence Laboratory School of Computer Science and Engineering Brain Science, Cognitive Science, Bioinformatics Programs Seoul National University Seoul 151-742, Korea btzhang@cse.snu.ac.kr http://bi.snu.ac.kr/

2. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 2 Signaling Networks

3. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 3 Protein-Protein Interaction Networks [Jeong et al., Nature 2001]

4. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 4 Regulatory Networks [Katia Basso, et al., Nature Genetics 37, 382 – 390, 2005] Transcription Factor

5. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 5 Molecular Networks

6. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 6 x1 x2 x3 x4 x5 x6 x7 x8x9 x10 x11 x12 x13 x14 x15 The “Hyperinteraction” Model of Biomolecules Vertex: Genes Proteins Neurotransmitters Neuromodulators Hormones Edge: Interactions Genetic Signaling Metabolic Synaptic [Zhang, DNA-2006] [Zhang, FOCI-2007]

7. The Hypernetworks

8. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 8 Hypergraphs A hypergraph is a (undirected) graph G whose edges connect a non-null number of vertices, i.e. G = (V, E), where V = {v 1, v 2, …, v n }, E = {E 1, E 2, …, E n }, and E i = {v i1, v i2, …, v im } An m-hypergraph consists of a set V of vertices and a subset E of V [m], i.e. G = (V, V [m] ) where V [m] is a set of subsets of V whose elements have precisely m members. A hypergraph G is said to be k-uniform if every edge E i in E has cardinality k. A hypergraph G is k-regular if every vertex has degree k. Rem.: An ordinary graph is a 2-uniform hypergraph.

9. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 9 An Example Hypergraph v5 v1 v3 v7 v2 v6 v4 G = (V, E) V = {v1, v2, v3, …, v7} E = {E1, E2, E3, E4, E5} E1 = {v1, v3, v4} E2 = {v1, v4} E3 = {v2, v3, v6} E4 = {v3, v4, v6, v7} E5 = {v4, v5, v7} E1 E4 E5 E2 E3

10. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 10 Hypernetworks A hypernetwork is a hypergraph of weighted edges. It is defined as a triple H = (V, E, W), where V = {v 1, v 2, …, v n }, E = {E 1, E 2, …, E n }, and W = {w 1, w 2, …, w n }. An m-hypernetwork consists of a set V of vertices and a subset E of V [m], i.e. H = (V, V [m], W) where V [m] is a set of subsets of V whose elements have precisely m members and W is the set of weights associated with the hyperedges. A hypernetwork H is said to be k-uniform if every edge E i in E has cardinality k. A hypernetwork H is k-regular if every vertex has degree k. Rem.: An ordinary graph is a 2-uniform hypergraph with w i =1. [Zhang, DNA-2006]

11. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 11 x1 x2 x3 x4 x5 x6 x7 x8x9 x10 x11 x12 x13 x14 x15 The Hypernetwork Memory [Zhang, DNA12-2006]

12. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 12 x1 =1 x2 =0 x3 =0 x4 =1 x5 =0 x6 =0 x7 =0 x8 =0 x9 =0 x10 =1 x11 =0 x12 =1 x13 =0 x14 =0 x15 =0 y = 1 x1 =0 x2 =1 x3 =1 x4 =0 x5 =0 x6 =0 x7 =0 x8 =0 x9 =1 x10 =0 x11 =0 x12 =0 x13 =0 x14 =1 x15 =0 y = 0 x1 =0 x2 =0 x3 =1 x4 =0 x5 =0 x6 =1 x7 =0 x8 =1 x9 =0 x10 =0 x11 =0 x12 =0 x13 =1 x14 =0 x15 =0 y =1 4 sentences (with labels) x4x4 x 10 y=1x1x1 x4x4 x 12 y=1x1x1 x 10 x 12 y=1x4x4 x3x3 x9x9 y=0x2x2 x3x3 x 14 y=0x2x2 x9x9 x 14 y=0x3x3 x6x6 x8x8 y=1x3x3 x6x6 x 13 y=1x3x3 x8x8 x 13 y=1x6x6 1 2 3 1 2 3 x1 =0 x2 =0 x3 =0 x4 =0 x5 =0 x6 =0 x7 =0 x8 =1 x9 =0 x10 =0 x11 =1 x12 =0 x13 =0 x14 =0 x15 =1 y =1 4 x 11 x 15 y=0x8x8 4 Round 1 Round 2 Round 3

13. An Example: Hypernetwork Model of Linguistic Memory

14. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 14 The Language Game Platform

15. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 15 A Language Game ? still ? believe ? did this.  I still can't believe you did this. We ? ? a lot ? gifts.  We don't have a lot of gifts.

16. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 16 Text Corpus: TV Drama Series Friends, 24, House, Grey Anatomy, Gilmore Girls, Sex and the City 289,468 Sentences (Training Data) 700 Sentences with Blanks (Test Data) I don't know what happened. Take a look at this. … What ? ? ? here. ? have ? visit the ? room. …

17. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 17 Step 1: Learning a Linguistic Memory k = 2 k = 3 k = 4 … Hypernetwork Memory

18. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 18 Step 2: Recalling from the Memory Heismybestfriend bestfriendmybestismyHeis Heisa?friend HeisastrongboyStrongfriendlikesprettygirl Heis aastrong boyStrongfriend likes strongprettygirlStrongfriendastrongbestfriend Heisastrongfriend X7 X6 X5 X8 X1 X2 X3 X4 Recall Self-assembly Storage

19. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 19 The Language Game: Results Why ? you ? come ? down ?  Why are you go come on down here ? think ? I ? met ? somewhere before  I think but I am met him somewhere before ? appreciate it if ? call her by ? ?  I appreciate it if you call her by the way I'm standing ? the ? ? ? cafeteria  I'm standing in the one of the cafeteria Would you ? to meet ? ? Tuesday ?  Would you nice to meet you in Tuesday and ? gonna ? upstairs ? ? a shower  I'm gonna go upstairs and take a shower ? have ? visit the ? room  I have to visit the ladies' room We ? ? a lot ? gifts  We don't have a lot of gifts ? ? don't need your ?  If I don't need your help ? ? ? decision  to make a decision ? still ? believe ? did this  I still can't believe you did this What ? ? ? here  What are you doing here ? you ? first ? of medical school  Are you go first day of medical school ? ? a dream about ? In ?  I had a dream about you in Copenhagen

20. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 20 Experimental Setup The order (k) of an hyperedge  Range: 2~4  Fixed order for each experiment The method of creating hyperedges from training data  Sliding window method  Sequential sampling from the first word The number of blanks (question marks) in test data  Range: 1~4  Maximum: k - 1

21. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 21 Learning Behavior Analysis (1/3) The performance monotonically increases as the learning corpus grows. The low-order memory performs best for the one-missing-word problem.

22. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 22 Learning Behavior Analysis (2/3) The medium-order (k=3) memory performs best for the two-missing-words problem.

23. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 23 Learning Behavior Analysis (3/3) The high-order (k=4) memory performs best for the three-missing-words problem.

24. Learning with Hypernetworks

25. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 25 x1 =1 x2 =0 x3 =0 x4 =1 x5 =0 x6 =0 x7 =0 x8 =0 x9 =0 x10 =1 x11 =0 x12 =1 x13 =0 x14 =0 x15 =0 y = 1 x1 =0 x2 =1 x3 =1 x4 =0 x5 =0 x6 =0 x7 =0 x8 =0 x9 =1 x10 =0 x11 =0 x12 =0 x13 =0 x14 =1 x15 =0 y = 0 x1 =0 x2 =0 x3 =1 x4 =0 x5 =0 x6 =1 x7 =0 x8 =1 x9 =0 x10 =0 x11 =0 x12 =0 x13 =1 x14 =0 x15 =0 y =1 4 examples x4x4 x 10 y=1x1x1 x4x4 x 12 y=1x1x1 x 10 x 12 y=1x4x4 x3x3 x9x9 y=0x2x2 x3x3 x 14 y=0x2x2 x9x9 x 14 y=0x3x3 x6x6 x8x8 y=1x3x3 x6x6 x 13 y=1x3x3 x8x8 x 13 y=1x6x6 1 2 3 1 2 3 x1 =0 x2 =0 x3 =0 x4 =0 x5 =0 x6 =0 x7 =0 x8 =1 x9 =0 x10 =0 x11 =1 x12 =0 x13 =0 x14 =0 x15 =1 y =1 4 x 11 x 15 y=0x8x8 4 Round 1 Round 2 Round 3

26. 26 Random Graph Process (RGP) 1 5 4 3 2

27. 27

28. 28 Collectively Autocatalytic Sets “A contemporary cell is a collectively autocatalytic whole in which DNA, RNA, the code, proteins, and metabolism linking the synthesis of species of some molecular species to the breakdown of other “high-energy” molecular species all weave together and conspire to catalyze the entire set of reactions required for the whole cell to reproduce.” (Kauffman)

29. 29 Reaction Graph

30. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 30 The Hypernetwork Model of Memory [Zhang, 2006]

31. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 31 Deriving the Learning Rule

32. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 32 Derivation of the Learning Rule

33. Comparison to Other Machine Learning Methods

34. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 34 Probabilistic Graphical Models (PGMs) Represent the joint probability distribution on some random variables in graphical form.  Undirected PGMs  Directed PGMs Generative: The probability distribution for some variables given values of other variables can be obtained.  Probabilistic inference C A B E D C and D are independent given B. C asserts dependency between A and B. B and E are independent given C.

35. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 35 Kinds of Graphical Models Graphical Models - Boltzmann Machines - Markov Random Fields - Bayesian Networks - Latent Variable Models - Hidden Markov Models - G enerative Topographic Mapping - N on-negative Matrix Factorization UndirectedDirected

36. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 36 Bayesian Networks  BN = (S, P) consists of a network structure S and a set of local probability distributions P Structure can be found by relying on the prior knowledge of causal relationships

37. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 37 From Bayes Nets to High-Order PGMs G F J A S G F J A S G F J A S (1) Naïve Bayes (2) Bayesian Net (3) High-Order PGM

38. Visual Memories Digit Recognition Face Classification Text Classification Movie Title Prediction

39. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 39 Digit Recognition: Dataset Original Data  Handwritten digits (0 ~ 9)  Training data: 2,630 (263 examples for each class)  Test data: 1,130 (113 examples for each class) Preprocessing  Each example is 8x8 binary matrix.  Each pixel is 0 or 1.

40. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 40 Probabilistic Library (DNA Representation) “Layered” Hypernetwork Pattern Classification

41. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 41 Simulation Results – without Error Correction |Train set| = 3760, |Test set| = 1797.

42. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 42 Performance Comparison MethodsAccuracy MLP with 37 hidden nodes0.941 MLP with no hidden nodes0.901 SVM with polynomial kernel0.926 SVM with RBF kernel0.934 Decision Tree0.859 Naïve Bayes0.885 kNN (k=1)0.936 kNN (k=3)0.951 Hypernet with learning (k = 10)0.923 Hypernet with sampling (k = 33)0.949

43. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 43 Error Correction Algorithm 1. Initialize the library as before. 2. maxChangeCnt := librarySize. 3. For i := 0 to iteration_limit 1.trainCorrectCnt := 0. 2.Run classification for all training patterns. For each correctly classifed patterns, increase trainCorrectCnt. 3.For each library elements 1. Initialize fitness value to 0. 2. For each misclassified training patterns if a library element is matched to that example 1.if classified correctly, then fitness of the library element gains 2 points. 2.Else it loses 1 points. 4.changeCnt := max{ librarySize * (1.5 * (trainSetSize - trainCorrectCnt) / trainSetSize + 0.01), maxChangeCnt * 0.9 }. 5.maxChangeCnt := changeCnt. 6.Delete changeCnt library elements of lowest fitness and resample library elements whose classes are that of deleted ones.

44. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 44 Simulation Results – with Error Correction iterationLimit = 37, librarySize = 382,300,

45. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 45 Performance Comparison AlgorithmsCorrect classification rate Random Forest (f=10, t=50)94.10 % KNN (k=4) Hypernetwork (Order=26) 93.49 % 92.99 % AdaBoost (Weak Learner: J48)91.93 % SVM (Gaussian Kernel, SMO)91.37 % MLP90.53 % Naïve Bayes J48 87.26 % 84.86 %

46. Face Classification Experiments

47. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 47 Face Data Set Yale dataset 15 people 11 images per person Total 165 images

48. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 48 Training Images of a Person 10 for training The remaining 1 for test

49. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 49 Bitmaps for Training Data (Dimensionality = 480)

50. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 50 Classification Rate by Leave-One-Out

51. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 51 Classification Rate (Dimensionality = 64 by PCA)

52. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 52 Learning Hypernets from Movie Captions  Order  Sequential  Range: 2~3  Corpus  Friends  Prison Break  24

53. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 53 Learning Hypernets from Movie Captions

54. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 54 Learning Hypernets from Movie Captions

55. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 55 Learning Hypernets from Movie Captions

56. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 56 Learning Hypernets from Movie Captions  Classification  Query generation - I intend to marry her : I ? to marry her I intend ? marry her I intend to ? her I intend to marry ?  Matching - I ? to marry her order 2: I intend, I am, intend to, …. order 3: I intend to, intend to marry, …  Count the number of max-perfect-matching hyperedges

57. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 57  Completion & Classification Examples QueryCompletionClassification who are youCorpus: Friends, 24, Prison Break ? are you who ? you who are ? what are you who are you Friends you need to wear itCorpus: 24, Prison Break, House ? need to wear it you ? to wear it you need ? wear it you need to ? it you need to wear ? i need to wear it you want to wear it you need to wear it you need to do it you need to wear a 24 House 24 Learning Hypernets from Movie Captions