Download presentation
Presentation is loading. Please wait.
Published byJesse Little Modified over 9 years ago
1
Hypernetwork Models of Memory Hypernetwork Models of Memory Byoung-Tak Zhang Biointelligence Laboratory School of Computer Science and Engineering Brain Science, Cognitive Science, Bioinformatics Programs Seoul National University Seoul 151-742, Korea btzhang@cse.snu.ac.kr http://bi.snu.ac.kr/
2
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 2 Signaling Networks
3
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 3 Protein-Protein Interaction Networks [Jeong et al., Nature 2001]
4
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 4 Regulatory Networks [Katia Basso, et al., Nature Genetics 37, 382 – 390, 2005] Transcription Factor
5
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 5 Molecular Networks
6
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 6 x1 x2 x3 x4 x5 x6 x7 x8x9 x10 x11 x12 x13 x14 x15 The “Hyperinteraction” Model of Biomolecules Vertex: Genes Proteins Neurotransmitters Neuromodulators Hormones Edge: Interactions Genetic Signaling Metabolic Synaptic [Zhang, DNA-2006] [Zhang, FOCI-2007]
7
The Hypernetworks
8
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 8 Hypergraphs A hypergraph is a (undirected) graph G whose edges connect a non-null number of vertices, i.e. G = (V, E), where V = {v 1, v 2, …, v n }, E = {E 1, E 2, …, E n }, and E i = {v i1, v i2, …, v im } An m-hypergraph consists of a set V of vertices and a subset E of V [m], i.e. G = (V, V [m] ) where V [m] is a set of subsets of V whose elements have precisely m members. A hypergraph G is said to be k-uniform if every edge E i in E has cardinality k. A hypergraph G is k-regular if every vertex has degree k. Rem.: An ordinary graph is a 2-uniform hypergraph.
9
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 9 An Example Hypergraph v5 v1 v3 v7 v2 v6 v4 G = (V, E) V = {v1, v2, v3, …, v7} E = {E1, E2, E3, E4, E5} E1 = {v1, v3, v4} E2 = {v1, v4} E3 = {v2, v3, v6} E4 = {v3, v4, v6, v7} E5 = {v4, v5, v7} E1 E4 E5 E2 E3
10
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 10 Hypernetworks A hypernetwork is a hypergraph of weighted edges. It is defined as a triple H = (V, E, W), where V = {v 1, v 2, …, v n }, E = {E 1, E 2, …, E n }, and W = {w 1, w 2, …, w n }. An m-hypernetwork consists of a set V of vertices and a subset E of V [m], i.e. H = (V, V [m], W) where V [m] is a set of subsets of V whose elements have precisely m members and W is the set of weights associated with the hyperedges. A hypernetwork H is said to be k-uniform if every edge E i in E has cardinality k. A hypernetwork H is k-regular if every vertex has degree k. Rem.: An ordinary graph is a 2-uniform hypergraph with w i =1. [Zhang, DNA-2006]
11
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 11 x1 x2 x3 x4 x5 x6 x7 x8x9 x10 x11 x12 x13 x14 x15 The Hypernetwork Memory [Zhang, DNA12-2006]
12
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 12 x1 =1 x2 =0 x3 =0 x4 =1 x5 =0 x6 =0 x7 =0 x8 =0 x9 =0 x10 =1 x11 =0 x12 =1 x13 =0 x14 =0 x15 =0 y = 1 x1 =0 x2 =1 x3 =1 x4 =0 x5 =0 x6 =0 x7 =0 x8 =0 x9 =1 x10 =0 x11 =0 x12 =0 x13 =0 x14 =1 x15 =0 y = 0 x1 =0 x2 =0 x3 =1 x4 =0 x5 =0 x6 =1 x7 =0 x8 =1 x9 =0 x10 =0 x11 =0 x12 =0 x13 =1 x14 =0 x15 =0 y =1 4 sentences (with labels) x4x4 x 10 y=1x1x1 x4x4 x 12 y=1x1x1 x 10 x 12 y=1x4x4 x3x3 x9x9 y=0x2x2 x3x3 x 14 y=0x2x2 x9x9 x 14 y=0x3x3 x6x6 x8x8 y=1x3x3 x6x6 x 13 y=1x3x3 x8x8 x 13 y=1x6x6 1 2 3 1 2 3 x1 =0 x2 =0 x3 =0 x4 =0 x5 =0 x6 =0 x7 =0 x8 =1 x9 =0 x10 =0 x11 =1 x12 =0 x13 =0 x14 =0 x15 =1 y =1 4 x 11 x 15 y=0x8x8 4 Round 1 Round 2 Round 3
13
An Example: Hypernetwork Model of Linguistic Memory
14
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 14 The Language Game Platform
15
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 15 A Language Game ? still ? believe ? did this. I still can't believe you did this. We ? ? a lot ? gifts. We don't have a lot of gifts.
16
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 16 Text Corpus: TV Drama Series Friends, 24, House, Grey Anatomy, Gilmore Girls, Sex and the City 289,468 Sentences (Training Data) 700 Sentences with Blanks (Test Data) I don't know what happened. Take a look at this. … What ? ? ? here. ? have ? visit the ? room. …
17
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 17 Step 1: Learning a Linguistic Memory k = 2 k = 3 k = 4 … Hypernetwork Memory
18
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 18 Step 2: Recalling from the Memory Heismybestfriend bestfriendmybestismyHeis Heisa?friend HeisastrongboyStrongfriendlikesprettygirl Heis aastrong boyStrongfriend likes strongprettygirlStrongfriendastrongbestfriend Heisastrongfriend X7 X6 X5 X8 X1 X2 X3 X4 Recall Self-assembly Storage
19
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 19 The Language Game: Results Why ? you ? come ? down ? Why are you go come on down here ? think ? I ? met ? somewhere before I think but I am met him somewhere before ? appreciate it if ? call her by ? ? I appreciate it if you call her by the way I'm standing ? the ? ? ? cafeteria I'm standing in the one of the cafeteria Would you ? to meet ? ? Tuesday ? Would you nice to meet you in Tuesday and ? gonna ? upstairs ? ? a shower I'm gonna go upstairs and take a shower ? have ? visit the ? room I have to visit the ladies' room We ? ? a lot ? gifts We don't have a lot of gifts ? ? don't need your ? If I don't need your help ? ? ? decision to make a decision ? still ? believe ? did this I still can't believe you did this What ? ? ? here What are you doing here ? you ? first ? of medical school Are you go first day of medical school ? ? a dream about ? In ? I had a dream about you in Copenhagen
20
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 20 Experimental Setup The order (k) of an hyperedge Range: 2~4 Fixed order for each experiment The method of creating hyperedges from training data Sliding window method Sequential sampling from the first word The number of blanks (question marks) in test data Range: 1~4 Maximum: k - 1
21
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 21 Learning Behavior Analysis (1/3) The performance monotonically increases as the learning corpus grows. The low-order memory performs best for the one-missing-word problem.
22
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 22 Learning Behavior Analysis (2/3) The medium-order (k=3) memory performs best for the two-missing-words problem.
23
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 23 Learning Behavior Analysis (3/3) The high-order (k=4) memory performs best for the three-missing-words problem.
24
Learning with Hypernetworks
25
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 25 x1 =1 x2 =0 x3 =0 x4 =1 x5 =0 x6 =0 x7 =0 x8 =0 x9 =0 x10 =1 x11 =0 x12 =1 x13 =0 x14 =0 x15 =0 y = 1 x1 =0 x2 =1 x3 =1 x4 =0 x5 =0 x6 =0 x7 =0 x8 =0 x9 =1 x10 =0 x11 =0 x12 =0 x13 =0 x14 =1 x15 =0 y = 0 x1 =0 x2 =0 x3 =1 x4 =0 x5 =0 x6 =1 x7 =0 x8 =1 x9 =0 x10 =0 x11 =0 x12 =0 x13 =1 x14 =0 x15 =0 y =1 4 examples x4x4 x 10 y=1x1x1 x4x4 x 12 y=1x1x1 x 10 x 12 y=1x4x4 x3x3 x9x9 y=0x2x2 x3x3 x 14 y=0x2x2 x9x9 x 14 y=0x3x3 x6x6 x8x8 y=1x3x3 x6x6 x 13 y=1x3x3 x8x8 x 13 y=1x6x6 1 2 3 1 2 3 x1 =0 x2 =0 x3 =0 x4 =0 x5 =0 x6 =0 x7 =0 x8 =1 x9 =0 x10 =0 x11 =1 x12 =0 x13 =0 x14 =0 x15 =1 y =1 4 x 11 x 15 y=0x8x8 4 Round 1 Round 2 Round 3
26
26 Random Graph Process (RGP) 1 5 4 3 2
27
27
28
28 Collectively Autocatalytic Sets “A contemporary cell is a collectively autocatalytic whole in which DNA, RNA, the code, proteins, and metabolism linking the synthesis of species of some molecular species to the breakdown of other “high-energy” molecular species all weave together and conspire to catalyze the entire set of reactions required for the whole cell to reproduce.” (Kauffman)
29
29 Reaction Graph
30
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 30 The Hypernetwork Model of Memory [Zhang, 2006]
31
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 31 Deriving the Learning Rule
32
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 32 Derivation of the Learning Rule
33
Comparison to Other Machine Learning Methods
34
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 34 Probabilistic Graphical Models (PGMs) Represent the joint probability distribution on some random variables in graphical form. Undirected PGMs Directed PGMs Generative: The probability distribution for some variables given values of other variables can be obtained. Probabilistic inference C A B E D C and D are independent given B. C asserts dependency between A and B. B and E are independent given C.
35
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 35 Kinds of Graphical Models Graphical Models - Boltzmann Machines - Markov Random Fields - Bayesian Networks - Latent Variable Models - Hidden Markov Models - G enerative Topographic Mapping - N on-negative Matrix Factorization UndirectedDirected
36
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 36 Bayesian Networks BN = (S, P) consists of a network structure S and a set of local probability distributions P Structure can be found by relying on the prior knowledge of causal relationships
37
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 37 From Bayes Nets to High-Order PGMs G F J A S G F J A S G F J A S (1) Naïve Bayes (2) Bayesian Net (3) High-Order PGM
38
Visual Memories Digit Recognition Face Classification Text Classification Movie Title Prediction
39
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 39 Digit Recognition: Dataset Original Data Handwritten digits (0 ~ 9) Training data: 2,630 (263 examples for each class) Test data: 1,130 (113 examples for each class) Preprocessing Each example is 8x8 binary matrix. Each pixel is 0 or 1.
40
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 40 Probabilistic Library (DNA Representation) “Layered” Hypernetwork Pattern Classification
41
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 41 Simulation Results – without Error Correction |Train set| = 3760, |Test set| = 1797.
42
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 42 Performance Comparison MethodsAccuracy MLP with 37 hidden nodes0.941 MLP with no hidden nodes0.901 SVM with polynomial kernel0.926 SVM with RBF kernel0.934 Decision Tree0.859 Naïve Bayes0.885 kNN (k=1)0.936 kNN (k=3)0.951 Hypernet with learning (k = 10)0.923 Hypernet with sampling (k = 33)0.949
43
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 43 Error Correction Algorithm 1. Initialize the library as before. 2. maxChangeCnt := librarySize. 3. For i := 0 to iteration_limit 1.trainCorrectCnt := 0. 2.Run classification for all training patterns. For each correctly classifed patterns, increase trainCorrectCnt. 3.For each library elements 1. Initialize fitness value to 0. 2. For each misclassified training patterns if a library element is matched to that example 1.if classified correctly, then fitness of the library element gains 2 points. 2.Else it loses 1 points. 4.changeCnt := max{ librarySize * (1.5 * (trainSetSize - trainCorrectCnt) / trainSetSize + 0.01), maxChangeCnt * 0.9 }. 5.maxChangeCnt := changeCnt. 6.Delete changeCnt library elements of lowest fitness and resample library elements whose classes are that of deleted ones.
44
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 44 Simulation Results – with Error Correction iterationLimit = 37, librarySize = 382,300,
45
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 45 Performance Comparison AlgorithmsCorrect classification rate Random Forest (f=10, t=50)94.10 % KNN (k=4) Hypernetwork (Order=26) 93.49 % 92.99 % AdaBoost (Weak Learner: J48)91.93 % SVM (Gaussian Kernel, SMO)91.37 % MLP90.53 % Naïve Bayes J48 87.26 % 84.86 %
46
Face Classification Experiments
47
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 47 Face Data Set Yale dataset 15 people 11 images per person Total 165 images
48
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 48 Training Images of a Person 10 for training The remaining 1 for test
49
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 49 Bitmaps for Training Data (Dimensionality = 480)
50
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 50 Classification Rate by Leave-One-Out
51
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 51 Classification Rate (Dimensionality = 64 by PCA)
52
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 52 Learning Hypernets from Movie Captions Order Sequential Range: 2~3 Corpus Friends Prison Break 24
53
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 53 Learning Hypernets from Movie Captions
54
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 54 Learning Hypernets from Movie Captions
55
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 55 Learning Hypernets from Movie Captions
56
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 56 Learning Hypernets from Movie Captions Classification Query generation - I intend to marry her : I ? to marry her I intend ? marry her I intend to ? her I intend to marry ? Matching - I ? to marry her order 2: I intend, I am, intend to, …. order 3: I intend to, intend to marry, … Count the number of max-perfect-matching hyperedges
57
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 57 Completion & Classification Examples QueryCompletionClassification who are youCorpus: Friends, 24, Prison Break ? are you who ? you who are ? what are you who are you Friends you need to wear itCorpus: 24, Prison Break, House ? need to wear it you ? to wear it you need ? wear it you need to ? it you need to wear ? i need to wear it you want to wear it you need to wear it you need to do it you need to wear a 24 House 24 Learning Hypernets from Movie Captions
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.