A M P C CC C Automatic Contextual Pattern Modeling Pengyu Hong Beckman Institute for Advanced Science and Technology University of Illinois at Urbana Champaign

A M P C CC C Overview Motivations Motivations Define the problem Define the problem Formulation the problem Formulation the problem Experimental results Experimental results Conclusions and discussions Conclusions and discussions Design the algorithm Design the algorithm

A M P C CC C Motivations The global features Edge+ Color histogram

A M P C CC C Motivations The color histograms of six images A simple example. What kind of visual pattern shared by the following histograms?

A M P C CC C Motivations The normalized wavelet texture histograms of those six images The global texture information is also given …

A M P C CC C Motivations The images

A M P C CC C Motivations

A M P C CC C Motivations The global features of an object are the mixtures of the local features of the primitives. ! The global features alone are not enough for distinguishing different objects/scenes in many cases.

A M P C CC C Motivations ! It is very important to model both the primitives and the relations. An object consists of several primitives among which various contextual relations are defined.

A M P C CC C Motivations In terms of images Examples of primitives Examples of primitives Regions Regions Edges Edges … Examples of relations Examples of relations Relative distance between two primitives Relative distance between two primitives Relative orientation between two primitives Relative orientation between two primitives The size ratio between two primitives The size ratio between two primitives …

A M P C CC C The representation Attributed relational graph (ARG) [Tsai1979] has been extensively used to represent objects/scenes. An example of ARG First, we need to choose an representation for the information in order to calculate it.

A M P C CC C The representation – ARG The lines represent the relations between the object primitives. The lines represent the relations between the object primitives. The nodes of an ARG represent the object primitives. The attributes (color histogram, shapes, texture, etc.) of the nodes represent the appearance features of the object primitives. The nodes of an ARG represent the object primitives. The attributes (color histogram, shapes, texture, etc.) of the nodes represent the appearance features of the object primitives. ARG

A M P C CC C The representation – ARG An example: The image is segmented and represented as an ARG. The nodes represents the regions. The color of the nodes denotes the mean color of the regions The lines represent the adjacent relations among the regions.

A M P C CC C The representation – ARG Separate the local features and allow the user to examine the objects/scenes on a finer scale. Separate the local features and allow the user to examine the objects/scenes on a finer scale. The advantage of the ARG representation.

A M P C CC C Scene 2 Scene 1 The representation – ARG Separate the local spatial transformations and the global spatial transformations of the object. Separate the local spatial transformations and the global spatial transformations of the object. The advantage of the ARG representation. Separate the local features and allow the user to examine the objects on a finer scale. Separate the local features and allow the user to examine the objects on a finer scale. Scene 2 Global translation and rotation + Local deformation

A M P C CC C Problem definition A set of sample ARGs Summarize Pattern model DetectionRecognitionSynthesis

A M P C CC C Problem definition Pattern model How to build this  Manually design  Learn from multiple observations ?

A M P C CC C Related work Maron and Lozano-Pérez 1998 Maron and Lozano-Pérez 1998 Develop a Bayes learning framework to learn visual patterns from multiple labeled images. Frey and Jojic 1999 Frey and Jojic 1999 Use generative model to jointly estimate the transformations and the appearance of the image pattern. Guo, Zhu, & Wu Guo, Zhu, & Wu Integrate descriptive model and generative model to learn visual pattern from multiple labeled images. Hong & Huang 2000, Hong, Wang & Huang 2000 Hong & Huang 2000, Hong, Wang & Huang 2000

A M P C CC C The contribution Develop the methodology and theory for automatically learning a probability parametric pattern model to summarize a set of observed samples. The probability parametric pattern model is called the pattern ARG model. It models both the appearance and the structure of the objects/scenes.

A M P C CC C Formulate the problem Assume the observations sample ARGs {G i } are the realizations of some underlying stochastic process governed by a probability distribution f(G). Assume the observations sample ARGs {G i } are the realizations of some underlying stochastic process governed by a probability distribution f(G). The objective of learning is to estimate a model p(G) to approximate f(G) by minimizing the Kullback-Leibler divergence KL(f || p). [Cover & Thomas 1991]: The objective of learning is to estimate a model p(G) to approximate f(G) by minimizing the Kullback-Leibler divergence KL(f || p). [Cover & Thomas 1991]:

A M P C CC C Formulate the problem Therefore, we have a maximum likelihood estimator (MLE).

A M P C CC C f(G)f(G)f(G)f(G) How to calculate p(o) ? Simplicity: The model uses a set of parameters to represent f(G). Generality: Use a set of components (mixtures) to approximate the true distribution. In practice, it is often necessary to impose structures on the distribution. For example, linear family

A M P C CC C Illustration of modeling o 13 o 11 o 14 o 12 o 22 o 21 o 23 o 24 oS4oS4oS4oS4 oS3oS3oS3oS3 oS2oS2oS2oS2 oS1oS1oS1oS1 A set of sample images {I i }, i = 1, … S. o 11 o 13 o 14 o 12 r 114 r 112 r 113 r 134 G1G1G1G1 o 22 o 24 o 23 o 21 r 212 r 213 r 214 r 234 G2G2G2G2 oS3oS3oS3oS3 oS1oS1oS1oS1 oS2oS2oS2oS2 oS4oS4oS4oS4 r S14 r S34 r S12 r S13 GSGSGSGS A set of sample ARGs {G i }, i = 1, … S.

A M P C CC C Illustration of modeling M << S A set of sample ARGs {G i }, i = 1, … S. Summarize o 11 o 13 o 14 o 12 r 114 r 112 r 113 r 134 G1G1G1G1 o 22 o 24 o 23 o 21 r 212 r 213 r 214 r 234 G2G2G2G2 oS3oS3oS3oS3 oS1oS1oS1oS1 oS2oS2oS2oS2 oS4oS4oS4oS4 r S14 r S34 r S12 r S13 GSGSGSGS A pattern ARG model of M components {  i }, i = 1, …, M  14  12  114  11  13  112  113  134 1111 MMMM M4M4M4M4 M2M2M2M2  M14 M1M1M1M1 M3M3M3M3  M12  M13  M34

A M P C CC C Hierarchically linear modeling A pattern ARG with M components  14  12  114  11  13  112  113  134 M4M4M4M4 M2M2M2M2  M14 M1M1M1M1 M3M3M3M3  M12  M13  M34 o 11 o 13 o 14 o 12 r 114 r 112 r 113 r 1111 MMMM 1111 MMMM On macro scale A sample ARG =  h  h

A M P C CC C Hierarchically linear modeling  14  12  114  11  13  112  113  134 M4M4M4M4 M2M2M2M2  M14 M1M1M1M1 M3M3M3M3  M12  M13  M34 A pattern ARG with M components o 12 A sample node =  h (  h   h  ) ++  12  11  14  13 M2M2M2M2 M1M1M1M1 M4M4M4M4 M3M3M3M3 + 1111 MMMM On micro scale

A M P C CC C The underlying distributions  14  12  114  11  13  112  113  134 Attributed distributions Relational distributions Each component of the pattern ARG model is a parametric model.

A M P C CC C The task is to… The parameters of the distribution functions The parameters of the distribution functions Learn the parameters of the pattern ARG model given the sample ARGs. The parameters ({  h }, {  h  }) that describe the contribution of the model components. The parameters ({  h }, {  h  }) that describe the contribution of the model components.

A M P C CC C Sometimes … The instances of the pattern are in various backgrounds. o 13 o 11 o 14 o 12 o 22 o 21 o 23 o 24 oS4oS4oS4oS4 oS3oS3oS3oS3 oS2oS2oS2oS2 oS1oS1oS1oS1

A M P C CC C Sometimes … It is labor intensive to manually extract each instance out of its background. The learning procedure should automatically extract the instances of the pattern ARG out of the sample ARGs.

A M P C CC C Modified version of modeling M << S Summarize A pattern ARG model with M components {  i }, i = 1, …, M  14  12  114  11  13  112  113  134 1111 M4M4M4M4 M2M2M2M2  M14 M1M1M1M1 M3M3M3M3  M12  M13  M34 MMMM GSGSGSGS oS3oS3oS3oS3 oS1oS1oS1oS1 oS2oS2oS2oS2 oS4oS4oS4oS4 r S14 r S34 r S12 r S13 G2G2G2G2 o 22 o 24 o 23 o 21 r 212 r 213 r 214 r 234 G1G1G1G1 o 11 o 13 o 14 o 12 r 114 r 112 r 113 r 134

A M P C CC C Learning via the EM algorithm The EM [ Dempster1977] algorithm is a technique of finding the maximum likelihood estimate of the parameters of the underlying distributions from a training set. The EM algorithm defines a likelihood function, which in this case is:

A M P C CC C Learning via the EM algorithm The sample ARG set. The correspondences between the sample ARGs and the pattern ARG model. The parameters to be estimated. is a function of  under the assumption that  =  (t) Underlying distribution

A M P C CC C Learning via the EM algorithm Analogy

A M P C CC C Expectation step Expectation step The EM algorithm works iteratively in two steps: Expectation & Maximization is calculated, where t is the number of iterations. Maximization step Maximization step  is updated by Modify the structure of the pattern ARG model Modify the structure of the pattern ARG model

A M P C CC C Initialize the pattern ARG model A sample ARG For example if the pattern ARG model has 3 components.

A M P C CC C The Expectation step Please refer to the paper for the details. Calculate the likelihood of the data It is not so complicated as it appears!

A M P C CC C The Maximization step The expressions for the parameters ({  h }, {  h  }), which describe the contribution of model components, can be derived without knowing the forms of the attributed distributions and those of the relational distributions. The expressions for the parameters ({  h }, {  h  }), which describe the contribution of model components, can be derived without knowing the forms of the attributed distributions and those of the relational distributions. For Gaussian attributed distributions and Gaussian relational distributions, we can obtain analytical expressions to estimate the distribution parameters. For Gaussian attributed distributions and Gaussian relational distributions, we can obtain analytical expressions to estimate the distribution parameters. Please refer to the paper for the details. Update  Derive the expressions for  (t+1).

A M P C CC C The Maximization step ({  h }, {  h  })

A M P C CC C The Maximization step The parameters of the Gaussian attributed distributions Mean Covariance matrix

A M P C CC C The Maximization step The parameters of the Gaussian relational distributions Mean Covariance matrix

A M P C CC C Modify the structure Null node Initialize the components of the pattern ARG model

A M P C CC C Modify the structure Modify the structure of the pattern ARG model. Modify the structure of the pattern ARG model. It is very possible that the model components are initialized so that they contain some nodes representing backgrounds. During the iterations of the algorithm, we examine the parameters ({  h }, {  h  } ) and decide which model nodes should be marked as background nodes.

A M P C CC C Modify the structure During the iterations of the algorithm, we examine the parameters ({  h }, {  h  }) and decide which model nodes should be marked as background nodes.

A M P C CC C Detect the pattern Use the learned pattern ARG model to detect the pattern. Given an new graph G new, we calculate the following likelihood

A M P C CC C Experimental results I In this experiment, the images are segmented. The color feature (RGB and its variances) of a segment is used. However, our theory can be applied directly on image pixels (see Discussions) or other image primitives (e.g. edges). Segmentation is just used to reduce the computational complexity. Automatic image pattern extraction Automatic image pattern extraction

A M P C CC C The images Experimental results I

A M P C CC C The segmentation results

A M P C CC C Experimental results I The ARGs

A M P C CC C Experimental Results I The learning results as subgraph in the sample ARGs

A M P C CC C Experimental results I The corresponding image segments

A M P C CC C Experimental results I Detection

A M P C CC C Experimental Results II Improve pattern detection Improve pattern detection

A M P C CC C Experimental results II Lighting 1 Lighting 2 (208, 150, 69) (202, 138, 60) (206, 144, 71) The ‘m’ (240, 173, 116) (240, 180, 109) (241, 192, 120) The ‘m’

A M P C CC C Bad guess good guess better guess and better guess Experimental results II We implemented the probability relaxation graph matching algorithm [ Christmas, Kittler & Petrou 1995]. The matching results depend on the values of the parameters

A M P C CC C Experimental results II Our approach Our approach

A M P C CC C Experimental results II Compare the results shown in the previous two slides. It will not be difficult to see the followings. The learning procedure automatically adjusts the parameters for graph matching. The learning results include the correspondences between the pattern ARG model and the sample ARGs. The learning procedure utilizes the evidence provided by multiple samples to get rid of backgrounds.

A M P C CC C Experimental results III Structural texture modeling and synthesis Structural texture modeling and synthesis Model the structure Model the appearance Normalize the texture elements to the same size

A M P C CC C Experimental results III Synthesize new texture … First, synthesize the structure.

A M P C CC C Experimental results III Synthesize new texture … Then, synthesize the appearance.

A M P C CC C Experimental Results III Synthesize new texture … Borrow the structure and synthesize new texture … Modify the appearance node of the learned model.

A M P C CC C Experimental results III sample synthesized

A M P C CC C Experimental Results IV Automatic FAQ detection Automatic FAQ detection Student Jack: “What are Java applets?” Student Tom: “Would you please define Java programs?” Student Jenny: “Could you tell me the definitions of Java applet and Java application?” For example ….

A M P C CC C Experimental Results IV Using the Word Concept Model [Li & Levinson 2001]. We can parse the questions into graphs. Student Jack: “What are Java applets?”

A M P C CC C Experimental Results IV Student Tom: “Would you please define Java programs?”

A M P C CC C Experimental Results IV Student Jenny: “Could you tell me the definitions of a Java applet and a Java application?”

A M P C CC C Experimental Results IV The summarized FAQ

A M P C CC C Experimental Results V Original video Segmented ARG sequence Foreground subgraph Foreground Summarization

A M P C CC C Experimental Results V …… Retrieve results

A M P C CC C Conclusions Develop the methodology and theory for evidence combining that fuses the appearance information and structure information of the observed samples. Choose representation Define and formulate the problem Design the algorithm to solve the problem

A M P C CC C Conclusions Automatically learns a compact parametric model to represent a pattern that is observed under various conditions. Automatically learns a compact parametric model to represent a pattern that is observed under various conditions. Automatically eliminates the backgrounds by using multiple samples. Automatically eliminates the backgrounds by using multiple samples. The mathematical framework

A M P C CC C Discussions I The learning results depend on the quality of the results of low-level image processing. The learning results depend on the quality of the results of low-level image processing. Low-level image processing Learning Sample images Learned high-level knowledge Human interaction or some super models

A M P C CC C Discussions I Low-level image processing Learning Sample images Corrected high-level knowledge

A M P C CC C Discussions I Knowledge based Low-level image processing Learning Sample images Corrected high-level knowledge

A M P C CC C Discussions II If enough computational power is available, we can work directly on pixel-level. If enough computational power is available, we can work directly on pixel-level. Each pixel is a node.

A M P C CC C Discussions II If enough computational power is available (e.g. parallel/distributed computing), we can work directly on pixel-level. If enough computational power is available (e.g. parallel/distributed computing), we can work directly on pixel-level. Or even more complicated relations

A M P C CC C Discussions III Multiple resolutions/layers for complex phenomena Multiple resolutions/layers for complex phenomena 1111 MMMM The pattern ARG model

A M P C CC C Discussions III Recognizer networks Recognizer networks Each node can represent a primitive recognizer. Face detection and recognition Face and facial motion tracking

A M P C CC C Discussions IV Automatic FAQ Detection Reconfigurable Hardware It is more than software It is more than software Computer programs Diagram (or Graph) Frequently executed codes

A M P C CC C Discussions V Higher dimensional data Higher dimensional data For example, molecular modeling …

A M P C CC C Discussions V Higher dimensional data Higher dimensional data

A M P C CC C Discussion VI Why? Why?

A M P C CC C Discussions VI Microarray data of genes Source: Dr. Robin E. Everts, 210 ERML, UIUC.

A M P C CC C Acknowledge Supported by USA ARL under Cooperative Agreement No. DAAL Supported by USA ARL under Cooperative Agreement No. DAAL Felzenszwalb & Huttenlocher for image segmentation program. Felzenszwalb & Huttenlocher for image segmentation program.

A M P C CC C