Manifold learning and pattern matching with entropic graphs Alfred O. Hero Dept. EECS, Dept Biomed. Eng., Dept. Statistics University of Michigan - Ann.

Slides:



Advertisements
Similar presentations
Applications of one-class classification
Advertisements

Coherent Laplacian 3D protrusion segmentation Oxford Brookes Vision Group Queen Mary, University of London, 11/12/2009 Fabio Cuzzolin.
Text mining Gergely Kótyuk Laboratory of Cryptography and System Security (CrySyS) Budapest University of Technology and Economics
Computer examples Tenenbaum, de Silva, Langford “A Global Geometric Framework for Nonlinear Dimensionality Reduction”
Manifold Learning Techniques: So which is the best?
1 Manifold Alignment for Multitemporal Hyperspectral Image Classification H. Lexie Yang 1, Melba M. Crawford 2 School of Civil Engineering, Purdue University.
Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.
DIMENSIONALITY REDUCTION: FEATURE EXTRACTION & FEATURE SELECTION Principle Component Analysis.
A Geometric Perspective on Machine Learning 何晓飞 浙江大学计算机学院 1.
Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,
Manifold Learning Dimensionality Reduction. Outline Introduction Dim. Reduction Manifold Isomap Overall procedure Approximating geodesic dist. Dijkstra’s.
Train a Classifier Based on the Huge Face Database
Presented by: Mingyuan Zhou Duke University, ECE April 3, 2009
Non-linear Dimensionality Reduction CMPUT 466/551 Nilanjan Ray Prepared on materials from the book Non-linear dimensionality reduction By Lee and Verleysen,
Clustering and Dimensionality Reduction Brendan and Yifang April
University of Joensuu Dept. of Computer Science P.O. Box 111 FIN Joensuu Tel fax Isomap Algorithm.
1 High dimensionality Evgeny Maksakov CS533C Department of Computer Science UBC.
“Random Projections on Smooth Manifolds” -A short summary
Graph Based Semi- Supervised Learning Fei Wang Department of Statistical Science Cornell University.
1 High dimensionality Evgeny Maksakov CS533C Department of Computer Science UBC.
RBF Neural Networks x x1 Examples inside circles 1 and 2 are of class +, examples outside both circles are of class – What NN does.
Clustering… in General In vector space, clusters are vectors found within  of a cluster vector, with different techniques for determining the cluster.
LLE and ISOMAP Analysis of Robot Images Rong Xu. Background Intuition of Dimensionality Reduction Linear Approach –PCA(Principal Component Analysis) Nonlinear.
Entropic graphs: Applications Alfred O. Hero Dept. EECS, Dept BME, Dept. Statistics University of Michigan - Ann Arbor
Manifold Learning: ISOMAP Alan O'Connor April 29, 2008.
Three Algorithms for Nonlinear Dimensionality Reduction Haixuan Yang Group Meeting Jan. 011, 2005.
Manifold Learning Using Geodesic Entropic Graphs Alfred O. Hero and Jose Costa Dept. EECS, Dept Biomed. Eng., Dept. Statistics University of Michigan -
Multisite Internet Data Analysis Alfred O. Hero, Clyde Shih, David Barsic University of Michigan - Ann Arbor
A Global Geometric Framework for Nonlinear Dimensionality Reduction Joshua B. Tenenbaum, Vin de Silva, John C. Langford Presented by Napat Triroj.
Atul Singh Junior Undergraduate CSE, IIT Kanpur.  Dimension reduction is a technique which is used to represent a high dimensional data in a more compact.
NonLinear Dimensionality Reduction or Unfolding Manifolds Tennenbaum|Silva|Langford [Isomap] Roweis|Saul [Locally Linear Embedding] Presented by Vikas.
Lightseminar: Learned Representation in AI An Introduction to Locally Linear Embedding Lawrence K. Saul Sam T. Roweis presented by Chan-Su Lee.
Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)
Nonlinear Dimensionality Reduction by Locally Linear Embedding Sam T. Roweis and Lawrence K. Saul Reference: "Nonlinear dimensionality reduction by locally.
Nonlinear Dimensionality Reduction Approaches. Dimensionality Reduction The goal: The meaningful low-dimensional structures hidden in their high-dimensional.
Representative Previous Work
Manifold learning: Locally Linear Embedding Jieping Ye Department of Computer Science and Engineering Arizona State University
Methods in Medical Image Analysis Statistics of Pattern Recognition: Classification and Clustering Some content provided by Milos Hauskrecht, University.
CSE 185 Introduction to Computer Vision Pattern Recognition.
Estimating Intrinsic Dimension Justin Eberhardt UMD, Mathematics and Statistics Advisor: Dr. Kang James.
University of Joensuu Dept. of Computer Science P.O. Box 111 FIN Joensuu Tel fax Speech and.
Graph Embedding: A General Framework for Dimensionality Reduction Dong XU School of Computer Engineering Nanyang Technological University
IEEE TRANSSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
Learning a Kernel Matrix for Nonlinear Dimensionality Reduction By K. Weinberger, F. Sha, and L. Saul Presented by Michael Barnathan.
THE MANIFOLDS OF SPATIAL HEARING Ramani Duraiswami | Vikas C. Raykar Perceptual Interfaces and Reality Lab University of Maryland, College park.
Computer Vision Lab. SNU Young Ki Baik Nonlinear Dimensionality Reduction Approach (ISOMAP, LLE)
Computer examples Tenenbaum, de Silva, Langford “A Global Geometric Framework for Nonlinear Dimensionality Reduction”
Transductive Regression Piloted by Inter-Manifold Relations.
MACHINE LEARNING 8. Clustering. Motivation Based on E ALPAYDIN 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2  Classification problem:
ISOMAP TRACKING WITH PARTICLE FILTER Presented by Nikhil Rane.
CSE 185 Introduction to Computer Vision Face Recognition.
Manifold learning: MDS and Isomap
1 LING 696B: MDS and non-linear methods of dimension reduction.
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Jan Kamenický.  Many features ⇒ many dimensions  Dimensionality reduction ◦ Feature extraction (useful representation) ◦ Classification ◦ Visualization.
H. Lexie Yang1, Dr. Melba M. Crawford2
Data Mining Course 0 Manifold learning Xin Yang. Data Mining Course 1 Outline Manifold and Manifold Learning Classical Dimensionality Reduction Semi-Supervised.
Non-Linear Dimensionality Reduction
Project 11: Determining the Intrinsic Dimensionality of a Distribution Okke Formsma, Nicolas Roussis and Per Løwenborg.
Data Mining Course 2007 Eric Postma Clustering. Overview Three approaches to clustering 1.Minimization of reconstruction error PCA, nlPCA, k-means clustering.
Nonlinear Dimension Reduction: Semi-Definite Embedding vs. Local Linear Embedding Li Zhang and Lin Liao.
Manifold Learning JAMES MCQUEEN – UW DEPARTMENT OF STATISTICS.
Spectral Methods for Dimensionality
Nonlinear Dimensionality Reduction
Intrinsic Data Geometry from a Training Set
Unsupervised Riemannian Clustering of Probability Density Functions
ISOMAP TRACKING WITH PARTICLE FILTERING
In summary C1={skin} C2={~skin} Given x=[R,G,B], is it skin or ~skin?
Outline Nonlinear Dimension Reduction Brief introduction Isomap LLE
NonLinear Dimensionality Reduction or Unfolding Manifolds
Presentation transcript:

Manifold learning and pattern matching with entropic graphs Alfred O. Hero Dept. EECS, Dept Biomed. Eng., Dept. Statistics University of Michigan - Ann Arbor

Multimodality Face Matching

Clustering Gene Microarray Data Cy5/Cy3 hybridization profiles

Image Registration

Vehicle Classification 128x128 images of three vehicles over 1 deg increments of 360 deg azimuth at 0 deg elevation The 3(360)=1080 images evolve on a lower dimensional imbedded manifold in R^(16384) Courtesy of Center for Imaging Science, JHU HMMV T62Truck

Image Manifold

What is manifold learning good for? Interpreting high dimensional data Discovery and exploitation of lower dimensional structure Deducing non-linear dependencies between populations Improving detection and classification performance Improving image compression performance

Random Sampling on a Manifold

Classifying on a Manifold Class A Class B

Background on Manifold Learning Manifold intrinsic dimension estimation –Local KLE, Fukunaga, Olsen (1971) –Nearest neighbor algorithm, Pettis, Bailey, Jain, Dubes (1971) –Fractal measures, Camastra and Vinciarelli (2002) –Packing numbers, Kegl (2002) Manifold Reconstruction –Isomap-MDS, Tenenbaum, de Silva, Langford (2000) –Locally Linear Embeddings (LLE), Roweiss, Saul (2000) –Laplacian eigenmaps (LE), Belkin, Niyogi (2002) –Hessian eigenmaps (HE), Grimes, Donoho (2003) Characterization of sampling distributions on manifolds –Statistics of directional data, Watson (1956), Mardia (1972) –Statistics of shape, Kendall (1984), Kent, Mardia (2001) –Data compression on 3D surfaces, Kolarov, Lynch (1997)

Assumption: is a conformal mapping A statistical sample Sampling distribution 2D manifold Sampling Embedding Sampling on a Domain Manifold

Alpha-Entropy and Divergence Alpha-entropy Alpha-divergence Other alpha-dissimilarity measures –Alpha-Jensen difference –Alpha geometric-arithmetic (GA) divergence

MST and Geodesic MST For a set of points in d- dimensional Euclidean space, the Euclidean MST with edge power weighting gamma is defined as edge lengths of a spanning tree over pairwise distance matrix of complete graph When the matrix is constructed from geodesic distances between points on, e.g. using ISOMAP, we obtain the Geodesic MST

A Planar Sample and its Euclidean MST

Convergence of Euclidean MST Beardwood, Halton, Hammersley Theorem:

Key Result for GMST Ref: Costa&Hero:TSP2003

Special Cases Isometric embedding (ISOMAP) Conformal embedding (C-ISOMAP)

Remarks Result holds for many other combinatorial optimization algorithms (Costa&Hero:2003) –K-NNG –Steiner trees –Minimal matchings –Traveling Salesman Tours a.s. convergence rates (Hero&etal:2002) For isometric embeddings Jacobian does not have to be estimated for dimension estimation

Joint Estimation Algorithm Assume large-n log-affine model Use bootstrap resampling to estimate mean MST length and apply LS to jointly estimate slope and intercept from sequence Extract d and H from slope and intercept

Random Samples on a Swiss Roll Ref: Grimes and Donoho (2003)

Bootstrap Estimates of GMST Length

loglogLinear Fit to GMST Length

Dimension and Entropy Estimates From LS fit find: Intrinsic dimension estimate Alpha-entropy estimate (nats)

Dimension Estimation Comparisons

Practical Application Yale face database 2 –Photographic folios of many people’s faces –Each face folio contains images at 585 different illumination/pose conditions –Subsampled to 64 by 64 pixels (4096 extrinsic dimensions) Objective: determine intrinsic dimension and entropy of a face folio

GMST for 3 Face Folios

Yale Face Database Results GMST LS estimation parameters –ISOMAP used to generate pairwise distance matrix –LS based on 25 resamplings over 26 largest folio sizes To represent any folio we might hope to attain –factor > 600 reduction in degrees of freedom (dim) –only 1/10 bit per pixel for compression –a practical parameterization/encoder? Ref: Costa&Hero 2003

Conclusions Characterizing high dimension sampling distributions –Standard techniques (histogram, density estimation) fail due to curse of dimensionality –Entropic graphs can be used to construct consistent estimators of entropy and information divergence –Robustification to outliers via pruning Manifold learning and model reduction –Standard techniques (LLE, MDS, LE, HE) rely on local linear fits –Entropic graph methods fit the manifold globally –Computational complexity is only n log n Advantages of Geodesic Entropic Graph Methods

Summary of Algorithm Run ISOMAP or C-ISOMAP algorithm to generate pairwise distance matrix on intrinsic domain of manifold Build geodesic entropic graph from pairwise distance matrix –MST: consistent estimator of manifold dimension and process alpha-entropy –K-NNG: consistent estimator of information divergence between labeled vectors Use bootstrap resampling and LS fitting to extract rate of convergence (intrinsic dimension) and convergence factor (entropy) of entropic graph

Swiss Roll Example Uniform Samples on 3D Imbedding of Swiss Roll

Geodesic Minimal Spanning Tree GMST over Uniform Samples on Swiss Roll

Geodesic MST on Imbedded Mixture GMST on Gaussian Samples on Swiss Roll

Classifying on a Manifold Class A Class B