Entropic graphs: Applications Alfred O. Hero Dept. EECS, Dept BME, Dept. Statistics University of Michigan - Ann Arbor

Slides:



Advertisements
Similar presentations
Coherent Laplacian 3D protrusion segmentation Oxford Brookes Vision Group Queen Mary, University of London, 11/12/2009 Fabio Cuzzolin.
Advertisements

Robust spectral 3D-bodypart segmentation along time Fabio Cuzzolin, Diana Mateus, Edmond Boyer, Radu Horaud Perception project meeting 24/4/2007 Submitted.
Active Appearance Models
Computer examples Tenenbaum, de Silva, Langford “A Global Geometric Framework for Nonlinear Dimensionality Reduction”
1 Manifold Alignment for Multitemporal Hyperspectral Image Classification H. Lexie Yang 1, Melba M. Crawford 2 School of Civil Engineering, Purdue University.
Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.
Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan.
Pattern Recognition and Machine Learning
A Geometric Perspective on Machine Learning 何晓飞 浙江大学计算机学院 1.
Manifold Learning Dimensionality Reduction. Outline Introduction Dim. Reduction Manifold Isomap Overall procedure Approximating geodesic dist. Dijkstra’s.
Train a Classifier Based on the Huge Face Database
Presented by: Mingyuan Zhou Duke University, ECE April 3, 2009
University of Joensuu Dept. of Computer Science P.O. Box 111 FIN Joensuu Tel fax Isomap Algorithm.
“Random Projections on Smooth Manifolds” -A short summary
ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.
LLE and ISOMAP Analysis of Robot Images Rong Xu. Background Intuition of Dimensionality Reduction Linear Approach –PCA(Principal Component Analysis) Nonlinear.
Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.
Manifold Learning: ISOMAP Alan O'Connor April 29, 2008.
Three Algorithms for Nonlinear Dimensionality Reduction Haixuan Yang Group Meeting Jan. 011, 2005.
Manifold Learning Using Geodesic Entropic Graphs Alfred O. Hero and Jose Costa Dept. EECS, Dept Biomed. Eng., Dept. Statistics University of Michigan -
Multisite Internet Data Analysis Alfred O. Hero, Clyde Shih, David Barsic University of Michigan - Ann Arbor
Manifold learning and pattern matching with entropic graphs Alfred O. Hero Dept. EECS, Dept Biomed. Eng., Dept. Statistics University of Michigan - Ann.
A Global Geometric Framework for Nonlinear Dimensionality Reduction Joshua B. Tenenbaum, Vin de Silva, John C. Langford Presented by Napat Triroj.
Atul Singh Junior Undergraduate CSE, IIT Kanpur.  Dimension reduction is a technique which is used to represent a high dimensional data in a more compact.
NonLinear Dimensionality Reduction or Unfolding Manifolds Tennenbaum|Silva|Langford [Isomap] Roweis|Saul [Locally Linear Embedding] Presented by Vikas.
Lightseminar: Learned Representation in AI An Introduction to Locally Linear Embedding Lawrence K. Saul Sam T. Roweis presented by Chan-Su Lee.
Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)
Nonlinear Dimensionality Reduction by Locally Linear Embedding Sam T. Roweis and Lawrence K. Saul Reference: "Nonlinear dimensionality reduction by locally.
Nonlinear Dimensionality Reduction Approaches. Dimensionality Reduction The goal: The meaningful low-dimensional structures hidden in their high-dimensional.
Manifold learning: Locally Linear Embedding Jieping Ye Department of Computer Science and Engineering Arizona State University
1 Graph Embedding (GE) & Marginal Fisher Analysis (MFA) 吳沛勳 劉冠成 韓仁智
Estimating Intrinsic Dimension Justin Eberhardt UMD, Mathematics and Statistics Advisor: Dr. Kang James.
Graph Embedding: A General Framework for Dimensionality Reduction Dong XU School of Computer Engineering Nanyang Technological University
IEEE TRANSSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
Data Reduction. 1.Overview 2.The Curse of Dimensionality 3.Data Sampling 4.Binning and Reduction of Cardinality.
THE MANIFOLDS OF SPATIAL HEARING Ramani Duraiswami | Vikas C. Raykar Perceptual Interfaces and Reality Lab University of Maryland, College park.
Computer Vision Lab. SNU Young Ki Baik Nonlinear Dimensionality Reduction Approach (ISOMAP, LLE)
Computer examples Tenenbaum, de Silva, Langford “A Global Geometric Framework for Nonlinear Dimensionality Reduction”
ELEC 303 – Random Signals Lecture 18 – Classical Statistical Inference, Dr. Farinaz Koushanfar ECE Dept., Rice University Nov 4, 2010.
A New Method of Probability Density Estimation for Mutual Information Based Image Registration Ajit Rajwade, Arunava Banerjee, Anand Rangarajan. Dept.
WISP: Nov. 04 Inference and Signal Processing for Networks ALFRED O. HERO III Depts. EECS, BME, Statistics University of Michigan - Ann Arbor
ISOMAP TRACKING WITH PARTICLE FILTER Presented by Nikhil Rane.
GRASP Learning a Kernel Matrix for Nonlinear Dimensionality Reduction Kilian Q. Weinberger, Fei Sha and Lawrence K. Saul ICML’04 Department of Computer.
Dimensionality Reduction
Manifold learning: MDS and Isomap
Nonlinear Dimensionality Reduction Approach (ISOMAP)
H. Lexie Yang1, Dr. Melba M. Crawford2
Data Mining Course 0 Manifold learning Xin Yang. Data Mining Course 1 Outline Manifold and Manifold Learning Classical Dimensionality Reduction Semi-Supervised.
Non-Linear Dimensionality Reduction
Project 11: Determining the Intrinsic Dimensionality of a Distribution Okke Formsma, Nicolas Roussis and Per Løwenborg.
Project 11: Determining the Intrinsic Dimensionality of a Distribution Okke Formsma, Nicolas Roussis and Per Løwenborg.
Tony Jebara, Columbia University Advanced Machine Learning & Perception Instructor: Tony Jebara.
Supervisor: Nakhmani Arie Semester: Winter 2007 Target Recognition Harmatz Isca.
Data Mining Course 2007 Eric Postma Clustering. Overview Three approaches to clustering 1.Minimization of reconstruction error PCA, nlPCA, k-means clustering.
CSC321: Lecture 25: Non-linear dimensionality reduction Geoffrey Hinton.
A Binary Linear Programming Formulation of the Graph Edit Distance Presented by Shihao Ji Duke University Machine Learning Group July 17, 2006 Authors:
CSC321: Extra Lecture (not on the exam) Non-linear dimensionality reduction Geoffrey Hinton.
Manifold Learning JAMES MCQUEEN – UW DEPARTMENT OF STATISTICS.
1 C.A.L. Bailer-Jones. Machine Learning. Data exploration and dimensionality reduction Machine learning, pattern recognition and statistical data modelling.
Spectral Methods for Dimensionality
Nonlinear Dimensionality Reduction
Intrinsic Data Geometry from a Training Set
Unsupervised Riemannian Clustering of Probability Density Functions
ISOMAP TRACKING WITH PARTICLE FILTERING
Outline Nonlinear Dimension Reduction Brief introduction Isomap LLE
Outline H. Murase, and S. K. Nayar, “Visual learning and recognition of 3-D objects from appearance,” International Journal of Computer Vision, vol. 14,
CS4670: Intro to Computer Vision
Using Manifold Structure for Partially Labeled Classification
NonLinear Dimensionality Reduction or Unfolding Manifolds
Goodfellow: Chapter 14 Autoencoders
Presentation transcript:

Entropic graphs: Applications Alfred O. Hero Dept. EECS, Dept BME, Dept. Statistics University of Michigan - Ann Arbor 1.Dimension reduction and pattern matching 2.Entropic graphs for manifold learning 3.Simulation studies 4.Applications to face and digit databases

1.Dimension Reduction and Pattern Matching 128x128 images of faces Different poses, illuminations, facial expressions The set of all face images evolve on a lower dimensional imbedded manifold in R^(16384)

Face Manifold

Classification on Face Manifold

Manifold Learning: What is it good for? Interpreting high dimensional data Discovery and exploitation of lower dimensional structure Deducing non-linear dependencies between populations Improving detection and classification performance Improving image compression performance

Background on Manifold Learning 1.Manifold intrinsic dimension estimation 1.Local KLE, Fukunaga, Olsen (1971) 2.Nearest neighbor algorithm, Pettis, Bailey, Jain, Dubes (1971) 3.Fractal measures, Camastra and Vinciarelli (2002) 4.Packing numbers, Kegl (2002) 2.Manifold Reconstruction 1.Isomap-MDS, Tenenbaum, de Silva, Langford (2000) 2.Locally Linear Embeddings (LLE), Roweiss, Saul (2000) 3.Laplacian eigenmaps (LE), Belkin, Niyogi (2002) 4.Hessian eigenmaps (HE), Grimes, Donoho (2003) 3.Characterization of sampling distributions on manifolds 1.Statistics of directional data, Watson (1956), Mardia (1972) 2.Data compression on 3D surfaces, Kolarov, Lynch (1997) 3.Statistics of shape, Kendall (1984), Kent, Mardia (2001)

A statistical sample Sampling distribution 2dim manifold Domain Sampling Embedding Sampling on a Domain Manifold Observed sample

Learning 3D Manifolds Ref: Tenenbaum&etal (2000) Sampling density fy = Uniform on manifold N=400N=800 Ref: Roweiss&etal (2000) Swiss RollS-Curve

Sampled S-curve What is shortest path between points A and B along manifold? A B Geodesic from A to B is shortest path Euclidean Path is poor approximation

Geodesic Graph Path Approximation A B Dykstra’s shortest path approximates geodesic k-NNG skeleton k=4

ISOMAP (PCA) Reconstruction Compute k-NN skeleton on observed sample Run Dykstra’s shortest path algorithm between all pairs of vertices of k-NN Generate Geodesic pairwise distance matrix approximation Perform MDS on Reconstruct sample in manifold domain

ISOMAP Convergence When domain mapping is an isometry, domain is open and convex, and true domain dimension d is known (de Silva&etal:2001): How to estimate d? How to estimate attributes of sampling density?

How to Estimate d? Landmark-ISOMAP residual curve For Abilene Netflow data set

2. Entropic Graphs in D-dimensional Euclidean space Euclidean MST with edge power weighting gamma: pairwise distance matrix over edge length matrix of spanning trees over Euclidean k-NNG with edge power weighting gamma: When obtain Geodesic MST

Example: Uniform Planar Sample

Example: MST on Planar Sample

Example: k-NNG on Planar Sample

Convergence of Euclidean MST Beardwood, Halton, Hammersley Theorem:

GMST Convergence Theorem Ref: Costa&Hero:TSP2003

k-NNG Convergence Theorem

Shrinkwrap Interpretation n=400 n=800 Dimension = “Shrinkage rate” as vary number of resampled points on M

Joint Estimation Algorithm Convergence theorem suggests log-linear model Use bootstrap resampling to estimate mean graph length and apply LS to jointly estimate slope and intercept from sequence Extract d and H from slope and intercept

3. Simulation Studies: Swiss Roll GMSTkNN K=4 n=400, f=Uniform on manifold

Estimates of GMST Length Bootstrap SE bar (83% CI)

loglogLinear Fit to GMST Length

GMST Dimension and Entropy Estimates From LS fit find: Intrinsic dimension estimate Alpha-entropy estimate ( ) –Ground truth:

MST/kNN Comparisons n=800n=400 n=800n=400 MST kNN

Entropic Graphs on S2 Sphere in 3D n=500, f=Uniform on manifold GMSTkNN

k-NNG on Sphere S4 in 5D Histogram of resampled d-estimates of k-NNG N=1000 points uniformly distributed on S4 (sphere) in 5D k=7 for all algorithms kNN resampled 5 times Length regressed on 10 or 20 samples at end of mean length sequence 30 experiments performed ISOMAP always estimates d=5 Table of relative frequencies of correct d estimate n

Estimated entropy (n = 600) True Entropy kNN/GMST Comparisons

GMST4-NN kNN/GMST Comparisons for Uniform Hyperplane

Improve Performance by Bootstrap Resampling Main idea: Averaging of weak learners –Using fewer (N) samples per MST estimate, generate large number (M) of weak estimates of d and H –Reduce bias by averaging these estimates (M>>1,N=1) –Better than optimizing estimate of MST length (M=1,N>>1) Illustration of bootstrap resampling method: A,B: N=1 vs C: M=1

Table of relative frequencies of correct d estimate using the GMST, with (N = 1) and without (M = 1) bias correction. kNN/GMST Comparisons for Uniform Hyperplane

4. Application: ISOMAP Face Database Synthesized 3D face surface Computer generated images representing 700 different angles and illuminations Subsampled to 64 x 64 resolution (D=4096) Disagreement over intrinsic dimensionality –d=3 (Tenenbaum) vs d=4 (Kegl) Resampling Histogram of d hat Mean GMST Length Function d=3 H=21.1 bits Mean kNNG (k=7) length d=4 H=21.8 bits

Application: Yale Face Database Description of Yale face database 2 –Photographic folios of many people’s faces –Each face folio contains images at 585 different illumination/pose conditions –Subsampled to 64 by 64 pixels (4096 extrinsic dimensions) Objective: determine intrinsic dimension and entropy of a typical face folio

Samples from Face database B

GMST for 3 Face Folios

Real valued intrinsic dimension estimates using 3-NN graph for face 1. Real valued intrinsic dimension estimates using 3-NN graph for face 2. Dimension Estimator Histograms for Face database B

Remarks on Yale Facebase B GMST LS estimation parameters –Local Geodesic approximation used to generate pairwise distance matrix –Estimates based on 25 resamplings over 18 largest folio sizes To represent any folio we might hope to attain –factor > 600 reduction in degrees of freedom (dim) –only 1/10 bit per pixel for compression –a practical parameterization/encoder?

Sample: MNIST Handwritten Digits Application: MNIST Digit Database

Estimated intrinsic dimension Histogram of intrinsic dimension estimates: GMST (left) and 5-NN (right) (M = 1, N = 10, Q = 15). MNIST Digit Database

ISOMAP (k = 6) residual variance plot. The digits database contains nonlinear transformations, such as width distortions of each digit, that are not adequately modeled by ISOMAP! MNIST Digit Database

Conclusions Entropic graphs give accurate global and consistent estimators of dimension and entropy Manifold learning and model reduction –LLE, LE, HE estimate d by finding local linear representation of manifold –Entropic graph estimates d from global resampling –Initialization of ISOMAP… with entropic graph estimator Computational considerations –GMST, kNN with pairwise distance matrix: O(E log E) –GMST with greedy neighborhood search: O(d n log n) –kNN with kdb tree partitioning: O(d n log n)

References A. O. Hero, B. Ma, O. Michel and J. D. Gorman, “Application of entropic graphs,” IEEE Signal Processing Magazine, Sept H. Neemuchwala, A.O. Hero and P. Carson, “Entropic graphs for image registration,” to appear in European Journal of Signal Processing, J. Costa and A. O. Hero, “Manifold learning with geodesic minimal spanning trees,” to appear in IEEE T- SP (Special Issue on Machine Learning), A. O. Hero, J. Costa and B. Ma, "Convergence rates of minimal graphs with random vertices," submitted to IEEE T-IT, March J. Costa, A. O. Hero and C. Vignat, "On solutions to multivariate maximum alpha-entropy Problems", in Energy Minimization Methods in Computer Vision and Pattern Recognition (EMM-CVPR), Eds. M. Figueiredo, R. Rangagaran, J. Zerubia, Springer-Verlag, 2003