1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M. Bronstein Department of Computer Science Technion – Israel.

Slides:



Advertisements
Similar presentations
Alignment Visual Recognition “Straighten your paths” Isaiah.
Advertisements

Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.
Engineering Optimization
1 Numerical Solvers for BVPs By Dong Xu State Key Lab of CAD&CG, ZJU.
CS 290H 7 November Introduction to multigrid methods
Presented by: Mingyuan Zhou Duke University, ECE April 3, 2009
Non-Rigid Registration of 3D Surface by Deformable 2D Triangular Meshes Speaker:James Chang.
Instructor: Mircea Nicolescu Lecture 13 CS 485 / 685 Computer Vision.
Classification and Prediction: Regression Via Gradient Descent Optimization Bamshad Mobasher DePaul University.
Two Technique Papers on High Dimensionality Allan Rempel December 5, 2005.
Algebraic MultiGrid. Algebraic MultiGrid – AMG (Brandt 1982)  General structure  Choose a subset of variables: the C-points such that every variable.
1 Processing & Analysis of Geometric Shapes Shortest path problems Shortest path problems The discrete way © Alexander & Michael Bronstein, ©
1 Michael Bronstein Computational metric geometry Computational metric geometry Michael Bronstein Department of Computer Science Technion – Israel Institute.
Multidimensional scaling
Scale Normalization for Isometric Shape Matching Yusuf Sahillioğlu and Yücel Yemez Computer Eng. Dept., Koç University, Istanbul, Turkey.
Numerical Optimization
Isometry invariant similarity
1 Michael Bronstein 3D face recognition Face recognition: New technologies, new challenges Michael M. Bronstein.
1 Numerical geometry of non-rigid shapes Lecture I – Introduction Numerical geometry of shapes Lecture I – Introduction non-rigid Michael Bronstein.
Dimensionality Reduction and Embeddings
Numerical geometry of non-rigid shapes
Numerical geometry of objects
1 Bronstein 2 and Kimmel Extrinsic and intrinsic similarity of nonrigid shapes Michael M. Bronstein Department of Computer Science Technion – Israel Institute.
Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.
Correspondence & Symmetry
1 Numerical geometry of non-rigid shapes Spectral Methods Tutorial. Spectral Methods Tutorial 6 © Maks Ovsjanikov tosca.cs.technion.ac.il/book Numerical.
1 Numerical geometry of non-rigid shapes Lecture II – Numerical Tools Numerical geometry of shapes Lecture II – Numerical Tools non-rigid Alex Bronstein.
Numerical geometry of non-rigid shapes
Optimization Methods One-Dimensional Unconstrained Optimization
1 Bronstein, Bronstein, and Kimmel Joint extrinsic and intrinsic similarity of non-rigid shapes Rock, paper, and scissors Joint extrinsic and intrinsic.
1 Numerical geometry of non-rigid shapes A journey to non-rigid world objects Numerical methods non-rigid Alexander Bronstein Michael Bronstein Numerical.
Non-Euclidean Embedding
Easy Optimization Problems, Relaxation, Local Processing for a single variable.
Flattening via Multi- Dimensional Scaling Ron Kimmel Computer Science Department Geometric Image Processing Lab Technion-Israel.
Numerical geometry of non-rigid shapes
A Global Geometric Framework for Nonlinear Dimensionality Reduction Joshua B. Tenenbaum, Vin de Silva, John C. Langford Presented by Napat Triroj.
1 Bronstein 2 & Kimmel An isometric model for facial animation and beyond AMDO, Puerto de Andratx, 2006 An isometric model for facial animation and beyond.
Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.
1 Numerical Geometry of Non-Rigid Shapes Invariant shape similarity Invariant shape similarity © Alexander & Michael Bronstein, © Michael Bronstein,
Advanced Topics in Optimization
Atul Singh Junior Undergraduate CSE, IIT Kanpur.  Dimension reduction is a technique which is used to represent a high dimensional data in a more compact.
Dimensionality Reduction
1 Numerical geometry of non-rigid shapes Non-Euclidean Embedding Non-Euclidean Embedding Lecture 6 © Alexander & Michael Bronstein tosca.cs.technion.ac.il/book.
1 Michael M. Bronstein Partial similarity of objects 17 December 2006 Partial similarity of objects, or how to compare a centaur to a horse Michael M.
1 Bronstein 2 & Kimmel Matching 2D articulated shapes using GMDS AMDO, Puerto de Andratx, 2006 Matching 2D articulated shapes using Generalized Multidimensional.
NonLinear Dimensionality Reduction or Unfolding Manifolds Tennenbaum|Silva|Langford [Isomap] Roweis|Saul [Locally Linear Embedding] Presented by Vikas.
Lightseminar: Learned Representation in AI An Introduction to Locally Linear Embedding Lawrence K. Saul Sam T. Roweis presented by Chan-Su Lee.
Dimensionality Reduction. Multimedia DBs Many multimedia applications require efficient indexing in high-dimensions (time-series, images and videos, etc)
1 M. Bronstein | Expression-invariant representation of faces and its applications for face recognition Expression-invariant representation of faces and.
כמה מהתעשייה? מבנה הקורס השתנה Computer vision.
Nonlinear Dimensionality Reduction Approaches. Dimensionality Reduction The goal: The meaningful low-dimensional structures hidden in their high-dimensional.
Computer Vision - Fitting and Alignment
Manifold learning: Locally Linear Embedding Jieping Ye Department of Computer Science and Engineering Arizona State University
7.1. Mean Shift Segmentation Idea of mean shift:
Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology Adaptive nonlinear manifolds and their applications to pattern.
Expression-invariant Face Recognition using Geodesic Distance Isometries Kerry Widder A Review of ‘Robust expression-invariant face recognition from partially.
Computer Vision Lab. SNU Young Ki Baik Nonlinear Dimensionality Reduction Approach (ISOMAP, LLE)
Multidimensional Scaling by Deterministic Annealing with Iterative Majorization Algorithm Seung-Hee Bae, Judy Qiu, and Geoffrey Fox SALSA group in Pervasive.
Manifold learning: MDS and Isomap
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Tony Jebara, Columbia University Advanced Machine Learning & Perception Instructor: Tony Jebara.
Introduction to Scientific Computing II Multigrid Dr. Miriam Mehl.
Deformation Modeling for Robust 3D Face Matching Xioguang Lu and Anil K. Jain Dept. of Computer Science & Engineering Michigan State University.
An SVM Learning Approach to Robotic Grasping Raphael Pelossof December Advanced Machine Learning.
Line Fitting James Hayes.
MultiGrid.
Spectral Methods Tutorial 6 1 © Maks Ovsjanikov
Efficient Methods for Roots of Univariate Scalar Beziers
Snakes, Shapes, and Gradient Vector Flow
Introduction to Scientific Computing II
Presentation transcript:

1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology

2 M. Bronstein Multigrid multidimensional scaling Agenda Applications of MDS Numerical optimization algorithms Motivation for multiresolution MDS methods Multigrid MDS Experimental results Conclusions

3 M. Bronstein Multigrid multidimensional scaling Dimensionality reduction Visualization Pattern recognition Feature extraction Data analysis WRIST ROTATION FINGER EXTENSION Low-dimensional representation of articulated hand images, showing intrinsic data dimensionality Images: World Wide Web

4 M. Bronstein Multigrid multidimensional scaling Given a surface sampled at points, and the geodesic distances on ; Find a mapping (isometric embedding) such that Isometric embedding

5 M. Bronstein Multigrid multidimensional scaling GLOBE (HEMISPHERE)PLANAR MAP Mapmaking Given: geodesic distances between cities on the Earth Find: the “best” (most distance-preserving) planar map of the cities Optimal planar representation of the upper hemisphere of the Earth

6 M. Bronstein Multigrid multidimensional scaling Pattern recognition A. Elad, R. Kimmel, Proc. CVPR 2001 ISOMETRIES OF A DEFORMABLE OBJECT ISOMETRY-INVARIANT REPRESENTATIONS (“CANONICAL FORMS”) Isometry-invariant representation of deformable objects using isometric embedding

7 M. Bronstein Multigrid multidimensional scaling Expression-invariant face recognition ISOMETRIC EMBEDDING FACIAL CONTOUR CROPPING FACE SUBSAMPLING CANONICAL FORM Facial expressions ~ isometries of the facial surface Obtain expression-invariant representation using isometric embedding Compare the canonical forms A. Bronstein, M. Bronstein, R. Kimmel, Proc. AVBPA 2003; IJCV 2005 Scheme of expression-invariant 3D face recognition based on isometric embedding DISTANCES COMPUTATION

8 M. Bronstein Multigrid multidimensional scaling Stress Given a set of distances ; and a configuration of points in -dimensional Euclidean space ; Representation quality can be measured as the -distortion of the distances (stress)

9 M. Bronstein Multigrid multidimensional scaling Multidimensional scaling Stress in matrix form: - matrix of geodesic distances (data); - matrix of Euclidean coordinates (variable); Multidimensional scaling (MDS) problem: optimization variables Optimum defined up to an isometry in

10 M. Bronstein Multigrid multidimensional scaling Minimization of the stress Generic iterative optimization algorithm: Start with an initial guess ; At -st iteration, make a step of size in direction such that Repeat until a stopping condition is met, e.g.

11 M. Bronstein Multigrid multidimensional scaling Optimization algorithms Gradient descent:, step size is constant or found using line search Newton:, step size is found using line search Truncated Newton: direction obtained by inexact solution of step size is chosen to guarantee descent Quasi-Newton: direction obtained by estimating using the gradients ; step size is found using line search

12 M. Bronstein Multigrid multidimensional scaling Difficulties Non-convex and nonlinear optimization problem (local convergence) Hessian structured but dense High computation complexity of and Exact line search is prohibitive for large

13 M. Bronstein Multigrid multidimensional scaling SMACOF algorithm SMACOF: steepest descent with constant step size where and Can be also written as a multiplicative update Complexity: per iteration

14 M. Bronstein Multigrid multidimensional scaling Multiresolution methods: motivation Data smoothness and locality (a point can be interpolated from its neighbors) Complexity: - MDS problem is easier on coarser resolution Local minima: multiple resolutions improve global convergence

15 M. Bronstein Multigrid multidimensional scaling Towards multigrid MDS Convex nonlinear optimization is equivalent to a nonlinear equation Multigrid spirit: solve problems of the form at different resolution levels - residual transferred from finer resolution levels M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

16 M. Bronstein Multigrid multidimensional scaling Modified stress Problem: the function is unbounded Modified stress: The penalty term forces the center of mass of to zero With modified stress, is bounded for every finite M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

17 M. Bronstein Multigrid multidimensional scaling Multigrid components Hierarchy of grids Restriction and prolongation operators to transfer data and variables from one resolution level to another Hierarchy of optimization problems Relaxation: steps of optimization algorithm M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

18 M. Bronstein Multigrid multidimensional scaling Coarsening schemes In parameterization domain (suitable for parametric surfaces, e.g. acquired by 3D scanner) Triangulation-based (suitable for general triangulated meshes) Farthest point sampling (based on the distances matrix; suitable for arbitrary multidimensional data)

19 M. Bronstein Multigrid multidimensional scaling V-cycle If (coarsest level), solve and return Else M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005 Relaxation Compute Apply MG on coarser resolution Correction Relaxation

20 M. Bronstein Multigrid multidimensional scaling Error smoothing BEFORE RELAXATIONAFTER RELAXATION Error smoothing using SMACOF relaxation M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

21 M. Bronstein Multigrid multidimensional scaling Numerical experiments Embedding of the “Swiss roll” surface – comparison of MDS algorithms convergence in a large scale problem Computation of canonical forms for face recognition Sensitivity to initialization and comparison on problems of different size Dimensionality reduction

22 M. Bronstein Multigrid multidimensional scaling Experiment I: Unrolling the Swiss roll Embedding of the Swiss roll objects into R 3 using MG-MDS. N=2145 INITIALIZATIONITERATION 1 ITERATION 2 ITERATION 3ITERATION 4 M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

23 M. Bronstein Multigrid multidimensional scaling Experiment I: Convergence comparison Convergence of different algorithms in the Swiss roll problem COMPLEXITY (MFLOPs) STRESS EXECUTION TIME (sec.) M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

24 M. Bronstein Multigrid multidimensional scaling Experiment II: Facial surface embedding Computation of a facial canonical form using MG-MDS: as few as 3 iterations are sufficient to obtain a good expression-invariant representation. N=1997 INITIALIZATIONITERATION 1 ITERATION 2 ITERATION 3 M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

25 M. Bronstein Multigrid multidimensional scaling Performance of SMACOF and MG (V-cycle, 3 resolution levels) MDS algorithms using random initialization M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005 Experiment III: Sensitivity to initialization

26 M. Bronstein Multigrid multidimensional scaling Boosting obtained by multigrid MDS (V-cycle, 3 resolution levels) compared to SMACOF. Initialization by the original points M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005 Experiment III: Performance comparison

27 M. Bronstein Multigrid multidimensional scaling Experiment IV: Dimensionality reduction Dimensionality reduction of 500-dimensional random data: as few as 3 iterations are sufficient to obtain distinguishable clusters. INITIALIZATIONITERATION 1 ITERATION 2 ITERATION 3 Two sets of random binary i.i.d. 500-dimensional vectors Set A: Set B:

28 M. Bronstein Multigrid multidimensional scaling MG-MDS significantly outperforms traditional MDS algorithms (~order of magnitude) The improvement is more pronounced for large N MG-MDS appears to be less sensitive to initialization and has better global convergence Conclusions

29 M. Bronstein Multigrid multidimensional scaling References M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "Multigrid multidimensional scaling", NLAA, to appear in 2006 M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "A multigrid approach for multi- dimensional scaling", Proc. Copper Mountain Conf. Multigrid Methods, A. M. Bronstein, M. M. Bronstein, and R. Kimmel. “Expression invariant face recognition: faces as isometric surfaces”, in “Face Processing: Advanced Modeling and Methods”, Academic Press, in press. A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Three-dimensional face recognition", Intl. Journal of Computer Vision (IJCV), Vol. 64/1, pp. 5-30, August A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Expression-invariant 3D face recognition", Proc. AVBPA, Lecture Notes in Comp. Science No. 2688, Springer, pp , 2003.