1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology

2 M. Bronstein Multigrid multidimensional scaling Agenda Applications of MDS Numerical optimization algorithms Motivation for multiresolution MDS methods Multigrid MDS Experimental results Conclusions

3 M. Bronstein Multigrid multidimensional scaling Dimensionality reduction Visualization Pattern recognition Feature extraction Data analysis WRIST ROTATION FINGER EXTENSION Low-dimensional representation of articulated hand images, showing intrinsic data dimensionality Images: World Wide Web

4 M. Bronstein Multigrid multidimensional scaling Given a surface sampled at points, and the geodesic distances on ; Find a mapping (isometric embedding) such that Isometric embedding

5 M. Bronstein Multigrid multidimensional scaling GLOBE (HEMISPHERE)PLANAR MAP Mapmaking Given: geodesic distances between cities on the Earth Find: the “best” (most distance-preserving) planar map of the cities Optimal planar representation of the upper hemisphere of the Earth

6 M. Bronstein Multigrid multidimensional scaling Pattern recognition A. Elad, R. Kimmel, Proc. CVPR 2001 ISOMETRIES OF A DEFORMABLE OBJECT ISOMETRY-INVARIANT REPRESENTATIONS (“CANONICAL FORMS”) Isometry-invariant representation of deformable objects using isometric embedding

7 M. Bronstein Multigrid multidimensional scaling Expression-invariant face recognition ISOMETRIC EMBEDDING FACIAL CONTOUR CROPPING FACE SUBSAMPLING CANONICAL FORM Facial expressions ~ isometries of the facial surface Obtain expression-invariant representation using isometric embedding Compare the canonical forms A. Bronstein, M. Bronstein, R. Kimmel, Proc. AVBPA 2003; IJCV 2005 Scheme of expression-invariant 3D face recognition based on isometric embedding DISTANCES COMPUTATION

8 M. Bronstein Multigrid multidimensional scaling Stress Given a set of distances ; and a configuration of points in -dimensional Euclidean space ; Representation quality can be measured as the -distortion of the distances (stress)

9 M. Bronstein Multigrid multidimensional scaling Multidimensional scaling Stress in matrix form: - matrix of geodesic distances (data); - matrix of Euclidean coordinates (variable); Multidimensional scaling (MDS) problem: optimization variables Optimum defined up to an isometry in

10 M. Bronstein Multigrid multidimensional scaling Minimization of the stress Generic iterative optimization algorithm: Start with an initial guess ; At -st iteration, make a step of size in direction such that Repeat until a stopping condition is met, e.g.

11 M. Bronstein Multigrid multidimensional scaling Optimization algorithms Gradient descent:, step size is constant or found using line search Newton:, step size is found using line search Truncated Newton: direction obtained by inexact solution of step size is chosen to guarantee descent Quasi-Newton: direction obtained by estimating using the gradients ; step size is found using line search

12 M. Bronstein Multigrid multidimensional scaling Difficulties Non-convex and nonlinear optimization problem (local convergence) Hessian structured but dense High computation complexity of and Exact line search is prohibitive for large

13 M. Bronstein Multigrid multidimensional scaling SMACOF algorithm SMACOF: steepest descent with constant step size where and Can be also written as a multiplicative update Complexity: per iteration

14 M. Bronstein Multigrid multidimensional scaling Multiresolution methods: motivation Data smoothness and locality (a point can be interpolated from its neighbors) Complexity: - MDS problem is easier on coarser resolution Local minima: multiple resolutions improve global convergence

15 M. Bronstein Multigrid multidimensional scaling Towards multigrid MDS Convex nonlinear optimization is equivalent to a nonlinear equation Multigrid spirit: solve problems of the form at different resolution levels - residual transferred from finer resolution levels M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

16 M. Bronstein Multigrid multidimensional scaling Modified stress Problem: the function is unbounded Modified stress: The penalty term forces the center of mass of to zero With modified stress, is bounded for every finite M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

17 M. Bronstein Multigrid multidimensional scaling Multigrid components Hierarchy of grids Restriction and prolongation operators to transfer data and variables from one resolution level to another Hierarchy of optimization problems Relaxation: steps of optimization algorithm M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

18 M. Bronstein Multigrid multidimensional scaling Coarsening schemes In parameterization domain (suitable for parametric surfaces, e.g. acquired by 3D scanner) Triangulation-based (suitable for general triangulated meshes) Farthest point sampling (based on the distances matrix; suitable for arbitrary multidimensional data)

19 M. Bronstein Multigrid multidimensional scaling V-cycle If (coarsest level), solve and return Else M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005 Relaxation Compute Apply MG on coarser resolution Correction Relaxation

20 M. Bronstein Multigrid multidimensional scaling Error smoothing BEFORE RELAXATIONAFTER RELAXATION Error smoothing using SMACOF relaxation M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

21 M. Bronstein Multigrid multidimensional scaling Numerical experiments Embedding of the “Swiss roll” surface – comparison of MDS algorithms convergence in a large scale problem Computation of canonical forms for face recognition Sensitivity to initialization and comparison on problems of different size Dimensionality reduction

22 M. Bronstein Multigrid multidimensional scaling Experiment I: Unrolling the Swiss roll Embedding of the Swiss roll objects into R 3 using MG-MDS. N=2145 INITIALIZATIONITERATION 1 ITERATION 2 ITERATION 3ITERATION 4 M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

23 M. Bronstein Multigrid multidimensional scaling Experiment I: Convergence comparison Convergence of different algorithms in the Swiss roll problem COMPLEXITY (MFLOPs) STRESS EXECUTION TIME (sec.) M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

24 M. Bronstein Multigrid multidimensional scaling Experiment II: Facial surface embedding Computation of a facial canonical form using MG-MDS: as few as 3 iterations are sufficient to obtain a good expression-invariant representation. N=1997 INITIALIZATIONITERATION 1 ITERATION 2 ITERATION 3 M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

25 M. Bronstein Multigrid multidimensional scaling Performance of SMACOF and MG (V-cycle, 3 resolution levels) MDS algorithms using random initialization M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005 Experiment III: Sensitivity to initialization

26 M. Bronstein Multigrid multidimensional scaling Boosting obtained by multigrid MDS (V-cycle, 3 resolution levels) compared to SMACOF. Initialization by the original points M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005 Experiment III: Performance comparison

27 M. Bronstein Multigrid multidimensional scaling Experiment IV: Dimensionality reduction Dimensionality reduction of 500-dimensional random data: as few as 3 iterations are sufficient to obtain distinguishable clusters. INITIALIZATIONITERATION 1 ITERATION 2 ITERATION 3 Two sets of random binary i.i.d. 500-dimensional vectors Set A: Set B:

28 M. Bronstein Multigrid multidimensional scaling MG-MDS significantly outperforms traditional MDS algorithms (~order of magnitude) The improvement is more pronounced for large N MG-MDS appears to be less sensitive to initialization and has better global convergence Conclusions

29 M. Bronstein Multigrid multidimensional scaling References M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "Multigrid multidimensional scaling", NLAA, to appear in 2006 M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "A multigrid approach for multi- dimensional scaling", Proc. Copper Mountain Conf. Multigrid Methods, A. M. Bronstein, M. M. Bronstein, and R. Kimmel. “Expression invariant face recognition: faces as isometric surfaces”, in “Face Processing: Advanced Modeling and Methods”, Academic Press, in press. A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Three-dimensional face recognition", Intl. Journal of Computer Vision (IJCV), Vol. 64/1, pp. 5-30, August A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Expression-invariant 3D face recognition", Proc. AVBPA, Lecture Notes in Comp. Science No. 2688, Springer, pp , 2003.