1. Globally Optimal Estimates for Geometric Reconstruction Problems Tom Gilat, Adi Lakritz Advanced Topics in Computer Vision Seminar Faculty of Mathematics and Computer Science Weizmann Institute 3 June 2007

2. outline  Motivation and Introduction  Background Positive SemiDefinite matrices (PSD) Linear Matrix Inequalities (LMI) SemiDefinite Programming (SDP)  Relaxations Sum Of Squares (SOS) relaxation Linear Matrix Inequalities (LMI) relaxation  Application in vision Finding optimal structure Partial relaxation and Schur ’ s complement

3. Motivation Geometric Reconstruction Problems Polynomial optimization problems (POPs)

4. Triangulation problem in L 2 norm Multi view - optimization2-views – exact solution

5. Triangulation problem in L 2 norm perspective camera i x y z err

6. Triangulation problem in L 2 norm minimize reprojection error in all cameras Polynomial minimization problem Non convex

7. More computer vision problems Reconstruction problem: known cameras, known corresponding points find 3D points that minimize the projection error of given image points –Similar to triangulation for many points and cameras Calculating homography given 3D points on a plane and corresponding image points, calculate homography Many more problems

8. Optimization problems

9. Introduction to optimization problems

10. optimization problems NP - complete

11. optimization problems optimization convexnon convex Linear Programming (LP) SemiDefinite Programming (SDP) solutions exist: interior point methods problems: local optimum or high computational cost

12. non convex optimization init level curves of f Many algorithms Get stuck in local minima MinMax Non convex feasible set

13. optimization problems optimization convexnon convex LPSDP solutions exist: interior point methods problems: local optimum or high computational cost global optimization – algorithms that converge to optimal solution relaxation of problem

14. outline  Motivation and Introduction  Background Positive SemiDefinite matrices (PSD) Linear Matrix Inequalities (LMI) SemiDefinite Programming (SDP)  Relaxations Sum Of Squares (SOS) relaxation Linear Matrix Inequalities (LMI) relaxation  Application in vision Finding optimal structure Partial relaxation and Schur ’ s complement

15. positive semidefinite (PSD) matrices Definition: a matrix M in R n×n is PSD if 1. M is symmetric: M=M T 2. for all M can be decomposed as AA T (Cholesky) Proof:

16. positive semidefinite (PSD) matrices Definition: a matrix M in R n×n is PSD if 1. M is symmetric: M=M T 2. for all M can be decomposed as AA T (Cholesky)

17. principal minors The kth order principal minors of an n×n symmetric matrix M are the determinants of the k×k matrices obtained by deleting n - k rows and the corresponding n - k columns of M first order: elements on diagonal second order:

18. diagonal minors The kth order principal minors of an n×n symmetric matrix M are the determinants of the k×k matrices obtained by deleting n - k rows and the corresponding n - k columns of M first order: elements on diagonal second order:

19. diagonal minors The kth order principal minors of an n×n symmetric matrix M are the determinants of the k×k matrices obtained by deleting n - k rows and the corresponding n - k columns of M first order: elements on diagonal second order:

20. diagonal minors The kth order principal minors of an n×n symmetric matrix M are the determinants of the k×k matrices obtained by deleting n - k rows and the corresponding n - k columns of M first order: elements on diagonal third order: det(M) second order:

21. Set of PSD matrices in 2D

22. Set of PSD matrices This set is convex Proof:

23. LMI – linear matrix inequality

24. LMI example: find the feasible set of the 2D LMI

25. reminder

26. LMI example: find the feasible set of the 2D LMI 1st order principal minors

27. LMI example: find the feasible set of the 2D LMI 2nd order principal minors

28. LMI example: find the feasible set of the 2D LMI 3rd order principal minors Intersection of all inequalities

29. Semidefinite Programming (SDP) = LMI an extension of LP

30. outline  Motivation and Introduction  Background Positive SemiDefinite matrices (PSD) Linear Matrix Inequalities (LMI) SemiDefinite Programming (SDP)  Relaxations Sum Of Squares (SOS) relaxation Linear Matrix Inequalities (LMI) relaxation  Application in vision Finding optimal structure Partial relaxation and Schur ’ s complement

31. Sum Of Squares relaxation (SOS) Unconstrained polynomial optimization problem (POP) means the feasible set is R n H. Waki, S. Kim, M. Kojima, and M. Muramatsu. SOS and SDP relaxations for POPs with structured sparsity. SIAM J. Optimization, 2006.

32. Sum Of Squares relaxation (SOS)

33. SOS relaxation for unconstrained polynomials

35. monomial basis example

36. SOS relaxation to SDP SDP

37. SOS relaxation to SDP example: SDP

38. SOS for constrained POPs possible to extend this method for constrained POPs by use of generalized Lagrange dual

39. SOS relaxation summary So we know how to solve a POP that is a SOS And we have a bound on a POP that is not an SOS H. Waki, S. Kim, M. Kojima, and M. Muramatsu. SOS and SDP relaxations for POPs with structured sparsity. SIAM J. Optimization, 2006. POP SOS problem SOS relaxation Global estimate SDP

40. relaxations SOS: POP SOS problem SOS relaxation Global estimate SDP POP linear & LMI problem LMI relaxation Global estimate SDP + converge LMI:

41. outline  Motivation and Introduction  Background Positive SemiDefinite matrices (PSD) Linear Matrix Inequalities (LMI) SemiDefinite Programming (SDP)  Relaxations Sum Of Squares (SOS) relaxation Linear Matrix Inequalities (LMI) relaxation  Application in vision Finding optimal structure Partial relaxation and Schur ’ s complement

42. LMI relaxations Constraints are handled Convergence to optimum is guaranteed Applies to all polynomials, not SOS as well

43. A maximization problem Note that: a.Feasible set is non-convex. b.Constraints are quadratic Feasible set

44. LMI – linear matrix inequality, a reminder

45. Goal An SDP: Motivation Polynomial Optimization Problem SDP with solution close to global optimum of the original problem What is it good for? SDP problems can be solved much more efficiently then general optimization problems.

46. POP Linear + LMI + rank constraints SDP LMI: LMI Relaxations is iterative process Step 1: introduce new variables Step 2: relax constraints Apply higher order relaxations

47. LMI relaxation – step 1 Replace monomials by “lifting variables” Rule: Example: (the R 2 case)

48. Introducing lifting variables Lifting

49. Not equivalent to the original problem. Lifting variables are not independent in the original problem: New problem is linear, in particular convex

50. Goal, more specifically Linear problem (obtained by lifing) “relations constraints” on lifting variables + SDP Relaxation

51. Question: how do we guarantee that the relations between lifting variables hold?

52. LMI relaxation – step 2 Apply lifting and get: Take the basis of the degree 1 polynomials. Note that: Because:

53. If the relations constraints hold then This is because we can decompose M as follows: Assuming relations hold Rank M = 1

54. We’ve seen: Relations constraints hold What about the opposite: Relations constraints hold This is true as well

55. By the following: LMI relaxation – step 2, continued Relations constraints hold All relations equalities are in the set of equalities

56. Conclusion of the analysis The “y feasible set” Subset of feasible set with Relations constraints hold

57. Original problem is equivalent to the following: together with the additional constraint Relaxation, at last We denote moment matrix of order 1 Relax by dropping the non-convex constraint

58. LMI relaxation of order 1 Feasible set

59. Rank constrained LMI vs. unconstrained

60. LMI relaxations of higher order It turns out that we can do better: Apply LMI relaxations of higher order A tighter SDP Relaxations of higher order incorporate the inequality constraints in LMI We show relaxation of order 2 It is possible to continue and apply relaxations Theory guarantees convergence to global optimum

61. Let be a basis of polynomials of degree 2. Again, Lifting gives: Again, we will relax by dropping the rank constraint. LMI relaxations of second order

62. Inequality constraints to LMI Replace our constraints by LMIs and have a tighter relaxation. Linear Constraint : Lifting LMI Constraint : For example,

63. This procedure brings a new SDP LMI relaxations of order 2 Second SDP feasible set is included in the first SDP feasible set Silimarly, we can continue and apply higher order relaxations.

64. If the feasible set defined by constraints is compact, then under mild additional assumptions, Lassere proved in 2001 that there is an asymptotic convergence guarantee: is the solution to k’th relaxation is the solution for the original problem (finding a maximum) Theoretical basis for the LMI relaxations Moreover, convergence is fast: is very close to for small k Lasserre J.B. (2001) "Global optimization with polynomials and the problem of moments" SIAM J. Optimization 11, pp 796--817.

65. The method provides a certificate of global optimality: An important experimental observation: SDP solution is global optimum Minimizing Low rank moment matrix Checking global optimality

66. We add to the objective function the trace of the moment matrix weighted by a sufficiently small positive scalar

67. LMI relaxations in vision Application

68. outline  Motivation and Introduction  Background Positive SemiDefinite matrices (PSD) Linear Matrix Inequalities (LMI) SemiDefinite Programming (SDP)  Relaxations Sum Of Squares (SOS) relaxation Linear Matrix Inequalities (LMI) relaxation  Application in vision Finding optimal structure Partial relaxation and Schur ’ s complement

69. Camera center Image plane Finding optimal structure A perspective camera P is the camera matrix and is the depth. Measured image points are corrupted by independent Gaussian noise. We want to minimize the least squares errors between measured and projected points. The relation between a U in the 3D space and u in the image plane is given by:

70. We therefore have the following optimization problem: is the Euclidean distance. is the set of unknowns Where are polynomials. Each term in the cost function can be written as: Our objective is therefore to minimize a sum of rational functions.

71. How can we turn the optimization problem into a polynomial optimization problem? Suppose that each term in has an upper bound, then Then our optimization problem is equivalent to the following: This is a polynomial optimization problem, for which we apply LMI relaxations. Note that we introduced many new variables – one for each term.

72. Problem: an SDP with a large number of variables can be computationally demanding. A large number of variables can arise from: LMI relaxations of high order Introduction of new variables as we’ve seen This is where partial relaxations come in. For that we introduce Schur’s complement. Partial Relaxations

73. Schur’s comlement Set:

74. Schur’s comlement - applying Derivation of right side: C - B T * A -1 * B > 0

75. Schur’s complement allows us to state our optimization problem as follows: Partial relaxations The only non-linearity is due to We can apply LMI relaxations only on and leave If we were to apply full relaxations for all variables, the problem would become Intractable for small N.

76. Partial relaxations Disadvantage of partial relaxations: we are not able to ensure asymptotic convergence to the global optimum. However, we have a numerical certificate of global optimality just as in the case of full relaxations: The moment matrix of the relaxed variables is of rank one Solution of partially relaxed problem is the global optimum

77. Full relaxation vs. partial relaxtion Application: Triangulation, 3 cameras Goal: find the optimal 3D point. Camera matrices are known, measured point is assumed to be in the origin of each view. Camera matrices:

78. Summary Geometric Vision Problems to POPs –Triangulation and reconstruction problem Relaxations of POPs –Sum Of Squares (SOS) relaxation Guarantees bound on optimal solution Usually solution is optimal –Linear Matrix Inequalities (LMI) relaxation First order LMI relaxation: lifting, dropping rank constraint Higher order LMI relaxation: linear constraints to LMIs Guarantee of convergence, reference to Lassere Certificate of global optimality Application in vision Finding optimal structure Partial relaxation and Schur’s complement Triangulation problem, benefit of partial relaxations

79. References  F. Kahl and D. Henrion. Globally Optimal Estimates for Geometric Reconstruction Problems. Accepted IJCV  H. Waki, S. Kim, M. Kojima, and M. Muramatsu. Sums of squares and semidefinite programming relaxations for polynomial optimization problems with structured sparsity. SIAM J. Optimization, 17(1):218 – 242, 2006.  J. B. Lasserre. Global optimization with polynomials and the problem of moments. SIAM J. Optimization, 11:796 – 817, 2001.  S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.  R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2004. Second Edition.