1. Ordinary Least-Squares Emmanuel Iarussi Inria

2. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface reconstruction 2

3. Color diffusion 3 Many graphics problems can be seen as finding the best set of parameters for a model, given some data

4. Shape registration 4 Many graphics problems can be seen as finding the best set of parameters for a model, given some data

5. Temperature (a)Gas pressure (b) Example: estimation of gas pressure for a given temperature sample 5

6. Assuming linear relationship 6

7. 7

8. Temperature measurements (m) Pressure measurements (m) 8

9. Assuming linear relationship Temperature measurements (m) Pressure measurements (m) = 9

10. Assuming linear relationship The relationship might not be exact Temperature measurements (m) Pressure measurements (m) = 10

11. Assuming linear relationship The relationship might not be exact Temperature measurements (m) Pressure measurements (m) … = Match “as best as possible” the observations 11

12. Assuming linear relationship The relationship might not be exact Match “as best as possible” the observations Temperature measurements (m) Pressure measurements (m) = 12

13. Many of CG problems can be formulated as minimizing the sum of squares of the residuals between some features in the model and the data. or 13

14. Matrix notation (observations) We can rewrite the residual function using linear algebra as: 14

15. Temperature Gas pressure Example: estimation of gas pressure for a given temperature and altitude samples Altitude 15

16. Multidimensional linear regression, using a model with n parameters 16

17. Multidimensional linear regression, using a model with n parameters before + +… 17

18. Multidimensional linear regression, using a model with n parameters Temperature measurements (m) Pressure measurements (m) Altitude measurements (m) 18

19. Multidimensional linear regression, using a model with n parameters Temperature measurements (m) Pressure measurements (m) Altitude measurements (m) = 19

20. = 20

21. = 21

22. = 22

23. Objective: find x subject to minimize: 23

24. Objective: find x subject to minimize: 24

25. Objective: find x subject to minimize: Convex bowl function has minima when: 25

26. Objective: find x subject to minimize: Convex bowl function has minima when: gradient 26

27. Expanding square term 27

28. Expanding square term 28

29. Differentiating with respect to x (gradient) Expanding square term 29

30. Expanding square term Differentiating with respect to x (gradient) 30

31. Expanding square term Normal equation x = (A.transpose()*A).solve(A.transpose()*b); Differentiating with respect to x (gradient) 31

32. Expanding square term Normal equation x = (A.transpose()*A).solve(A.transpose()*b); Issue: matrix multiplication Differentiating with respect to x (gradient) 32

33. Differentiating with respect to x Expanding square term Normal equation x = (A.transpose()*A).solve(A.transpose()*b); Issue: matrix multiplication Solution: find expression 33

34. Let's go back to gradient = 0 34

35. Let's go back to gradient = 0 Differentiating again with respect to x 35

36. Let's go back to gradient = 0 Differentiating again with respect to x 36

37. Let's go back to gradient = 0 Differentiating again with respect to x hessian 37

38. Let's go back to gradient = 0 Differentiating again with respect to x hessian 38

39. Let's go back to gradient = 0 Differentiating again with respect to x x = (A.transpose()*A).solve(A.transpose()*b); Normal equation: 39

40. Let's go back to gradient = 0 Differentiating again with respect to x x = (A.transpose()*A).solve(A.transpose()*b); Normal equation: 40

41. Let's go back to gradient = 0 Differentiating again with respect to x x = (A.transpose()*A).solve(A.transpose()*b); x = (H).solve(c); Two alternatives for solving: Constant terms in gradient 41

42. Example: Diffusion 42

43. Example: Diffusion Minimize difference between neighbors … 43

44. Example: Diffusion Minimize difference between neighbors … 44

45. Example: Diffusion Minimize difference between neighbors … 45

46. Example: Diffusion Minimize difference between neighbors … 46

47. Solving with normal equation: … 1.Build A 47

48. Solving with normal equation: … 1.Build A 48

49. Solving with normal equation: … 1.Build A 49

50. Solving with normal equation: … 1.Build A 50

51. Solving with normal equation: … 1.Build A 51

52. Solving with normal equation: … 1.Build A 52

53. Solving with normal equation: 2. Compute and 53

54. Solving with normal equation: 3.Solve using: Cholesky decomposition Conjugate Gradient … x = (A.transpose()*A).solve(A.transpose()*b); in Eigen (i.e): 54

55. Solving with normal equation: 3.Solve using: Cholesky decomposition Conjugate Gradient … x = (A.transpose()*A).solve(A.transpose()*b); in Eigen (i.e): x wil have: 55

56. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 56

57. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 57

58. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 58

59. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 59

60. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 60

61. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 61

62. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 62

63. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 63

64. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 64

65. Solving with Hessian: 65

66. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 66

67. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 67

68. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 68

69. Solving with Hessian: 1. Gradient vector: first-order partial derivatives of energy terms 69

70. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 70

71. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 71

72. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 72

73. Solving with Hessian: 2. Hessian matrix: second-order partial derivatives of energy terms 73

74. Solving with Hessian: 74

75. Solving with Hessian: To recap: 75

76. Solving with Hessian: 3. Build system … 76

77. Solving with Hessian: 3. Build system … 77

78. Solving with Hessian: 3. Build system: add constraint … 78

79. Solving with Hessian: … 3. Build system: add constraint 79

80. Solving with Hessian: … 3. Build system: add constraint 80

81. Solving with Hessian: … 3. Build system: add constraint 81

82. Solving with Hessian: … 3. Build system: add constraint 82

83. Solving with Hessian: … 3. Build system: add constraint 83

84. Solving with Hessian: … 3. Build system: add constraint 84

85. Solving with Hessian: … 3. Build system: add constraint 85

86. Solving with Hessian: … 3. Build system: add constraint 86

87. Solving with Hessian: 4. Solve using: Cholesky decomposition Conjugate Gradient … 87

88. Solving with Hessian: x = (H).solve(b`); in Eigen (i.e): x will have: 4. Solve using: Cholesky decomposition Conjugate Gradient … 88

89. References: 89 Solving Least Squares Problems. C. L. Lawson and R. J. Hanson. 1974. Practical Least Squares for Computer Graphics. Fred Pighin and J.P. Lewis. Course notes 2007.