1. Regularized Least-Squares

2. Outline Why regularization? Truncated Singular Value Decomposition Damped least-squares Quadratic constraints

3. Regularized Least-Squares Why regularization? We have seen that

4. Regularized Least-Squares Why regularization? We have seen that But what happens if the system is almost dependent? –The solution becomes very sensitive to the data –Poor conditioning

5. Regularized Least-Squares The 1-dimensional case The 1-dimensional normal equation

6. Regularized Least-Squares The 1-dimensional case The 1-dimensional normal equation

7. Regularized Least-Squares The 1-dimensional case The 1-dimensional normal equation

8. Regularized Least-Squares Why regularization Contradiction between data and model

9. Regularized Least-Squares A more interesting example: scattered data interpolation

10. Regularized Least-Squares “True” curve

11. Regularized Least-Squares Radial basis functions

12. Regularized Least-Squares Radial basis functions

13. Regularized Least-Squares Rbf are popular Modeling –J. C. Carr, R. K. Beatson, J. B. Cherrie, T. J. Mitchell,W. R. Fright, B. C. McCallum, and T. R. Evans. Reconstruction and representation of 3d objects with radial basis functions. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, pages 67–76, August 2001. –G. Turk and J. F. O’Brien. Modelling with implicit surfaces that interpolate. ACM Transactions on Graphics, 21(4):855–873, October 2002. Animation –J. Noh and U. Neumann. Expression cloning. In Proceedings of ACMSIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, pages 277–288, August 2001. –F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, pages 75–84, July 1998.

14. Regularized Least-Squares Radial basis functions At every point

15. Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem

16. Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem

17. Regularized Least-Squares Rbf results

18. Regularized Least-Squares p i 0 close to p i 1

19. Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem

20. Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem If p i 0 close to p i 1, A is near singular

21. Regularized Least-Squares p i 0 close to p i 1

22. Regularized Least-Squares p i 0 close to p i 1

23. Regularized Least-Squares Rbf results with noise

24. Regularized Least-Squares Rbf results with noise

25. Regularized Least-Squares The Singular Value Decomposition Every matrix A ( nxm ) can be decomposed into: –where U is an nxn orthogonal matrix V is an mxm orthogonal matrix D is an nxm diagonal matrix

26. Regularized Least-Squares The Singular Value Decomposition Every matrix A ( nxm ) can be decomposed into: –where U is an nxn orthogonal matrix V is an mxm orthogonal matrix D is an nxm diagonal matrix

27. Regularized Least-Squares Geometric interpretation

28. Regularized Least-Squares Solving with the SVD

29. Regularized Least-Squares Solving with the SVD

30. Regularized Least-Squares Solving with the SVD

31. Regularized Least-Squares Solving with the SVD

32. Regularized Least-Squares Solving with the SVD

33. Regularized Least-Squares A is nearly rank defficient

34. Regularized Least-Squares A is nearly rank defficient

35. Regularized Least-Squares A is nearly rank defficient

36. Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0

37. Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0 =>some of the are close to

38. Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0 =>some of the are close to Problem with

39. Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0 =>some of the are close to Problem with Truncate the SVD

40. Regularized Least-Squares p i 0 close to p i 1

41. Regularized Least-Squares Rbf fit with truncated SVD

42. Regularized Least-Squares Rbf results with noise

43. Regularized Least-Squares Rbf fit with truncated SVD

44. Regularized Least-Squares Choosing cutoff value k The first k such as

45. Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning

46. Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning ?

47. Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning

48. Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

49. Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

50. Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

51. Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

52. Regularized Least-Squares “Skinning Mesh Animations”, James and Twigg, siggraph

53. Regularized Least-Squares Problem with the TSVD We have to compute the SVD of A, and O() process: impractical for large marices Little control over regularization

54. Regularized Least-Squares Damped least-squares Replace by where is a scalar and L is a matrix

55. Regularized Least-Squares Damped least-squares Replace by where is a scalar and L is a matrix The solution verifies

56. Regularized Least-Squares Examples of L DiagonalDifferential Limit scaleEnforce smoothness

57. Regularized Least-Squares Rbf results with noise

58. Regularized Least-Squares

61. Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph Reconstruct a mesh given –Control points –Connectivity (planar mesh)

62. Regularized Least-Squares Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph Reconstruct a mesh given –Control points –Connectivity (planar mesh) Smooth reconstruction

63. Regularized Least-Squares Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph Reconstruct a mesh given –Control points –Connectivity (planar mesh) Smooth reconstruction In matrix form

64. Regularized Least-Squares Reconstruction Minimize reconstruction error where

65. Regularized Least-Squares “Least-Squares Meshes”, Sorkine and Cohen-Or, siggraph

66. Regularized Least-Squares Quadratic constraints Solve or

67. Regularized Least-Squares Quadratic constraints Solve or

68. Regularized Least-Squares Example

69. Regularized Least-Squares Example

70. Regularized Least-Squares Example

71. Regularized Least-Squares Discussion If, there is no solution (since there is no x for which )

72. Regularized Least-Squares Discussion If, there is no solution (since there is no x for which ) If, the solution exists and is unique

73. Regularized Least-Squares Discussion If, there is no solution (since there is no x for which ) If, the solution exists and is unique –Either the solution of is in the feasible set

74. Regularized Least-Squares Discussion If, there is no solution (since there is no x for which ) If, the solution exists and is unique –Either the solution of is in the feasible set –or the solution is at the boundary Solve

75. Regularized Least-Squares Discussion Solve where is a Lagrange multiplier

76. Regularized Least-Squares Conclusion TSVD really useful if you need an SVD

77. Regularized Least-Squares Conclusion TSVD really useful if you need an SVD Regularization constrains the solution: –Value, differential operator, other properties –Soft (damping) or hard constraint (quadratic) –Linear or non-linear

78. Regularized Least-Squares Conclusion TSVD really useful if you need an SVD Regularization constrains the solution: –Value, differential operator, other properties –Soft (damping) or hard constraint (quadratic) –Linear or non-linear Danger of over-damping or constraining

79. Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions

80. Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions ?

81. Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions

82. Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions