Regularized Least-Squares. Outline Why regularization? Truncated Singular Value Decomposition Damped least-squares Quadratic constraints.

Slides:

Advertisements

Similar presentations

Ordinary Least-Squares

Advertisements

Mark Trew CT Halfway Down Halfway down the stairs Is a stair Where I sit. There isn't any Other stair Quite like It. I'm not at the bottom, I'm.

3D Geometry for Computer Graphics

L1 sparse reconstruction of sharp point set surfaces

Least-squares Meshes Olga Sorkine and Daniel Cohen-Or Tel-Aviv University SMI 2004.

Robot Vision SS 2005 Matthias Rüther 1 ROBOT VISION Lesson 3: Projective Geometry Matthias Rüther Slides courtesy of Marc Pollefeys Department of Computer.

3D Reconstruction – Factorization Method Seong-Wook Joo KG-VISA 3/10/2004.

MATH 685/ CSI 700/ OR 682 Lecture Notes

Slides by Olga Sorkine, Tel Aviv University. 2 The plan today Singular Value Decomposition  Basic intuition  Formal definition  Applications.

INFORMATIK Differential Coordinates for Interactive Mesh Editing Yaron Lipman Olga Sorkine Daniel Cohen-Or David Levin Tel-Aviv University Christian Rössl.

Principal Component Analysis

3D Geometry for Computer Graphics. 2 The plan today Least squares approach  General / Polynomial fitting  Linear systems of equations  Local polynomial.

Methods For Nonlinear Least-Square Problems

Radial Basis Functions Pedro Teodoro. 2 What For Radial Basis Functions (RBFs) allows for interpolate/approximate scattered data in nD.

Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.

Uncalibrated Geometry & Stratification Sastry and Yang

Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004 Presented by Pat Chan 23/11/2004.

Inverse Kinematics How do I put my hand here? IK: Choose these angles!

Previously Two view geometry: epipolar geometry Stereo vision: 3D reconstruction epipolar lines Baseline O O’ epipolar plane.

3D Geometry for Computer Graphics

Math for CSLecture 11 Mathematical Methods for Computer Science Lecture 1.

Ordinary least squares regression (OLS)

Linear and generalised linear models

Faces: Analysis and Synthesis Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski.

Computer Graphics Recitation The plan today Least squares approach  General / Polynomial fitting  Linear systems of equations  Local polynomial.

Regularized meshless method for solving the Cauchy problem Speaker: Kuo-Lun Wu Coworker : Kue-Hong Chen 、 Jeng-Tzong Chen and Jeng-Hong Kao 以正規化無網格法求解柯西問題.

Linear and generalised linear models Purpose of linear models Least-squares solution for linear models Analysis of diagnostics Exponential family and generalised.

Desingularized meshless method for solving the Cauchy problem Speaker: Kuo-Lun Wu Coworker : Kue-Hong Chen 、 Jeng-Tzong Chen and Jeng-Hong Kao 以去奇異無網格法求解柯西問題.

Normal Estimation in Point Clouds 2D/3D Shape Manipulation, 3D Printing March 13, 2013 Slides from Olga Sorkine.

SVD(Singular Value Decomposition) and Its Applications

Motion Blending (Multidimensional Interpolation) Jehee Lee.

CHAPTER FIVE Orthogonality Why orthogonal? Least square problem Accuracy of Numerical computation.

Computer Graphics Group Tobias Weyand Mesh-Based Inverse Kinematics Sumner et al 2005 presented by Tobias Weyand.

A D V A N C E D C O M P U T E R G R A P H I C S CMSC 635 January 15, 2013 Spline curves 1/23 Curves and Surfaces.

CPSC 491 Xin Liu Nov 17, Introduction Xin Liu PhD student of Dr. Rokne Contact Slides downloadable at pages.cpsc.ucalgary.ca/~liuxin.

AN ORTHOGONAL PROJECTION

Progress in identification of damping: Energy-based method with incomplete and noisy data Marco Prandina University of Liverpool.

Systems of Equations and Inequalities Systems of Linear Equations: Substitution and Elimination Matrices Determinants Systems of Non-linear Equations Systems.

Computer Animation Rick Parent Computer Animation Algorithms and Techniques Optimization & Constraints Add mention of global techiques Add mention of calculus.

CSCE 441: Computer Graphics Forward/Inverse kinematics Jinxiang Chai.

Inverting the Jacobian and Manipulability

Review: Differential Kinematics

Parameter estimation. 2D homography Given a set of (x i,x i ’), compute H (x i ’=Hx i ) 3D to 2D camera projection Given a set of (X i,x i ), compute.

MATH 685/ CSI 700/ OR 682 Lecture Notes Lecture 4. Least squares.

INFORMATIK A Multi-scale Approach to 3D Scattered Data Interpolation with Compactly Supported Basis Functions Yutaka Ohtake Yutaka Ohtake Alexander Belyaev.

Surface Reconstruction using Radial Basis Functions Michael Kunerth, Philipp Omenitsch and Georg Sperl 1 Institute of Computer Graphics and Algorithms.

using Radial Basis Function Interpolation

University of Washington v The Hebrew University * Microsoft Research Synthesizing Realistic Facial Expressions from Photographs Frederic Pighin Jamie.

Fall 1999 Copyright © R. H. Taylor Given a linear systemAx -b = e, Linear Least Squares (sometimes written Ax  b) We want to minimize the sum.

1. Systems of Linear Equations and Matrices (8 Lectures) 1.1 Introduction to Systems of Linear Equations 1.2 Gaussian Elimination 1.3 Matrices and Matrix.

2.5 – Determinants and Multiplicative Inverses of Matrices.

ECE 530 – Analysis Techniques for Large-Scale Electrical Systems Prof. Hao Zhu Dept. of Electrical and Computer Engineering University of Illinois at Urbana-Champaign.

Parameter estimation class 5 Multiple View Geometry CPSC 689 Slides modified from Marc Pollefeys’ Comp

Problems in solving generic AX = B Case 1: There are errors in data such that data cannot be fit perfectly (analog: simple case of fitting a line while.

Chapter 61 Chapter 7 Review of Matrix Methods Including: Eigen Vectors, Eigen Values, Principle Components, Singular Value Decomposition.

CS246 Linear Algebra Review. A Brief Review of Linear Algebra Vector and a list of numbers Addition Scalar multiplication Dot product Dot product as a.

Lecture 16: Image alignment

Review of Linear Algebra

Homogeneous Coordinates (Projective Space)

Parameter estimation class 5

Epipolar geometry.

Estimating 2-view relationships

Statistical Learning Dong Liu Dept. EEIS, USTC.

CSCE 441: Computer Graphics Forward/Inverse kinematics

SVD: Physical Interpretation and Applications

Derivative of scalar forms

Mesh Parameterization: Theory and Practice

Projective Transformations for Image Transition Animations

Skeletal Motion, Inverse Kinematics

Presentation transcript:

Regularized Least-Squares

Outline Why regularization? Truncated Singular Value Decomposition Damped least-squares Quadratic constraints

Regularized Least-Squares Why regularization? We have seen that

Regularized Least-Squares Why regularization? We have seen that But what happens if the system is almost dependent? –The solution becomes very sensitive to the data –Poor conditioning

Regularized Least-Squares The 1-dimensional case The 1-dimensional normal equation

Regularized Least-Squares The 1-dimensional case The 1-dimensional normal equation

Regularized Least-Squares The 1-dimensional case The 1-dimensional normal equation

Regularized Least-Squares Why regularization Contradiction between data and model

Regularized Least-Squares A more interesting example: scattered data interpolation

Regularized Least-Squares “True” curve

Regularized Least-Squares Radial basis functions

Regularized Least-Squares Radial basis functions

Regularized Least-Squares Rbf are popular Modeling –J. C. Carr, R. K. Beatson, J. B. Cherrie, T. J. Mitchell,W. R. Fright, B. C. McCallum, and T. R. Evans. Reconstruction and representation of 3d objects with radial basis functions. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, pages 67–76, August –G. Turk and J. F. O’Brien. Modelling with implicit surfaces that interpolate. ACM Transactions on Graphics, 21(4):855–873, October Animation –J. Noh and U. Neumann. Expression cloning. In Proceedings of ACMSIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, pages 277–288, August –F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, pages 75–84, July 1998.

Regularized Least-Squares Radial basis functions At every point

Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem

Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem

Regularized Least-Squares Rbf results

Regularized Least-Squares p i 0 close to p i 1

Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem

Regularized Least-Squares Radial basis functions At every point Solve the least-squares problem If p i 0 close to p i 1, A is near singular

Regularized Least-Squares p i 0 close to p i 1

Regularized Least-Squares p i 0 close to p i 1

Regularized Least-Squares Rbf results with noise

Regularized Least-Squares Rbf results with noise

Regularized Least-Squares The Singular Value Decomposition Every matrix A ( nxm ) can be decomposed into: –where U is an nxn orthogonal matrix V is an mxm orthogonal matrix D is an nxm diagonal matrix

Regularized Least-Squares The Singular Value Decomposition Every matrix A ( nxm ) can be decomposed into: –where U is an nxn orthogonal matrix V is an mxm orthogonal matrix D is an nxm diagonal matrix

Regularized Least-Squares Geometric interpretation

Regularized Least-Squares Solving with the SVD

Regularized Least-Squares Solving with the SVD

Regularized Least-Squares Solving with the SVD

Regularized Least-Squares Solving with the SVD

Regularized Least-Squares Solving with the SVD

Regularized Least-Squares A is nearly rank defficient

Regularized Least-Squares A is nearly rank defficient

Regularized Least-Squares A is nearly rank defficient

Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0

Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0 =>some of the are close to

Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0 =>some of the are close to Problem with

Regularized Least-Squares A is nearly rank defficient =>some of the are close to 0 =>some of the are close to Problem with Truncate the SVD

Regularized Least-Squares p i 0 close to p i 1

Regularized Least-Squares Rbf fit with truncated SVD

Regularized Least-Squares Rbf results with noise

Regularized Least-Squares Rbf fit with truncated SVD

Regularized Least-Squares Choosing cutoff value k The first k such as

Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning

Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning ?

Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning

Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

Regularized Least-Squares Example: inverse skinning “Skinning Mesh Animations”, James and Twigg, siggraph Skinning Inverse skinning –Let be a set of pairs of geometry and skeleton configurations

Regularized Least-Squares “Skinning Mesh Animations”, James and Twigg, siggraph

Regularized Least-Squares Problem with the TSVD We have to compute the SVD of A, and O() process: impractical for large marices Little control over regularization

Regularized Least-Squares Damped least-squares Replace by where is a scalar and L is a matrix

Regularized Least-Squares Damped least-squares Replace by where is a scalar and L is a matrix The solution verifies

Regularized Least-Squares Examples of L DiagonalDifferential Limit scaleEnforce smoothness

Regularized Least-Squares Rbf results with noise

Regularized Least-Squares

Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph Reconstruct a mesh given –Control points –Connectivity (planar mesh)

Regularized Least-Squares Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph Reconstruct a mesh given –Control points –Connectivity (planar mesh) Smooth reconstruction

Regularized Least-Squares Example: “Least-Squares Meshes”, Sorkin and Cohen-Or, siggaph Reconstruct a mesh given –Control points –Connectivity (planar mesh) Smooth reconstruction In matrix form

Regularized Least-Squares Reconstruction Minimize reconstruction error where

Regularized Least-Squares “Least-Squares Meshes”, Sorkine and Cohen-Or, siggraph

Regularized Least-Squares Quadratic constraints Solve or

Regularized Least-Squares Quadratic constraints Solve or

Regularized Least-Squares Example

Regularized Least-Squares Example

Regularized Least-Squares Example

Regularized Least-Squares Discussion If, there is no solution (since there is no x for which )

Regularized Least-Squares Discussion If, there is no solution (since there is no x for which ) If, the solution exists and is unique

Regularized Least-Squares Discussion If, there is no solution (since there is no x for which ) If, the solution exists and is unique –Either the solution of is in the feasible set

Regularized Least-Squares Discussion If, there is no solution (since there is no x for which ) If, the solution exists and is unique –Either the solution of is in the feasible set –or the solution is at the boundary Solve

Regularized Least-Squares Discussion Solve where is a Lagrange multiplier

Regularized Least-Squares Conclusion TSVD really useful if you need an SVD

Regularized Least-Squares Conclusion TSVD really useful if you need an SVD Regularization constrains the solution: –Value, differential operator, other properties –Soft (damping) or hard constraint (quadratic) –Linear or non-linear

Regularized Least-Squares Conclusion TSVD really useful if you need an SVD Regularization constrains the solution: –Value, differential operator, other properties –Soft (damping) or hard constraint (quadratic) –Linear or non-linear Danger of over-damping or constraining

Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions

Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions ?

Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions

Regularized Least-Squares Example: inverse kinematic Problem: solve for joint angles given end-effector positions