Basis Expansion and Regularization Prof. Liqing Zhang Dept. Computer Science & Engineering, Shanghai Jiaotong University

Basis Expansion and Regularization Outline Piece-wise Polynomials and Splines Wavelet Smoothing Smoothing Splines Automatic Selection of the Smoothing Parameters Nonparametric Logistic Regression Multidimensional Splines Regularization and Reproducing Kernel Hilbert Spaces 2017/4/15 Basis Expansion and Regularization

Piece-wise Polynomials and Splines Linear basis expansion Some basis functions that are widely used 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Three approaches for controlling the complexity of the model. Restriction Selection Regularization: 2017/4/15 Basis Expansion and Regularization

Piecewise Polynomials and Splines 2017/4/15 Basis Expansion and Regularization

Piecewise Cubic Polynomials Increasing orders of continuity at the knots. A cubic spline with knots at and : Cubic spline truncated power basis 2017/4/15 Basis Expansion and Regularization

Piecewise Cubic Polynomials An order-M spline with knots , j=1,…,K is a piecewise-polynomial of order M, and has continuous derivatives up to order M-2. A cubic spline has M=4. Truncated power basis set: 2017/4/15 Basis Expansion and Regularization

Natural cubic spline Natural cubic spline adds additional constraints, namely that the function is linear beyond the boundary knots. Linear Cubic Natural boundary constraints 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization B-spline The augmented knot sequenceτ: Bi,m(x), the i-th B-spline basis function of order m for the knot-sequenceτ, m≤M. 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization B-spline The sequence of B-spline up to order 4 with ten knots evenly spaced from 0 to 1 The B-spline have local support; they are nonzero on an interval spanned by M+1 knots. 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Smoothing Splines Base on the spline basis method: So , is the noise. Minimize the penalized residual sum of squares is a fixed smoothing parameter f can be any function that interpolates the data the simple least squares line fit 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Smoothing Splines The solution is a natural spline: Then the criterion reduces to: where So the solution: The fitted smoothing spline: 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Smoothing Splines Function of age, that response the relative change in bone mineral density measured at the spline in adolescents Separate smoothing splines fit the males and females, 12 degrees of freedom 脊骨BMD—骨质密度 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Smoothing Matrix the N-vector of fitted values The finite linear operator — the smoother matrix Compare with the linear operator in the LS-fitting: M cubic-spline basis functions, knot sequenceξ Similarities and differences: Both are symmetric, positive semidefinite matrices idempotent（幂等的) ; shrinking rank: 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Smoothing Matrix Effective degrees of freedom of a smoothing spline in the Reinsch form: Since , solution: is symmetric and has a real eigen-decomposition is the corresponding eigenvalue of K 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Smoothing spline fit of ozone(臭氧) concentration versus Daggot pressure gradient. Smoothing parameter df=5 and df=10. The 3rd to 6th eigenvectors of the spline smoothing matrices 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization The smoother matrix for a smoothing spline is nearly banded, indicating an equivalent kernel with local support. 2017/4/15 Basis Expansion and Regularization

Bias-Variance Tradeoff Example: For The diagonal contains the pointwise variances at the training Bias is given by is the (unknown) vector of evaluations of the true f 2017/4/15 Basis Expansion and Regularization

Bias-Variance Tradeoff df=5, bias high, standard error band narrow df=9, bias slight, variance not increased appreciably df=15, over learning, standard error widen 2017/4/15 Basis Expansion and Regularization

Bias-Variance Tradeoff The EPE and CV curves have the a similar shape. And, overall the CV curve is approximately unbiased as an estimate of the EPE curve The integrated squared prediction error (EPE) combines both bias and variance in a single summary: N fold (leave one) cross-validation: 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Logistic Regression Logistic regression with a single quantitative input X The penalized log-likelihood criterion 2017/4/15 Basis Expansion and Regularization

Multidimensional Splines Tensor product basis The M1×M2 dimensional tensor product basis , basis function for coordinate X1 , basis function for coordinate X2 2017/4/15 Basis Expansion and Regularization

Tenor product basis of B-splines, some selected pairs 2017/4/15 Basis Expansion and Regularization

Multidimensional Splines High dimension smoothing Splines J is an appropriate penalty function a smooth two-dimensional surface, a thin-plate spline. The solution has the form 2017/4/15 Basis Expansion and Regularization

Multidimensional Splines The decision boundary of an additive logistic regression model. Using natural splines in each of two coordinates. df = 1 +(4-1) + (4-1) = 7 2017/4/15 Basis Expansion and Regularization

Multidimensional Splines The results of using a tensor product of natural spline basis in each coordinate. df = 4 x 4 = 16 2017/4/15 Basis Expansion and Regularization

Multidimensional Splines A thin-plate spline fit to the heart disease data. The data points are indicated, as well as the lattice of points used as knots. 2017/4/15 Basis Expansion and Regularization

Reproducing Kernel Hilbert space A regularization problems has the form: L(y,f(x)) is a loss-function. J(f) is a penalty functional, and H is a space of functions on which J(f) is defined. The solution span the null space of the penalty functional J 2017/4/15 Basis Expansion and Regularization

Spaces of Functions Generated by Kernel Important subclass are generated by the positive kernel K(x,y). The corresponding space of functions Hk is called reproducing kernel Hilbert space. Suppose that K has an eigen-expansion Elements of H have an expansion 2017/4/15 Basis Expansion and Regularization

Spaces of Functions Generated by Kernel The regularization problem become The finite-dimension solution(Wahba,1990) Reproducing properties of kernel function 2017/4/15 Basis Expansion and Regularization

Spaces of Functions Generated by Kernel The penalty functional The regularization function reduces to a finite-dimensional criterion K is NxN matrix, the ij-th entry K(xi, xj) 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization RKHS Penalized least squares The solution of : The fitted values: The vector of N fitted value is given by 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Example of RKHS Polynomial regression Suppose M huge Given Loss function: The penalty polynomial regression: 2017/4/15 Basis Expansion and Regularization

Penalized Polynomial Regression Kernel: has eigen-functions E.g. d=2, p=2: The penalty polynomial regression: 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization RBF kernel & SVM kernel Gaussian Radial Basis Functions Support Vector Machines 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Wavelet smoothing Another type of bases——Wavelet bases Wavelet bases are generated by translations and dilations of a single scaling function . If ,then generates an orthonormal basis for functions with jumps at the integers. form a space called reference space V0 The dilations form an orthonormal basis for a space V1 V0 Generally, we have 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Wavelet smoothing The L2 space dividing Mother wavelet generate function form an orthonormal basis for W0. Likewise form a basis for Wj. 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization Wavelet smoothing Wavelet basis on W0 : Wavelet basis on Wj : The symmlet-p wavelet: A support of 2p-1 consecutive intervals. p vanishing moments: 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization 2017/4/15 Basis Expansion and Regularization

Adaptive Wavelet Filtering Wavelet transform: y: response vector, W: NxN orthonormal wavelet basis matrix Stein Unbiased Risk Estimation (SURE) The solution: Fitted function is given by inverse wavelet transform: , LS coefficients truncated to 0 Simple choice for 2017/4/15 Basis Expansion and Regularization

Basis Expansion and Regularization 2017/4/15 Basis Expansion and Regularization