1 Robust Nonparametric Regression by Controlling Sparsity Gonzalo Mateos and Georgios B. Giannakis ECE Department, University of Minnesota Acknowledgments: NSF grants no. CCF , EECS , May 24, 2011

2 2 Nonparametric regression If one trusts data more than any parametric model  Then go nonparametric regression:  lives in a (possibly -dimensional) space of “ smooth ’’ functions Our focus  Nonparametric regression robust against outliers  Robustness by controlling sparsity Ill-posed problem  Workaround: regularization [Tikhonov ’ 77], [Wahba ’ 90]  RKHS with reproducing kernel and norm Given, function estimation allows predicting  Estimate unknown from a training data set

3 3 Our work in context Robust nonparametric regression  Huber ’ s function [Zhu et al ’ 08]  No systematic way to select thresholds Robustness and sparsity in linear (parametric) regression  Huber ’ s M-type estimator as Lasso [Fuchs ‘ 99]; contamination model  Bayesian framework [Jin-Rao ‘ 10][Mitra et al ’ 10]; rigid choice of Noteworthy applications  Load curve data cleansing [Chen et al ’ 10]  Spline-based PSD cartography [Bazerque et al ’ 09]

4 4 Variational LTS Least-trimmed squares (LTS) regression [Rousseeuw ’ 87] Variational (V)LTS counterpart  is the -th order statistic among Simple but intractable beyond small problems (VLTS)  residuals discarded Q: How should we go about minimizing ? (VLTS) is nonconvex; existence of minimizer(s)? A : Try all subsamples of size, solve, and pick the best

5 5 Modeling outliers  Nominal data obey ; outliers something else Remarks  Both and are unknown  If outliers sporadic, then vector is sparse! Natural (but intractable) nonconvex estimator Outlier variables s.t. outlier otherwise

6 6 VLTS as sparse regression Lagrangian form (P0) The equivalence  Formally justifies the regression model and its estimator (P0)  Ties sparse regression with robust estimation  Tuning parameter controls sparsity in number of outliers Proposition 1: If solves (P0) with chosen s.t., then solves (VLTS) too.

7 7 Just relax! (P1)  (P1) convex, and thus efficiently solved  Role of sparsity controlling is central Q: Does (P1) yield robust estimates ? A: Yap! Huber estimator is a special case where (P0) is NP-hard relax

8 8 Alternating minimization (P1) jointly convex in AM solver Remarks  Single Cholesky factorization of  Soft-thresholding  Reveals the intertwining between Outlier identification Function estimation with outlier compensated data (P1)

9 9 Lassoing outliers Enables effective methods to select  Lasso solvers return entire robustification path (RP) Cross-validation (CV) fails with multiple outliers [Hampel ’ 86] Proposition 2: as and, with Minimizers of (P1) are fully determined by w/ Alternative to AM solve Lasso [Tibshirani ’ 94]

10 Robustification paths Lasso path of solutions is piecewise linear  LARS returns whole RP [Efron ’ 03]  Same cost of a single LS fit ( ) Lasso is simple in the scalar case  Coordinate descent is fast! [Friedman ‘ 07]  Exploits warm starts, sparsity  Other solvers: SpaRSA [Wright et al ’ 09], SPAMS [Mairal et al ’ 10] Coeffs. Leverage these solvers consider 2-D grid  values of  For each, values of

11 Selecting and  Number of outliers known: from RP, obtain range of s.t.. Discard outliers (known), and use CV to determine Relies on RP and knowledge on the data model  Variance of the nominal noise known: from RP, for each on the grid, obtain an entry of the sample variance matrix as The best are s.t.  Variance of the nominal noise unknown: replace above with a robust estimate, e.g., median absolute deviation (MAD)

12 Nonconvex regularization Nonconvex penalty terms approximate better in (P0) Options: SCAD [Fan-Li ’ 01], or sum-of-logs [Candes et al ’ 08] Iterative linearization-minimization of around Remarks  Initialize with, use and  Bias reduction (cf. adaptive Lasso [Zou ’ 06])

13 Robust thin-plate splines Specialize to thin-plate splines [Duchon ’ 77], [Wahba ’ 80] Smoothing penalty only a seminorm in Still, Proposition 2 holds for appropriate Solution:  Radial basis function  Augment w/ member of the nullspace of  Given, unknowns found in closed form

14 Simulation setup Training set : noisy samples of Gaussian mixture examples, i.i.d. Outliers: i.i.d. for True function Data Nominal: w/ i.i.d. ( known)

15 Robustification paths Grid parameters:  grid: Outlier Inlier Paths obtained using SpaRSA [Wright et al ’ 09]

16 Results True function Nonrobust predictions Robust predictionsRefined predictions Effectiveness in rejecting outliers is apparent

17 Generalization capability In all cases, 100% outlier identification success rate Figures of merit  Training error:  Test error: Nonconvex refinement leads to consistently lower

18 Load curve data cleansing Load curve: electric power consumption recorded periodically Reliable data: key to realize smart grid vision Uruguay ’ s aggregate power consumption (MW) Deviation from nominal models (outliers) Faulty meters, communication errors Unscheduled maintenance, strikes, sporting events B-splines for load curve prediction and denoising [Chen et al ’ 10]

19 Real data tests Nonrobust predictions Robust predictionsRefined predictions

20 Concluding summary Robust nonparametric regression  VLTS as -(pseudo)norm regularized regression (NP-hard)  Convex relaxation variational M-type estimator Lasso Real data tests for load curve cleansing Controlling sparsity amounts to controlling number of outliers  Sparsity controlling role of is central  Selection of using the Lasso robustification paths  Different options dictated by available knowledge on the data model Refinement via nonconvex penalty terms  Bias reduction and improved generalization capability