1 Estimating Structured Vector Autoregressive Models Igor Melnyk and Arindam Banerjee

2 Consider Vector Autoregressive Model (VAR) where: - multivariate time series - model parameters, order of model - Gaussian noise:,, Widely used model Financial time series : [Tsay ’05] Dynamic systems: [Ljung ’98] Brain function connectivity [Valdes-Sosa et. al. ’05] Anomaly detection in aviation data Introduction

3 Objective: estimate Let be VAR output across steps Estimation Problem

4

5 Objective: estimate Regularized estimator Estimation Problem vectorize

6 - regularization norm, - regularization parameter Examples: - sparsity - group sparsity K-support norm Overlapping group sparsity Main properties: Samples are correlated - any norm, separable along rows of, arbitrary with row Regularized Estimator

7 Properties of VAR model Samples are correlated - separable along rows of, arbitrary within row Regularized Estimator

8 Constrained VAR (Dantzig selector form) [Han & Liu ’13] - based formulation under Gaussian noise [Qiu et. al. ’15] Extension of above to heavy-tailed noise Regularized VAR [Han & Liu. ’13] is and group ; assumptions on data dependency [Kock & Callot ’15] is ; exploited martingale property o data [Loh et. al. ’11] is ; considered only first-order VAR [Basu et. al. ’15] is ; any-order VAR; spectral analysis of data correlation Regularized linear regression [Meinshausen et. al. ’09, Wainwright ’09, Rudelson ’13] is [Negahban et. al. ’09] is any decomposable norm [Banerjee et. al. ’14] is any norm Our approach Use framework of [Banerjee et. al. ’14] and generic chaining of [Talagrand ’06] Use spectral analysis of [Basu et. al. ’15], exploit martingale property of data Related Work

9 Any row: Projected rows: for Data Matrix X

10 Any row of X: Matrix X: Any Row

11 To characterize, consider, for example Can be viewed as composed from consecutive outputs of VAR model Then eigenvalue of can be bounded using spectral density of VAR Matrix X: Any Row

12 Block-Toeplitz matrix Eigenvalues can be bounded is a spectral density Can be viewed as Fourier transform of Matrix X: Any Row

13 Therefore, each row of X: Using VAR spectral density Smallest eigenvalue of can be bounded as where Matrix X: Any Row

14 Now consider, for Since, it follows that where and Matrix X: Multiple Projected Rows

15 To characterize, observe that Can be viewed as composed from consecutive output of VAR model of order 1 Matrix X: Multiple Projected Rows

16 Eigenvalues of block-Toeplitz matrix can be bounded as is a spectral density of VAR model of order 1 Largest eigenvalue of can be bounded as where Matrix X: Multiple Projected Rows

17 Therefore, multiple projected rows of X: Eigenvalues of its covariance matrix Largest eigenvalue of where Matrix X: Multiple Projected Rows

18 Return back to our estimator Denote error between true and estimated parameter Our task Establish conditions on under which error is bounded Error Analysis Framework

19 Assume for Then the error belongs to a set Moreover, assume that for and Then the error is bounded (deterministically) as where Error Analysis Framework Bound on regularization parameter Restricted eigenvalue condition

20 Define a unit set Define Gaussian width of set for For, with probability where are absolute constants Result: Bound on Regularization Parameter

21 Let,, then Let, where - unit sphere Define Gaussian width for For with probability at least As a consequence, Result: Restricted Eigenvalue Condition

22 Select number of data samples such that Then choose regularization parameter such that Restricted eigenvalue condition will hold w. h. p., i.e., Norm of estimation error will be bounded w. h. p. by Lasso: and, therefore Group Lasso: and Interpretation of Results - sparsity - group sparsity - number of groups - size of largest group

23 Need to find such that holds w.h.p. Dual norm Establish bound Proof Strategy for Bound on

24 Proof Strategy for Bound on establish sub-exponential tails use generic chaining argument of Talagrand - summation over martingale difference sequence - majorizing measure inequality

25 Need to show holds w.h.p. for Separate problem into parts Establish bound of the form Proof Strategy for RE

26 Proof Strategy for RE establish concentration of Lipschitz function of Gaussian random vector use generic chaining argument of Talagrand

27 Investigate scaling of errors and lambda Lasso Experiments

28 Investigate scaling of errors and lambda Group Lasso Experiments