1. Time Series Prediction and Support Vector Machines ICONS Presentation Spring 2006 N. Sapankevych 16 April 2006

2. N. Sapankevych - 4/16/06 2 Time Series Prediction and SVMs Research Overview SVMs for Time Series Prediction Applications Q&A AGENDA

3. N. Sapankevych - 4/16/06 3 Time Series Prediction and SVMs Research focus on Support Vector Machines (SVMs) and their applications for time series prediction – Extension of previous research work (and qualifiers) w/ MLPs and SVMs as classifiers Goal to complete SVM time series prediction survey paper by end of Spring 2006 semester – Work in progress (over 50 papers down-selected from several hundred published in dozens of different journals) – Build on research work formulated in qualifying exams Further goal to define thesis topic – Extend SVM time series prediction research – Potential collaboration w/ UCF (more on that later) RESEARCH OVERVIEW

4. N. Sapankevych - 4/16/06 4 Time Series Prediction and SVMs Quick SVM history and background – Invented by Vapnik early 1990’s First instantiation actually demonstrated in COLT ‘92 – SVM another “class” of learning machine – Primary application for (mostly binary) classification problems – Also, SVMs used for regression applications (function estimation) – More recently, research on use in time series prediction applications (focus of research) Note here SVR (Support Vector Regression) plays significant part in this TIME SERIES PREDICTION USING SVMs

5. N. Sapankevych - 4/16/06 5 Time Series Prediction and SVMs Why use Artificial Neural Networks (ANNs) for time series prediction? – Traditional methods assume “model” or “data generating process” ARMA and its derivatives Kalman {other – IMM, MHT, Particle Filter, etc.} Stationary (and sometimes Gaussian) processes assumed (drawback in some cases) Assumed model itself may be incorrect in some cases – Neural Networks “let the data speak for itself” Model-free Can work in both linear or non-linear applications Performance tradeoff with computational complexity Found to perform well w/o all the data TIME SERIES PREDICTION USING SVMs (con’t)

6. N. Sapankevych - 4/16/06 6 Time Series Prediction and SVMs Many SVM advantages over other ANN-based learning machines such as MLPs (based on architecture) – Guaranteed unique solution to cost function (quadratic programming) Not necessarily so in MLPs – Map non-linear functions to linear space using Kernel functions Would need more layers/neurons as compared to MLPs – Fewer free parameters for optimization than MLPs Selection of Kernel function and other constants somewhat of an “art” for both linear and non-linear applications Selection of parameters may be just as hard, however TIME SERIES PREDICTION USING SVMs (con’t)

7. N. Sapankevych - 4/16/06 7 Time Series Prediction and SVMs SVM basics: – For supervised learning and for a regression application, define and minimize the “risk” functional – Define the “loss” function (quadratic in this case) – x is input vector (input) – y is output vector (response) –  is free variable – P(x,y) is NOT KNOWN – There are other loss functions  -insensitive loss function most popular (Vapnik) TIME SERIES PREDICTION USING SVMs (con’t)

8. N. Sapankevych - 4/16/06 8 Time Series Prediction and SVMs More on loss functions: –  -insensitive loss function most popular (Vapnik – linear model shown below) – Note zero term for values within  boundary – These loss functions have been shown (Vapnik, others) to be robust for function estimation TIME SERIES PREDICTION USING SVMs (con’t)

9. N. Sapankevych - 4/16/06 9 Time Series Prediction and SVMs Empirical Risk Minimization (ERM) vs. Structural Risk Minimization (SRM) – ERM means minimize the following by finding  : – SRM means minimize the following by adjusting  (C and as well): – Why the extra term? If you don’t have much data, ERM may be inadequate method by which to measure “goodness” of fit – add regularization term (also known as capacity control term) – What does the extra term do? “Flattens” the fit Controls Capacity TIME SERIES PREDICTION USING SVMs (con’t)

10. N. Sapankevych - 4/16/06 10 Time Series Prediction and SVMs More on SRM and ERM: – One fundamental difference between SVMs and MLPs is the regularization term – Regularization term assumes weights (w) are for linear regression function What if function to estimate is not linear? – Use Kernels: transform data to higher dimensional space TIME SERIES PREDICTION USING SVMs (con’t)

11. N. Sapankevych - 4/16/06 11 Time Series Prediction and SVMs SVM Architecture (from Scholkopf tutorial) TIME SERIES PREDICTION USING SVMs (con’t)

12. N. Sapankevych - 4/16/06 12 Time Series Prediction and SVMs How do you solve for weights? – Solution turns out to be quadratic programming problem – Use Lagrange multipliers to find optimal weights Non-zero multipliers associated w/ “Support Vectors” {details found in my Qualifier #2 presentation – classification example below} TIME SERIES PREDICTION USING SVMs (con’t)

13. N. Sapankevych - 4/16/06 13 Time Series Prediction and SVMs Much more to this effort (work in progress) and more questions to answer specifically for time series prediction: – How to train SVM for this application? SMO Gradient Ascent Chunking {other} Computational complexity and data quantity issue for training How often do you need to retrain and when? – Sparse data vs. performance Same “kind” of problem for Kalman Filter (time updates – covariance growth) – How to pick a Kernel function? – How to pick a regularization function? – How to pick regularization constants? – How to manage error? Stability? – Any “real time” applications? Computational complexity may be prohibitive TIME SERIES PREDICTION USING SVMs (con’t)

14. N. Sapankevych - 4/16/06 14 Time Series Prediction and SVMs Current research effort focused on exhaustive literature search for SVM Time Series Prediction and its applications – Note K.-R. Muller et. al. “Predicting Time Series with Support Vector Machines” and Vapnik’s “Statistical Learning Theory” key publications in this area – Other tutorials and resources available SVM research relatively new endeavor (about 10+ years) – Published articles growing in number and by application APPLICATIONS

15. N. Sapankevych - 4/16/06 15 Time Series Prediction and SVMs Largest quantity of published articles found to date is from financial market prediction applications – Journal publications indicate SVMs perform well given non-linear nature of stock price prediction – Other related financial applications such as credit rating forecasting Several other applications – Utility forecasting (power load and consumption) – Manufacturing industry (machinery MTBF and reliability forecasting) – Medical applications (drug effects prediction) – Environmental (rainfall, pollution) Notably missing (as I have found so far) – Signal Processing – Electromechanical Control Systems – Remote Sensing applications (radar tracking, etc.) – Maybe SVMs not well suited for these real time applications? APPLICATIONS (con’t)

16. N. Sapankevych - 4/16/06 16 Time Series Prediction and SVMs Potential collaborative research opportunity and application to my current research – Dr. W. Linwood Jones at UCF Central Florida Remote Sensing Laboratory (CFRSL) http://www.engr.ucf.edu/centers/cfrsl/Index.htm – CFRSL focus on satellite-based remote sensing applications SeaWinds microwave scatterometer measuring ocean surface wind vectors (QuickSCAT satellite) Several algorithm development efforts (among other things) – Rainfall, windspeed, surface temperature, and other derived remote sensing products – Work in progress to establish cooperative research effort and (possibly) generate thesis topic Possible research: multi-sensor applications using SVMs APPLICATIONS (con’t)

17. N. Sapankevych - 4/16/06 17 Time Series Prediction and SVMs Q&A Alexander – 7 months and getting bigger!