Support Vector Machine Applications Electrical Load Forecasting ICONS Presentation Spring 2007 N. Sapankevych 20 April 2007

N. Sapankevych - 20 April SVMs and Electrical Load Forecasting Research Status SVMs, Time Series Prediction, and Electrical Load Forecasting Relevance Vector Machines Way forward Q&A AGENDA

N. Sapankevych - 20 April SVM time series prediction survey complete – Major research focuses on financial market forecasting and electrical load prediction – Paper to be submitted to IEEE Neural Network Journal Research focus on Support Vector Machines (SVMs) and their applications for time series prediction – Expand to Relevance Vector Machines Thesis topic definition – Apply SVM time series prediction research Expand research to use relevance vector machines – Potential collaboration w/ Prof. Domijan and Prof. Islam of USF EE Department Electrical energy consumption prediction and resource management allocation wrt severe weather RESEARCH OVERVIEW SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Summary of SVM Time Series Prediction Survey SVM Time Series Prediction Survey Summary SVMs and Electrical Load Forecasting ApplicationNumber of Published Papers Summarized in this Survey Financial Market Prediction21 Electric Utility Forecasting17 Control Systems and Signal Processing 8 Miscellaneous Applications8 General Business Applications5 Environmental Parameter Estimation4 Machine Reliability Forecasting3 Comprehensive survey: – 66 specific application papers + 17 general references/books + 11 SVM specific websites

N. Sapankevych - 20 April Advantages and challenges of SVM Time Series Prediction SVM Time Series Prediction Survey Summary (con’t) SVMs and Electrical Load Forecasting Time Series Prediction Method AdvantagesChallenges Autoregressive Filter-Can be computationally efficient for low order models -Convergence guaranteed -Minimizes mean square error by design -Assumes linear, stationary processes -Can be computationally expensive for higher order models Kalman Filter-Computationally efficient by design -Convergence guaranteed -Minimizes mean square error by design -Assumes linear, stationary processes -Assumes process model is known Multi-layer Perceptron-Not model dependent -Not dependent on linear, stationary processes -Can be computationally efficient (feed forward process) -Number of free parameters large -Selection of free parameters usually calculated empirically -Not guaranteed to converge to optimal solution -Can be computationally expensive (training process) SVM/SVR-Not model dependent -Not dependent on linear, stationary processes -Guaranteed to converge to optimal solution -Small number of free paramters -Can be computationally efficient -Selection of free parameters usually calculated empirically -Can be computationally expensive (training process)

N. Sapankevych - 20 April Seventeen papers reviewed and documented for electrical load prediction using SVMs – Should note more than half of them use same general approach but different data set Majority of papers use same basic approach to predict (short term) electrical load demand – Some models use weather trends as inputs – Free parameters determined empirically A few papers propose use of other algorithms, such as genetic algorithms and simulated annealing, to assist in parameter selection – The accurate and optimal selection of the free parameters remains the challenge for SVM Time Series Prediction Note for the same kernel functions selected, different numerical values found empirically for different applications No set metrics for measures of effectiveness – RMSE, MSE, MAPE Prediction horizons short – Typically less than seven samples No error estimate in prediction (for all applications) SVM Time Series Prediction Electrical Load Forecasting Summary SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Developed by Tipping in 1999 – Michael E. Tipping, The Relevance Vector Machine, patent number (Microsoft Research) Work focused on addressing SVM disadvantages – No probabilistic output Uncertainty of prediction (regression) or classification (binary selection) not given – Although, distance from maximum hyperplane (as example) could infer accuracy? – In general, need to empirically determine errors relative to free parameters – Specific rules related to generating kernel functions (Mercer’s conditions) Relevance Vector Machines SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Formulation uses same form as SVM, but a Bayesian framework is adopted Goal, from a regression (and ultimately a time series prediction application) point of view, is to provide the predicted “state” as well as an estimate of the uncertainty Relevance Vector Machines (con’t) SVMs and Electrical Load Forecasting SVM Architecture There is an underlying assumption in this formulation is that Gaussian statistics are assumed for prediction – Can this assumption hold for highly non-linear, non-stationary systems?

N. Sapankevych - 20 April The formulation outline below is summarized from: M. E. Tipping, “The Relevance Vector Machine”, Advances in Neural Information Processing Systems 12, pages MIT Press, Recall formulation of regression function using SVM Goal of SVM to determine optimal weights w – Define input vectors x n and “targets” t n – The targets t are the training data (output) based on the input vector x Now assume Gaussian distribution for targets – P(t|x) is Gaussian: N(t|y(x),  2 ) Relevance Vector Machines (con’t) SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April The likelihood of the data set can be expressed as: – p(t|w,  2 )  maximize this likelihood By introducing hyperparameters  n (necessary to avoid over-fitting) and applying Bayes rule, the posterior probability is generated – p(w|t, ,  2 ) The integration of this equation (over the weights) produces the marginal likelihood (evidence) – p(t| ,  2 ) – Note that Tipping states that integrating out the remaining two priors can not be done in closed form, and proposes another procedure (see ref) Goal now is to optimize the hyperparameters and estimate the noise variance Relevance Vector Machines (con’t) SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Tipping showed results of adding Gaussian noise to sinc function (regression) Relevance Vector Machines (con’t) SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Tipping showed results of classification problem Relevance Vector Machines (con’t) SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Why use RVMs? – More sparse representation of the data – Can be more accurate – Kernel function selection not as stringent (Mercer’s Theorem – Kernel matrix positive semidefinite, etc.) – Uncertainty of estimate now provided Drawbacks – Training more complex with Bayesian framework Questions – Is Gaussian assumption on training data representative of applications where SVM is used? Relevance Vector Machines (con’t) SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Propose integrated predictive model, now based on RVM approach, to predict power loads and potential infrastructure failure probabilities (resource management) in inclement weather Way Forward: Electrical Load Forecasting SVMs and Electrical Load Forecasting NWS Products Trajectory Size Intensity Duration Accuracy Infrastructure Data Geospatial Data Population Density Data Physical Infrastructure Laydown Power Infrastructure MTBFs (as function of Wx and maintenance) Spatial-Temporal Power Loading Cost Data Personnel allocation Equipment maintenance and replacement kW-Hr rates Priorities (residential vs. commercial vs. “critical”) Integrated Predictive Model PREPROCESS Power Load Prediction Decision Aids to Resource Management Directors DRAFT ARCHITECTURE

N. Sapankevych - 20 April What needs to be done – Publish survey paper (final review w/ Prof. Sankar today – per request) – Work w/ Prof. Sankar, Prof. Domijan and Prof. Islam to: Clearly define thesis topic Define and obtain necessary data (at least on electrical load side of problem) – Develop RVM approach to problem Compare to SVM? Compare to other models (MLP, ARMA, other?) Implement simulation – Review thesis topic w/ committee members Get buy-in Way Forward: Electrical Load Forecasting SVMs and Electrical Load Forecasting

N. Sapankevych - 20 April Q&A SVMs and Electrical Load Forecasting Alexander – 19 months and getting bigger!