Data mining and statistical learning, lecture 2 Outline  An example of data mining  SAS Enterprise miner

Data mining and statistical learning, lecture 2 Daily electricity consumption in Sweden

Data mining and statistical learning, lecture 2 ln daily electricity consumption in Sweden

Data mining and statistical learning, lecture 2 Available data  Daily levels of the total electricity consumption in Sweden  Daily levels of temperature, wind speed, and precipitation at a large number of weather stations in Sweden  Population in all municipalities in Sweden  Calendar data (Julian day, weekdays, holidays)

Data mining and statistical learning, lecture 2 Selecting, exploring, and modifying data Too much weather data!  We assigned a weather station to each municipality, and computed population-weighted mean values for the temperature, wind speed and precipitation in the whole of Sweden  Then we examined the relationship between the electricity consumption and the population-weighted weather data

Data mining and statistical learning, lecture 2 ln daily electricity consumption vs population- weighted mean temperature in Sweden

Data mining and statistical learning, lecture 2 Cubic spline with one knot (at x=1) Between knots, the spline function is identical to a third order polynomial At knots the function and its first two derivatives are continuous

Data mining and statistical learning, lecture 2 Some examples of additive models A nonlinear, additive model A mixed linear and nonlinear, additive model

Data mining and statistical learning, lecture 2 Modelling ln daily electricity consumption as a spline function of the population-weighted mean temperature in Sweden proc gam data=mining.electricity; model lnConsumption = spline(Mean_temp, df=20); ID Time(day); output out=smhiouttemp pred resid; run;

Data mining and statistical learning, lecture 2 Modelling ln daily electricity consumption as a spline function of the population-weighted mean temperature in Sweden: residual analysis

Data mining and statistical learning, lecture 2 Modelling ln daily electricity consumption in Sweden - residual analysis Spline of temperature Spline of Julian day Weekday dummies

Data mining and statistical learning, lecture 2 Modelling ln daily electricity consumption in Sweden - residual analysis Spline of temperature Spline of Julian day Weekday dummies Splines of contemporaneous and time-lagged weather data Splines of Julian day and time Weekday and holiday dummies

Data mining and statistical learning, lecture 2 Deviance analysis of the investigated models of ln daily electricity consumption in Sweden The residual deviance of a fitted model is minus twice its log-likelihood If the error terms are normally distributed, the deviance is equal to the sum of squared residuals

Data mining and statistical learning, lecture 2 Modelling ln daily electricity consumption in Sweden: time series plot of residuals

Data mining and statistical learning, lecture 2 Model selection in data-rich environments Divide the given data sets into two parts Use the training set to fit all potential models Use the test set to validate the tested models TrainingTest

Data mining and statistical learning, lecture 2 Model selection and unbiased estimation of the predictive power of the selected model Divide the given data sets into three parts Use the training set to fit all potential models Use the validation set to select a model Use the test set to compute an unbiased estimate of the predictive power of the selected model TrainingValidationTest

Data mining and statistical learning, lecture 2 SAS Enterprise Miner A toolbox for the five elements of data mining offering:  Convenient handling of large and complex datasets  Convenient comparison and assessment of many models  Widely used procedures for prediction, classification and association analysis

Data mining and statistical learning, lecture 2 SAS Enterprise Miner Run the miner  Import data  Create a project  Create a dataflow diagram  Edit the nodes of the diagram  Run a diagram  Assess the results Write and run SAS code