1. Data Analysis Learning from Data
Traditional methodology is statistics
What can we learn from analysis of historical data? Building up models.
Relationship to Optimization
Objective – find the “best” functional relationship that underlies the data.
Constraints – underlying function must have some limits (can always simply match the data but that is not meaningful)
Traditional Examples
Curve fitting – linear regression minimizes the square error given a polynomial function of the input variables
Classification –minimize number of classification errors given the a decision rule constrained to a polynomial or simple linear threshold
Model identification – determine relevant parameters in a given dynamic model to minimize error
Modern Examples – emphasize underlying process poorly understood
 Data Mining

2. Exploratory Analysis Simple Univariate Measures
Measures of central tendency
Measures of variation
Measures of similarity
 Have to start with Statistics

3. Exploratory Analysis Simple Multivariate Measures
Mean and variance
Correlation
Independence r = 0

4. Exploratory Analysis ANOVA - Analysis of Variance Hypothesis test
Essentially determining which input variables are significant
Hypothesis test
p-value is the probability of obtaining a finding a result similar to the one obtained, assuming the null hypothesis is true, i.e., the finding was the result of chance alone (p typically 5%)
Based on -square distribution
Many involved statistical tests
e.g., Kruskal-Wallis test for K independent samples

5. Simple Models from Data Regression
Maximum likelihood estimate (pseudo-inverse)
Function can be non-linear (still minimizing square error)

6. Simple Regression Example
Data points
(1,3); (8,9); (11,11); (4,5); (3,2)
Find two coefficients for straightline approximation
Several examples adapted from Data Mining by Kantardzic

7. Linear Regression Comments
Weighting can account for better quality data (usually by inverse of variance)
Every data point gets a vote
Sensitive to outliers – use least absolute value fit to minimize impact of outliers as we talked about last week
Maximum likelihood estimate only for normally distributed variables

8. Preprocessing of Data Bad data – detection of outliers
Least absolute value methods
Least square error
Bisquare
iterative method to
drop outliers
Notice curve fit
Figure from National Instruments

9. Preprocessing of Data Data reduction/transformation
Correlation coefficients with dependent variable
Essentially significance tests for different models
Principal component analysis
Transformation of variables based on variance
Figure from Mathworks

10. Preprocessing of Data Principal component analysis
Find a linear projection onto a unit vector u that has maximum variance
Assume a zero mean data set with covariance matrix C is represented as
Let and then the variance of the new data is
Form the Lagrangian to solve
Applying Kuhn-Tucker gives  as the eigenvalues of C

11. Preprocessing of Data Principal component analysis
Covariance matrix for independent variables
Eigenvectors from largest eigenvalues form the transformation
Can select “principal axes” by ratio test

12. Preprocessing of Data PCA Example
Covariance matrix
Ordered eigenvalues
R if selecting first two eigenvalues is 95%

13. Bayesian Analysis Incorporating Prior Information
Assume some information is already known, a conditional probability (H: Hypothesis X: Observation)
Example: Classification rule – given sample X what is the probability it belongs to Ci
Assuming independent attributes of X, say xt

14. Bayesian Analysis Example
Data
Prior probabilities
Conditional probabilities for Xi
Given Xnew= 1, 2, 2
conditional for X
find class, X probability from given samples
conclusion class C=2
Sample
Attribute X1 X2 X3
Class C 1 2 3 4 5 6 7

15. Decision Trees Classifying data
X1>0
X2>0
X1<2 C1 C2 C3
Yes
Series of classification/decision rules
Typical best rule – maximize the information gain (minimize the entropy)
Entropy
Information gain – weight Entropy by number (or probability) in each new class formed to compare decision over some attribute

16. Decision Tree Example Data Frequency of occurrence for probabilities
Initial entropy
Weighted entropy based on X1
Entropy based on X3
X1 is better choice
Attribute X1 X2 X3
Class C A 70
True 1 90 2 85
False 95 B 78 65 75 80 96

17. Some Motivation Increasingly large amounts of data gathered
Insufficient time to perform detailed analysis and develop precise models
Operation of the power system information driven rather than “signal” driven
Not possible to derive models from first principles (economics vs. physics)

18. Other Data Driven Methods
Linear methods tend to work well only for a narrow range of inputs
Discussed methods tend to work best under certain statistical properties
Need some robustness with respect to noisy inputs
Other approaches
Support Vector Machines – we’ve already done
Artificial Neural Networks – we’ll do the simplest version

19. What is Data Mining? A textbook definition My take
Data mining is the process of selection, exploration and modeling of large quantities of data to discover regularities or relations that are at first unknown with the aim of obtaining clear and useful results for the owner of the database. (Applied Data Mining by Paolo Giudici)
My take
Data mining concerns a wide variety of techniques useful for analyzing large data sets or for gathering information based primarily on data not on predefined models

20. Some Other Thoughts on Data Mining
Massive amounts of data that are not being analyzed that will increase with Smart Grid
Both operational and non-operational
Within a utility
Across different companies
Importance of communication systems
Decentralized
Robust but getting information to where needed

21. Problems with Data Mining
Nonlinear models are particularly susceptible to erroneous conclusions (overfitting)
Can always find relation in data if there is no underlying model (e.g., Super Bowl winner “predicts” stock market)
Preconceptions can always be reinforced if one searches long enough (i.e., most political discourse)
Increasing the amount of data (particularly unfiltered) increases the likelihood of spurious relationships (see the WWW)

22. Some Data-Driven Applications in Power Systems
Bayesian Analysis
Price Forecasting – determining Probability distribution
Reliability Analysis
Clustering
Price Modeling (Yesterday’s example)
Decision Trees
Security Analysis
Artificial Neural Nets – I’ll do something brief
Load Forecasting (huge number of papers)
Diagnostics