Data Mining: Concepts and Techniques — Chapter 3 — Cont. More on Feature Selection: Chi-squared test Principal Component Analysis

Attribute Selection Question: Are attributes A1 and A2 independent? ID Outlook Temperature Humidity Windy Play 1 100 40 90 T 2 F 3 50 4 10 30 5 15 70 6 7 8 9 11 12 13 14 Question: Are attributes A1 and A2 independent? If they are very dependent, we can remove either A1 or A2 If A1 is independent on a class attribute A2, we can remove A1 from our training data

Deciding to remove attributes in feature selection Dependent (ChiSq=small) ? A2=class attribute Independent (Chisq=large ? A1 Dependent (ChiSq=small) ? A2 = class attribute Independent (Chisq=large ?

Chi-Squared Test (cont.) Question: Are attributes A1 and A2 independent? These features are nominal valued (discrete) Null Hypothesis: we expect independence Outlook Temperature Sunny High Cloudy Low

The Weather example: Observed Count temperature Outlook High Low Outlook Subtotal Sunny 2 Cloudy 1 Temperature Subtotal: Total count in table =3 Outlook Temperature Sunny High Cloudy Low

The Weather example: Expected Count If attributes were independent, then the subtotals would be Like this (this table is also known as temperature Outlook High Low Subtotal Sunny 2*2/3=4/3=1.3 2*1/3=2/3=0.6 2 Cloudy 2*1/3=0.6 1*1/3=0.3 1 Subtotal: Total count in table =3 Outlook Temperature Sunny High Cloudy Low

Question: How different between observed and expected? If Chi-squared value is very large, then A1 and A2 are not independent  that is, they are dependent! Degrees of freedom: if table has n*m items, then freedom = (n-1)*(m-1) In our example Degree = 1 Chi-Squared=?

Chi-Squared Table: what does it mean? If calculated value is much greater than in the table, then you have reason to reject the independence assumption When your calculated chi-square value is greater than the chi2 value shown in the 0.05 column (3.84) of this table  you are 95% certain that attributes are actually dependent! i.e. there is only a 5% probability that your calculated X2 value would occur by chance

Example Revisited (http://helios.bto.ed.ac.uk/bto/statistics/tress9.html) We don’t have to have two-dimensional count table (also known as contingency table) Suppose that the ratio of male to female students in the Science Faculty is exactly 1:1, But, the Honours class over the past ten years there have been 80 females and 40 males. Question: Is this a significant departure from the (1:1) expectation? Observed Honours Male Female Total 40 80 120

Expected (http://helios.bto.ed.ac.uk/bto/statistics/tress9.html) Suppose that the ratio of male to female students in the Science Faculty is exactly 1:1, but in the Honours class over the past ten years there have been 80 females and 40 males. Question: Is this a significant departure from the (1:1) expectation? Note: the expected is filled in, from 1:1 expectation, instead of calculated Expected Honours Male Female Total 60 120

Chi-Squared Calculation Female Male Total Observed numbers (O) 80 40 120 Expected numbers (E) 60 O - E 20 -20 (O-E)2 400 (O-E)2 / E 6.67 Sum=13.34 = X2

Chi-Squared Test (Cont.) Then, check the chi-squared table for significance http://helios.bto.ed.ac.uk/bto/statistics/table2.html#Chi%20squared%20test Compare our X2 value with a c2 (chi squared) value in a table of c2 with n-1 degrees of freedom n is the number of categories, i.e. 2 in our case -- males and females). We have only one degree of freedom (n-1). From the c2 table, we find a "critical value of 3.84 for p = 0.05. 13.34 > 3.84, and the expectation (that the Male:Female in honours major are 1:1) is wrong!

Chi-Squared Test in Weka: weather.nominal.arff

Chi-Squared Test in Weka

Chi-Squared Test in Weka

Example of Decision Tree Induction Initial attribute set: {A1, A2, A3, A4, A5, A6} A4 ? A1? A6? Class 2 Class 1 Class 2 Class 1 Reduced attribute set: {A1, A4, A6} >

Principal Component Analysis Given N data vectors from k-dimensions, find c <= k orthogonal vectors that can be best used to represent data The original data set is reduced to one consisting of N data vectors on c principal components (reduced dimensions) Each data vector Xj is a linear combination of the c principal component vectors Y1, Y2, … Yc Xj= m+W1*Y1+W2*Y2+…+Wk*Yc, i=1, 2, … N M is the mean of the data set W1, W2, … are the ith components Y1, Y2, … are the ith Eigen vectors Works for numeric data only Used when the number of dimensions is large

Principal Component Analysis See online tutorials such as http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf X2 Y1 Y2 x Note: Y1 is the first eigen vector, Y2 is the second. Y2 ignorable. X1 Key observation: variance = largest!

Principal Component Analysis: one attribute first Temperature 42 40 24 30 15 18 35 Question: how much spread is in the data along the axis? (distance to the mean) Variance=Standard deviation^2

Now consider two dimensions X=Temperature Y=Humidity 40 90 30 15 70 Covariance: measures the correlation between X and Y cov(X,Y)=0: independent Cov(X,Y)>0: move same dir Cov(X,Y)<0: move oppo dir

More than two attributes: covariance matrix Contains covariance values between all possible dimensions (=attributes): Example for three attributes (x,y,z):

Background: eigenvalues AND eigenvectors Eigenvectors e : C e = e How to calculate e and : Calculate det(C-I), yields a polynomial (degree n) Determine roots to det(C-I)=0, roots are eigenvalues  Check out any math book such as Elementary Linear Algebra by Howard Anton, Publisher John,Wiley & Sons Or any math packages such as MATLAB

Steps of PCA Let be the mean vector (taking the mean of all rows) Adjust the original data by the mean X’ = X – Compute the covariance matrix C of adjusted X Find the eigenvectors and eigenvalues of C. For matrix C, vectors e (=column vector) having same direction as Ce : eigenvectors of C is e such that Ce=e,  is called an eigenvalue of C. Ce=e  (C-I)e=0 Most data mining packages do this for you.

Steps of PCA (cont.) Calculate eigenvalues  and eigenvectors e for covariance matrix: Eigenvalues j corresponds to variance on each component j Thus, sort by j Take the first n eigenvectors ei; where n is the number of top eigenvalues These are the directions with the largest variances

An Example Mean1=24.1 Mean2=53.8 X1 X2 X1' X2' 19 63 -5.1 9.25 39 74 14.9 20.25 30 87 5.9 33.25 23 -30.75 15 35 -9.1 -18.75 43 -10.75 32 -21.75 73 19.25

Covariance Matrix C= Using MATLAB, we find out: Eigenvectors: 75 106 482 C= Using MATLAB, we find out: Eigenvectors: e1=(-0.98,-0.21), 1=51.8 e2=(0.21,-0.98), 2=560.2 Thus the second eigenvector is more important!

If we only keep one dimension: e2 yi -10.14 -16.72 -31.35 31.374 16.464 8.624 19.404 -17.63 We keep the dimension of e2=(0.21,-0.98) We can obtain the final data as

Using Matlab to figure it out

PCA in Weka

Wesather Data from UCI Dataset (comes with weka package)

PCA in Weka (I)

Summary of PCA PCA is used for reducing the number of numerical attributes The key is in data transformation Adjust data by mean Find eigenvectors for covariance matrix Transform data Note: only linear combination of data (weighted sum of original data)

Missing and Inconsistent values Linear regression: Data are modeled to fit a straight line least-square method to fit Y=a+bX Multiple regression: Y = b0 + b1 X1 + b2 X2. Many nonlinear functions can be transformed into the above.

Regression Height Y1 Y1’ y = x + 1 X1 Age

Clustering for Outlier detection Outliers can be incorrect data. Clusters  majority behavior

Data Reduction with Sampling Allow a mining algorithm to run in complexity that is potentially sub-linear to the size of the data Choose a representative subset of the data Simple random sampling may have very poor performance in the presence of skew (uneven) classes Develop adaptive sampling methods Stratified sampling: Approximate the percentage of each class (or subpopulation of interest) in the overall database Used in conjunction with skewed data

Sampling SRSWOR (simple random sample without replacement) SRSWR Raw Data SRSWOR (simple random sample without replacement) SRSWR

Sampling Example Cluster/Stratified Sample Raw Data

Summary Data preparation is a big issue for data mining Data preparation includes Data warehousing Data reduction and feature selection Discretization Missing values Incorrect values Sampling