1. Dimension Reduction and Feature Selection Craig A. Struble, Ph.D. Department of Mathematics, Statistics, and Computer Science Marquette University

2. MSCS 282: Data Mining - Craig A. Struble2 Overview  Dimension Reduction Correlation Principal Component Analysis Singular Value Decomposition  Feature Selection Information Content …

3. MSCS 282: Data Mining - Craig A. Struble3 Dimension Reduction  The number of attributes causes complexity of learning, clustering, etc. to grow exponentially “Curse of dimensionality”  We need methods to reduce the number of attributes  Dimension reduction reduces attributes without (directly) considering relevance of the attribute. Not really removing attributes, but combining/recasting them.

4. MSCS 282: Data Mining - Craig A. Struble4 Correlation  A causal, complementary, parallel, or reciprocal relationship  The simultaneous change in value of two numerically valued random variables  So, if one attribute’s value changes in a predictable way whenever another one changes, why keep them both?

5. MSCS 282: Data Mining - Craig A. Struble5 Correlation Analysis  Pearson’s Correlation Coefficient  Positive means both increase simultaneously  Negative means one increases as other decreases  If r A,B has a large magnitude, A and B are strongly correlated and one of the attributes can be removed

6. MSCS 282: Data Mining - Craig A. Struble6 Correlation Analysis X (Years Experience)Y (Salary in $1000s) 330 857 964 1372 336 643 1159 2190 120 1683 Strong relationship

7. MSCS 282: Data Mining - Craig A. Struble7 Principal Component Analysis  Karhunen-Loeve or K-L method  Combine “essence” of attributes to create a (hopefully) smaller set of variables the describe the data  An instance with k attributes is a point in k- dimensional space  Find c k-dimensional orthogonal vectors that best represent the data such that c <= k  These vectors are combinations of attributes.

8. MSCS 282: Data Mining - Craig A. Struble8 Principal Component Analysis  Normalize the data  Compute c orthonormal vectors, which are the principal components  Sort in order of decreasing “significance” Measured in terms of data variance  Can reduce data dimension by choosing only the most significant principal components

9. MSCS 282: Data Mining - Craig A. Struble9 Singular Value Decomposition  One method of PCA  Let A be an m by n matrix. Then A can be written as the product of matrices such that U is an m by n matrix, V is an n by n matrix, and  is an n by n diagonal matrix with singular values  1 >=  2 >=…>=  n >=0. Furthermore, U and V are orthogonal matrices

10. MSCS 282: Data Mining - Craig A. Struble10 Singular Value Decomposition

11. MSCS 282: Data Mining - Craig A. Struble11 Singular Value Decomposition > x <- t(array(1:12,dim=c(3,4))) > str(s <- svd(x)) $u [,1] [,2] [,3] [1,] -0.1408767 -0.82471435 -0.3128363 [2,] -0.3439463 -0.42626394 0.7522216 [3,] -0.5470159 -0.02781353 -0.5659342 [4,] -0.7500855 0.37063688 0.1265489 $v [,1] [,2] [,3] [1,] -0.5045331 0.76077568 -0.4082483 [2,] -0.5745157 0.05714052 0.8164966 [3,] -0.6444983 -0.64649464 -0.4082483 > a <- diag(s$d) [,1] [,2] [,3] [1,] 25.46241 0.000000 0.000000e+00 [2,] 0.00000 1.290662 0.000000e+00 [3,] 0.00000 0.000000 8.920717e-16

12. MSCS 282: Data Mining - Craig A. Struble12 Singular Value Decomposition  The amount of variance captured by a singular value is  The entropy of the data set is

13. MSCS 282: Data Mining - Craig A. Struble13 Feature Selection  Select the most “relevant” subset of attributes  Wrapper approach Features are selected as part of the mining algorithm  Filter approach Features selected before mining algorithm  Wrapper approach is generally more accurate but also more computationally expensive

14. MSCS 282: Data Mining - Craig A. Struble14 Feature Selection  Feature selection is actually a search problem Want to select subset of features giving most accurate model a,b,c a,b a,c b,c a bc 

15. MSCS 282: Data Mining - Craig A. Struble15 Feature Selection  Any search heuristics will work Branch and bound “Best-first” or A* Genetic algorithms etc.  Bigger problem is to estimate the relevance of attributes without building classifier.

16. MSCS 282: Data Mining - Craig A. Struble16 Feature Selection  Using entropy Calculate information gain of each attribute Select the l attributes with the highest information gain Removes attributes that are the same for all data instances

17. MSCS 282: Data Mining - Craig A. Struble17 Feature Selection  Stepwise forward selection Start with empty attribute set Add “best” of attributes Add “best” of remaining attributes Repeat. Take the top l  Stepwise backward selection Start with entire attribute set Remove “worst” of attributes Repeat until l are left.

18. MSCS 282: Data Mining - Craig A. Struble18 Feature Selection  Other methods Sample data, build model for subset of data and attributes to estimate accuracy. Select attributes with most or least variance Select attributes most highly correlated with goal attribute.  What does feature selection provide you? Reduced data size Analysis of “most important” pieces of information to collect.