Dimension Reduction and Feature Selection Craig A. Struble, Ph.D. Department of Mathematics, Statistics, and Computer Science Marquette University

MSCS 282: Data Mining - Craig A. Struble2 Overview  Dimension Reduction Correlation Principal Component Analysis Singular Value Decomposition  Feature Selection Information Content …

MSCS 282: Data Mining - Craig A. Struble3 Dimension Reduction  The number of attributes causes complexity of learning, clustering, etc. to grow exponentially “Curse of dimensionality”  We need methods to reduce the number of attributes  Dimension reduction reduces attributes without (directly) considering relevance of the attribute. Not really removing attributes, but combining/recasting them.

MSCS 282: Data Mining - Craig A. Struble4 Correlation  A causal, complementary, parallel, or reciprocal relationship  The simultaneous change in value of two numerically valued random variables  So, if one attribute’s value changes in a predictable way whenever another one changes, why keep them both?

MSCS 282: Data Mining - Craig A. Struble5 Correlation Analysis  Pearson’s Correlation Coefficient  Positive means both increase simultaneously  Negative means one increases as other decreases  If r A,B has a large magnitude, A and B are strongly correlated and one of the attributes can be removed

MSCS 282: Data Mining - Craig A. Struble6 Correlation Analysis X (Years Experience)Y (Salary in $1000s) Strong relationship

MSCS 282: Data Mining - Craig A. Struble7 Principal Component Analysis  Karhunen-Loeve or K-L method  Combine “essence” of attributes to create a (hopefully) smaller set of variables the describe the data  An instance with k attributes is a point in k- dimensional space  Find c k-dimensional orthogonal vectors that best represent the data such that c <= k  These vectors are combinations of attributes.

MSCS 282: Data Mining - Craig A. Struble8 Principal Component Analysis  Normalize the data  Compute c orthonormal vectors, which are the principal components  Sort in order of decreasing “significance” Measured in terms of data variance  Can reduce data dimension by choosing only the most significant principal components

MSCS 282: Data Mining - Craig A. Struble9 Singular Value Decomposition  One method of PCA  Let A be an m by n matrix. Then A can be written as the product of matrices such that U is an m by n matrix, V is an n by n matrix, and  is an n by n diagonal matrix with singular values  1 >=  2 >=…>=  n >=0. Furthermore, U and V are orthogonal matrices

MSCS 282: Data Mining - Craig A. Struble10 Singular Value Decomposition

MSCS 282: Data Mining - Craig A. Struble11 Singular Value Decomposition > x <- t(array(1:12,dim=c(3,4))) > str(s <- svd(x)) $u [,1] [,2] [,3] [1,] [2,] [3,] [4,] $v [,1] [,2] [,3] [1,] [2,] [3,] > a <- diag(s$d) [,1] [,2] [,3] [1,] e+00 [2,] e+00 [3,] e-16

MSCS 282: Data Mining - Craig A. Struble12 Singular Value Decomposition  The amount of variance captured by a singular value is  The entropy of the data set is

MSCS 282: Data Mining - Craig A. Struble13 Feature Selection  Select the most “relevant” subset of attributes  Wrapper approach Features are selected as part of the mining algorithm  Filter approach Features selected before mining algorithm  Wrapper approach is generally more accurate but also more computationally expensive

MSCS 282: Data Mining - Craig A. Struble14 Feature Selection  Feature selection is actually a search problem Want to select subset of features giving most accurate model a,b,c a,b a,c b,c a bc 

MSCS 282: Data Mining - Craig A. Struble15 Feature Selection  Any search heuristics will work Branch and bound “Best-first” or A* Genetic algorithms etc.  Bigger problem is to estimate the relevance of attributes without building classifier.

MSCS 282: Data Mining - Craig A. Struble16 Feature Selection  Using entropy Calculate information gain of each attribute Select the l attributes with the highest information gain Removes attributes that are the same for all data instances

MSCS 282: Data Mining - Craig A. Struble17 Feature Selection  Stepwise forward selection Start with empty attribute set Add “best” of attributes Add “best” of remaining attributes Repeat. Take the top l  Stepwise backward selection Start with entire attribute set Remove “worst” of attributes Repeat until l are left.

MSCS 282: Data Mining - Craig A. Struble18 Feature Selection  Other methods Sample data, build model for subset of data and attributes to estimate accuracy. Select attributes with most or least variance Select attributes most highly correlated with goal attribute.  What does feature selection provide you? Reduced data size Analysis of “most important” pieces of information to collect.