Download presentation
Presentation is loading. Please wait.
Published bySuryadi Budiono Modified over 6 years ago
1
Feature Selection Ioannis Tsamardinos Machine Learning Course, 2006
Computer Science Dept. University of Crete (Some slides borrowed from Aliferis,Tsamardinos, 2004 Medinfo Tutorial)
2
Outline What is Variable/Feature Selection Filters & Wrappers
What is Relevancy Connecting Wrappers, Filters, and Relevancy SVM-Based Variable Selection Markov-Blanket-Based Variable Selection
3
Back to the Fundamentals: The feature selection problem
Journal of Machine Learning Research special issue: “Variable Selection refers to the problem of selecting input variables that are most predictive of a given outcome” Kohavi and John [1997]: “[variable selection is the problem of selecting]… the subset of features such that the accuracy of the induced classifier … is maximal” Problem: according to which classifier is predictive power measured by? A specific one? All possible classifiers? What about different cost features?
4
Why Feature Selection To reduce cost or risk associated with observing the variables To increase predictive power To reduce the size of the models, so they are easier to understand and trust To understand the domain
5
Definition of Feature Selection
Let M be a metric, scoring a model and a feature subset according to predictions and features used Let A be a learning algorithm used to build the model Feature Selection Problem: Select a feature subset s, that maximizes the score that M gives to the model learned by A using features s Feature Selection Problem 2: Select a feature subset s and a learner A’, that maximizes the score M gives to the model learned by A’ using features s
6
Examples M is the accuracy + a preference for smaller models, A is a SVM Find the minimal feature subset that maximizes the accuracy of a SVM Other possibilities for M: calibrated accuracy, AUC, trade-off between accuracy and cost of features
7
Filters and Wrappers
8
Wrappers An algorithm for solving the feature selection problem that is allowed to evaluate (has access to) learner A on different feature subsets Typical wrapper: Search (greedily or otherwise) the feature subset space Evaluate using M each subset s during search Report the one that maximizes M
9
Wrappers Say we have predictors A, B, C and classifier M. We want to predict T given the smallest possible subset of {A,B,C}, while achieving maximal performance (accuracy) FEATURE SET CLASSIFIER PERFORMANCE {A,B,C} M 98% {A,B} M 98% {A,C} M 77% {B,C} M 56% {A} M 89% {B} M 90% {C} M 91% {.} M 85%
10
An Example of a Greedy Wrapper
Since the search space is exponential, we have to use heuristic search Solution {A,B} 98 start {A} 89 {A,C} 77 {A,B}98 {A,B,C}98 {} 85 {B} 90 {B,C} 56 {A,C}77 {C} 91 end Subset returned by greedy search {B,C}56
11
Wrappers A common example of heuristic search is hill climbing: keep adding features one at a time until no further improvement can be achieved (“forward greedy wrapping”) Alternatively we can start with the full set of predictors and keep removing features one at a time until no further improvement can be achieved (“backward greedy wrapping”) A third alternative is to interleave the two phases (adding and removing) either in forward or backward wrapping (“forward-backward wrapping”). Of course other forms of search can be used; most notably: Exhaustive search Genetic Algorithms Branch-and-Bound (e.g., cost=# of features, goal is to reach performance th or better)
12
Example Feature Selection Methods in Bioinformatics: GA/KNN
Wrapper approach whereby: heuristic search=Genetic Algorithm, and classifier=KNN
13
Filters An algorithm for solving the feature selection problem that is not allowed to evaluate (does not have access to) learner A Typical filters select the feature subset according to certain statistical properties
14
Filter Example: Univariate Association Filtering
Rank features according to their association with the target (univariately) Select the first k features FEATURE ASSOCIATION WITH TARGET No Threshold gives optimal solution {C} % {B} % {A} %
15
Order all predictors according to strength of association with target
Example Feature Selection Methods in Biomedicine: Univariate Association Filtering Order all predictors according to strength of association with target Choose the first k predictors and feed them to the classifier Various measures of association may be used: X2, G2, Pearson r, Fisher Criterion Scoring, etc. How to choose k? What if we have too many variables?
16
Filter algorithm where feature selection is done as follows:
Example Feature Selection Methods in Biomedicine: Recursive Feature Elimination Filter algorithm where feature selection is done as follows: build linear Support Vector Machine classifiers using V features compute weights of all features and choose the best V/2 repeat until 1 feature is left choose the feature subset that gives the best performance (using cross-validation) give best feature set to the classifier of choice.
17
What is Relevancy
18
Relevant and Irrelevant Features
Large effort and debate to define relevant (irrelevant) features (AI Journal vol. 97) Why? Intuition: For classification (presumably) we only need relevant features We can throw away the irrelevant features The set of relevant features must be the solution to the feature selection problem! What is Relevant must be independent of the classifier A used to build the final model! Relevant Features teach us something about the domain
19
Relevancy and Filters Consider a definition of relevancy
Construct an algorithm that attempts to identify the relevant features It is a filtering algorithm (independent of the classifier used) Relevancy a family of filtering algorithms
20
The Argument of Kohavi and John [1997]
Take a handicapped perceptron: sgn(wx) instead of sgn(wx + w0) Add an irrelevant variable to the data with value always 1 For some problems, the irrelevant variables is necessary Filtering (presumably) returns only “relevant” features Thus, filtering is suboptimal, wrapping is not sgn(wx) 1 1 x4 x3 x2 x1 x0
21
KJ Definitions of Relevancy
KJ-Strongly Relevant Variable (for target T) X is KJ-Strongly Relevant if it is necessary for optimal density estimation V set of all variables, S=V \ {X, T} P(T | X, S) P(T | S)
22
KJ Definitions of Relevancy
KJ-Weakly Relevant Variable (for target T) X is Weakly Relevant if it is not necessary for optimal density estimation, but still informative (i.e., there is some subset that makes conditioned on which it becomes informative) V set of all variables X not strongly relevant and There exists U V \ {X, T} P(T | X, U) P(T | X)
23
KJ Definition of Irrelevancy
A variable X is KJ-irrelevant to T if it is not weakly or strongly relevant to T Intuitively X provides no information for T conditioned on any subset of other variables
24
Connecting Wrappers, Filters, and Relevancy
25
Negative Results on Relevancy and Filters
Kohavi and John argument: “filtering returns only relevant variables and sometimes KJ-irrelevant or KJ-weakly relevant variables maybe needed” True: There is no definition of relevancy independent of the classifier A used to build the final model, or independent of the metric M that evaluates the model of A, such that the relevant features are the solution to the feature selection problem [Tsamardinos, Aliferis, AI&Stats 2003] Have to assume a (family) of algorithm(s) and metric(s) to define what is relevant
26
Negative Results on Wrappers
Wrappers are subject to the No Free Lunch theorem for black-box optimization if the choice of metric or the classifier is unconstrained [Tsamardinos, Aliferis, AI&Stats 2003] => Averaged out on all possible problems, each wrapper is the same as the random search Requires an exponential search to provably find the optimal feature subset
27
Connecting with Bayesian Networks
Faithful Bayesian Network A Irrelevant Features KJ Strongly Relevant Features B C D KJ-irrelevant Features (anything without a path to T) K F T H E I Markov Blanket of T Weakly Relevant Features (anything with a path to T)
28
Markov Blanket in Faithful Bayesian Networks
Markov Blanket of T KJ-Strongly Relevant Features Smallest set of variables, conditioned on which all other variables become independent of T Set of parents, children, and spouses of T
29
OPTIMAL Solutions to a class of Feature Selection Problems
MB(T) is smallest subset of variables, conditioned on which all other variables become independent of T The Markov Blanket of T should be all we need True when: Classifier can utilize the information in those variables (e.g. is a universal approximator) The metric prefers the smallest models with optimal calibrated accuracy (otherwise the Markov Blanket may include unnecessary variables)
30
SVM Based Variable Selection
31
Linear SVMs Identify Irrelevant Features
Theorem: Both the hard and soft margin linear SVM will assign a weight of zero to irrelevant features (in the sample limit) Set up the sample limit SVM Prove there is a unique and w, b in the sample limit Prove in this w, b the weight of the irrelevant features is zero
32
Linear SVMs may not Identify KJ-Strongly Relevant Variables
Consider an exclusive OR The soft margin linear SVM has a zero weight vector But, both features are KJ-strongly relevant Similar result expected for non-linear SVMs x2 1 1 -1 -1 1 1
33
Linear SVMs may Retain KJ-Weakly-Relevant Features
X2 1 1 X1 1+
34
Feature Selection with (Linear) SVMs
The SVM will correct remove irrelevant features The SVM may incorrect also remove strongly relevant features The SVM may incorrectly retain weakly relevant features
35
Markov Blanket Based Feature Selection
36
Optimal Feature Selection with Markov Blankets
MMMB and Hiton algorithms [KDD 2003, AMIA 2003] Can identify the MB(T) among thousands of variables Provably correct in the sample limit and in faithful distributions Provably the MB(T) is the solution under the conditions specified Excellent results in real datasets from biomedicine The different between the two methods is conditioning vs maximizing the margin
37
Causal Discovery Recall: feature selection to understand the domain
Markov Blanket of T Causal interpretation: direct causes, direct effects, direct causes of direct effects … When Faithfulness Causal Sufficiency Acyclicity
38
(999 variables, consists of 37 tiles of Alarm network)
Network: Alarm-1k (999 variables, consists of 37 tiles of Alarm network) Classification Algorithm: RBF SVM Training sample size = 1000 Testing sample size = 1000
39
Target = 46 Target Member of MB
40
Feature Selection Method = HITON_MB
Classification performance = 87.7% 3 True positives 2 False positives 0 False negatives Target False negative True positive False positive
41
Feature Selection Method = MMMB
Classification performance = 87.7% 3 True positives 2 False positives 0 False negatives Target False negative True positive False positive
42
Feature Selection Method = RFE Linear
Classification performance = 74.6% 2 True positives 189 False positives 1 False negative Target False negative True positive False positive
43
Feature Selection Method = RFE Polynomial
Classification performance = 85.6% 2 True positives 33 False positives 1 False negative Target False negative True positive False positive
44
Feature Selection Method = BFW
Classification performance = 82.5% 3 True positives 161 False positives 0 False negatives Target False negative True positive False positive
45
Classification Algorithm: See 5.0 Decision Trees
Network: Gene (801 variables) Classification Algorithm: See 5.0 Decision Trees Training sample size = 1000 Testing sample size = 1000
46
Target = 220 Target Member of MB
47
Feature Selection Method = MMMB
Classification performance with DT = 96.4% 9 True positives 4 False positives 0 False negatives Target False negative True positive False positive
48
Feature Selection Method = HITON_MB
Classification performance with DT = 96.2% 9 True positives 4 False positives 0 False negatives Target False negative True positive False positive
49
Feature Selection Method = BFW
Classification performance with DT = 96.3% 4 True positives 20 False positives 5 False negatives Target False negative True positive False positive
50
Is This A General Phenomenon Or A Contrived Example?
51
Random Targets in Tiled ALARM
52
Random Targets in GENE
53
Conclusions A formal definition of the feature selection problems allows to draw connections between relevant/irrelevant variables, the Markov Blanket, and solutions to the feature selection problem Need to specify the algorithm and metric to design algorithms that provably solve the feature selection problem
54
Conclusions Linear SVMs (current formulations) correctly identify the irrelevant variables, but do not solve the feature selection problem (under the conditions specified) Markov Blanket based algorithms exists that are probably correct in the sample limit for faithful distributions Questions: SVM formulations that provably return the solution Extend the results to the non-linear case
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.