Sequential Genetic Search for Ensemble Feature Selection Alexey Tsymbal, Padraig Cunningham Department of Computer Science Trinity College Dublin Ireland Mykola Pechenizkiy Department of Computer Science University of Jyväskylä Finland IJCAI’2005, Edinburgh, Scotland August 1-5, 2005

2 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Contents  Introduction –Classification and Ensemble Classification  Ensemble Feature Selection –strategies –sequential genetic search  Our GAS-SEFS strategy –Genetic Algorithm-based Sequential Search for Ensemble Feature Selection  Experiment design  Experimental results  Conclusions and future work

3 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. CLASSIFICATION New instance to be classified Class Membership of the new instance J classes, n training observations, p features Given n training instances (x i, y i ) where x i are values of attributes and y is class Goal: given new x 0, predict class y 0 Training Set The Task of Classification Examples: - prognostics of recurrence of breast cancer; - diagnosis of thyroid diseases; - antibiotic resistance prediction

4 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Ensemble classification How to prepare inputs for generation of the base classifiers?

5 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Ensemble classification How to combine the predictions of the base classifiers?

6 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Ensemble feature selection  How to prepare inputs for generation of the base classifiers ? –Sampling the training set –Manipulation of input features –Manipulation of output targets (class values)  Goal of traditional feature selection –find and remove features that are unhelpful or misleading to learning (making one feature subset for single classifier)  Goal of ensemble feature selection –find and remove features that are unhelpful or destructive to learning making different feature subsets for a number of classifiers –find feature subsets that will promote diversity (disagreement) between classifiers

7 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Search in EFS Search space: 2 #Features * #Classifiers Search strategies include:  Ensemble Forward Sequential Selection (EFSS)  Ensemble Backward Sequential Selection (EBSS)  Hill-Climbing (HC)  Random Subspacing Method (RSM)  Genetic Ensemble Feature Selection (GEFS) Fitness function:

8 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Measuring Diversity The kappa statistic: The fail/non-fail disagreement measure : the percentage of test instances for which the classifiers make different predictions but for which one of them is correct:

9 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Random Subspace Method  RSM itself is simple but effective technique for EFS –the lack of accuracy in the ensemble members is compensated for by their diversity –does not suffer from the curse of dimensionality –RS is used as a base in other EFS strategies, including Genetic Ensemble Feature Selection.  Generation of initial feature subsets using (RSM)  A number of refining passes on each feature set while there is improvement in fitness

10 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Genetic Ensemble Feature Selection  Genetic search – important direction in FS research –GA as effective global optimization technique  GA for EFS: –Kuncheva, 1993: Ensemble accuracy instead of accuracies of base classifiers Fitness function is biased towards particular integration method Preventive measures to avoid overfitting –Alternative: use of individual accuracy and diversity Overfitting of individual is more desirable than overfitting of ensemble –Opitz, 1999: Explicitly used diversity in fitness function RSM for initial population New candidates by crossover and mutation Roulette-wheel selection (p proportional to fitness)

11 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Genetic Ensemble Feature Selection recombination mutation x f phenotype space population of genotypes (base classifiers) coding scheme fitness selection Current ensemble of base classifiers

12 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Basic Idea behind GA for EFS Ensemble (generation) BC 1 BC i BC Ens. Size RSM GA Current Population (diversity) New Population (fitness) init

13 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Basic Idea behind GAS-SEFS Ensemble BC 1 BC i BC i+1 GA i+1 BC i+1 diversity RSM Generation Current Population (accuracies) New Population (fitness) new BC (fitness) init

14 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. GAS-SEFS 1 of 2 GAS SEFS  GAS-SEFS (Genetic Algorithm-based Sequential Search for Ensemble Feature Selection) –instead of maintaining a set of feature subsets in each generation like in GA, consists in applying a series of genetic processes, one for each base classifier, sequentially. –After each genetic process one base classifier is selected into the ensemble. –GAS-SEFS uses the same fitness function, but diversity is calculated with the base classifiers already formed by previous genetic processes In the first GA process – accuracy only. –GAS-SEFS uses the same genetic operators as GA.

15 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. GAS-SEFS 2 of 2  GA and GAS-SEFS peculiarities: –Full feature sets are not allowed in RS –The crossover operator may not produce a full feature subset. –Individuals for crossover are selected randomly proportional to log(1+fitness) instead of just fitness –The generation of children identical to their parents is prohibited. –To provide a better diversity in the length of feature subsets, two different mutation operators are used Mutate1_0 deletes features randomly with a given probability; Mutate0_1 adds features randomly with a given probability.

16 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Complexity of GA-based search does not depend on the #features GAS-SEFS: GA: where S is the number of base classifiers, S’ is the number of individuals (feature subsets) in one generation, and N gen is the number of generations. EFSS and EBSS: where S is the number of base classifiers, N is the total number of features, and N’ is the number of features included or deleted on average in an FSS or BSS search. Computational complexity

17 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Integration of classifiers Motivation for the Dynamic Integration: Each classifier is best in some sub-areas of the whole data set, where its local error is comparatively less than the corresponding errors of the other classifiers. Static Dynamic Selection/Combination Dynamic Voting with Selection (DVS) Weighted Voting (WV) Dynamic Selection (DS) Static Selection (CVM)

18 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Experimental Design  Parameter settings for GA and GAS-SEFS: –a mutation rate - 50%; –a population size – 10; –a search length of 40 feature subsets/individuals: 20 are offsprings of the current population of 10 classifiers generated by crossover, 20 are mutated offsprings (10 with each mutation operator). –10 generations of individuals were produced; –400 (GA) and 4000 (GAS-SEFS) feature subsets.  To evaluate GA and GAS-SEFS: –5 integration methods –Simple Bayes as Base Classifier –stratified random-sampling with 60%/20%/20% of instances in the training/validation/test set; –70 test runs on each of 21 UCI data set for each strategy and diversity.

19 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. GA vs GAS-SEFS on two groups of datasets Ensemble accuracies for GA and GAS-SEFS on two groups of data sets (1): = 9 features with four ensemble sizes DVS F/N-F disagreement Ensemble Size

20 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. GA vs GAS-SEFS for Five Integration Methods Ensemble Size = 10 Ensemble accuracies for five integration methods on Tic-Tac-Toe

21 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Conclusions and Future Work  Diversity in ensemble of classifiers is very important  We have considered two genetic search strategies for EFS.  The new strategy, GAS-SEFS, consists in employing a series of genetic search processes –one for each base classifier.  GAS-SEFS results in better ensembles having greater accuracy –especially for data sets with relatively larger numbers of features. –one reason – each of the core GA processes leads to significant overfitting of a corresponding ensemble member  GAS-SEFS is significantly more time-consuming than GA. –GAS-SEFS = ensemble_size * GA  [Oliveira et al., 2003] better results for single FSS based on Pareto-front dominating solutions. –Adaptation of this technique to EFS is an interesting topic for further research.

22 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Thank you! Alexey Tsymbal, Padraig Cunningham Dept of Computer Science Trinity College Dublin Ireland Mykola Pechenizkiy Department of Computer Science and Information Systems University of Jyväskylä Finland

Additional Slides

24 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. References [Kuncheva, 1993] Ludmila I. Kuncheva. Genetic algorithm for feature selection for parallel classifiers, Information Processing Letters 46: , [Kuncheva and Jain, 2000] Ludmila I. Kuncheva and Lakhmi C. Jain. Designing classifier fusion systems by genetic algorithms, IEEE Transactions on Evolutionary Computation 4(4): , [Oliveira et al., 2003] Luiz S. Oliveira, Robert Sabourin, Flavio Bortolozzi, and Ching Y. Suen. A methodology for feature selection using multi-objective genetic algorithms for handwritten digit string recognition, Pattern Recognition and Artificial Intelligence 17(6): , [Opitz, 1999] David Opitz. Feature selection for ensembles. In Proceedings of the 16th National Conference on Artificial Intelligence, pages , 1999, AAAI Press.

25 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. GAS-SEFS Algorithm

26 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Other interesting findings alpha –were different for different data sets, –for both GA and GAS-SEFS, alpha for the dynamic integration methods is bigger than for the static ones (2.2 vs 0.8 on average). –GAS-SEFS needs slightly higher values of alpha than GA (1.8 vs 1.5 on average). GAS-SEFS always starts with a classifier, which is based on accuracy only, and the subsequent classifiers need more diversity than accuracy. # of selected features falls as the ensemble size grows, –this is especially clear for GAS-SEFS, as the base classifiers need more diversity. integration methods (for both GA and GAS-SEFS): –the static, SS and WV, and the dynamic DS start to overfit the validation set already after 5 generations and show lower accuracies, –accuracies of DV and DVS continue to grow up to 10 generations.

27 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Paper Summary New strategy for genetic ensemble feature selection, GAS- SEFS, is introduced In contrast with previously considered algorithm (GA), it is sequential; a serious of genetic processes for each base classifier More time-consuming, but with better accuracy Each base classifier has a considerable level of overfitting with GAS-SEFS, but the ensemble accuracy grows Experimental comparisons demonstrate clear superiority on 21 UCI datasets, especially for datasets with many features (gr1 vs gr2)

28 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Simple Bayes as Base Classifier Bayes theorem: P(C|X) = P(X|C)·P(C) / P(X) Naïve assumption: attribute independence P(x 1,…,x k |C) = P(x 1 |C)·…·P(x k |C) If i-th attribute is categorical: P(x i |C) is estimated as the relative freq of samples having value x i as i-th attribute in class C If i-th attribute is continuous: P(x i |C) is estimated thru a Gaussian density function Computationally easy in both cases

29 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. Dataset’s characteristics Data setInstancesClasses Features Categ.Num. Balance Breast Cancer Car Diabetes Glass Recognition Heart Disease Ionosphere Iris Plants LED LED Liver Disorders Lymphography MONK MONK MONK Soybean Thyroid Tic-Tac-Toe Vehicle Voting Zoo

30 IJCAI’2005 Edinburgh, Scotland, August 1-5, 2005 Sequential Genetic Search for Ensemble Feature Selection by Tsymbal A., Pechenizkiy M.,Cunningham P. GA vs GAS-SEFS for Five Integration Methods Ensemble accuracies for GA (left) and GAS-SEFS (right) for five integration methods and four ensemble sizes on Tic-Tac-Toe Ensemble Size