1 GAs and Feature Weighting Rebecca Fiebrink MUMT March 2005

2 of 25 Outline Genetic algorithms –History of GAs –How GAs work Simple model Variations Feature selection and weighting –Feature selection –Feature weighting Using GAs for feature weighting –Siedlecki and Sklansky –Punch –The ACE project

3 Part 1: Genetic Algorithms

4 of 25 History of GAs 1859: Charles Darwin, Origin of Species 1950’s – 60’s: Computers simulate evolution 1960’s – 70’s: John Holland invents genetic algorithms –Adaptation in Natural and Artificial Systems, book published 1975 –Context of “adaptive systems,” not just “optimization” 1980’s – Present: Further exploration, widespread adoptation –Kenneth De Jong –Goldberg: Genetic Algorithms in Search, Optimization, and Machine Learning (“classic” textbook published 1989) Related areas: evolutionary programming, genetic programming (Carnegie Mellon GA online FAQ, Cantu-Paz 2000)

5 of 25 How GAs work Problem is a search space –Variable parameters  dimensions –Problem has minima and maxima within this space –Minima and maxima are hard to find Potential solutions describe points in this space –It is “easy” to calculate “goodness” of any one solution and compare it to other solutions Maintain a set of potential solutions –Guess a set of initial solutions –Combine these solutions with each other –Keep “good” solutions and discard “bad” solutions –Repeat combination and selection process for some time

6 of 25 A simple GA Start Terminate? Stop Select Parents Produce Offspring Mutate Offspring Evaluate Population

7 of 25 GA terminology Population: The set of all current potential solutions Deme: A subset of the population that interbreeds (might be the whole population) Chromosome: A solution structure (often binary); contains one or more genes Gene: A feature or parameter Allele: A specific value for a gene Phenotype: A member of the population, a real-valued solution vector Generation: A complete cycle of evaluation, selection, crossover, and mutation in a deme Locus: A particular position (bit) on the chromosome string Crossover: The process of combining two chromosomes; simplest method is single-point crossover where two chromosomes swap parts on either side of a random locus Mutation: A random change to a phenotype (i.e., changing an allele)

8 of 25 Crossover Illustrated Parent 1 Child 1 Parent 2 Child 2

9 of 25 GA Variations Variables: –Population size –Selection method –Crossover method –Mutation rate –How to encode a solution in a chromosome No single choice is best for all problems (Wolpert & Macready 1997, cited in Cantu- Paz 2000).

10 of 25 More variations: Parallel GAs Categories: –Single-population Master/Slave –Multiple population –Fine-grained –Hybrids Master Slaves (Cantu-Paz 2000)

11 of 25 Applications of GAs NP problems (e.g., Traveling Salesman) Airfoil design Noise control Fluid dynamics Circuit partitioning Image processing Liquid crystals Water networks Music (Miettinen et al. 1999, Coley 1999)

12 Part 2: Feature Selection and Weighting

13 of 25 Features What are they? –A classifier’s “handle” to the problem –Numerical or binary values representing independent or dependent attributes of each instance to be classified What are they in music? –Spectral centroid –Number of instruments –Composer –Presence of Banjo –Beat histogram - …

14 of 25 Feature Selection – Why? “Curse of dimensionality”: –The size of the training set must grow exponentially with the dimensionality of the feature space The set of best features to use may vary according to classification scheme, goal, or data set We need a way to select only a good subset of all possible features –May or may not be interested in “best” subset

15 of 25 Feature Selection – How? Common approaches –Dimensionality reduction (principle component analysis or factor analysis) –Experimentation: Choose a subset, train a classifier with it, and evaluate its performance. Example: –A piece might have vector (values) –Try selections, … ? (1=yes, 0=no) –Which works best? Setbacks in experimentation: –For n potential features, there are 2 n possible subsets: a huge search space! – Evaluating each classifier takes time –Choose vectors using sequential search, branch, and bound, or GAs

16 of 25 Feature Weighting Among a group of selected (useful) features, some may be more useful than others Weight the features for optimal classifier performance Experimental approach: Similar to feature selection –Example: Say is optimal selection for feature set Try weights like,, … ? Practical and theoretical constraints of feature selection are magnified

17 Part 3: Using GAs for Feature Weighting

18 of 25 GAs and Feature Weighting A chromosome is a vector of weights –Each gene corresponds to a feature weight E.g., for selection E.g., for weighting The fitness of a chromosome is a measure of its performance training and validating an actual classifier

19 of 25 Siedlecki and Sklansky 1989: A note on genetic algorithms for large-scale feature selection Choose feature subset for Knn classifier Propose using GAs when feature set > 20 Binary chromosomes (0/1 for each feature) Goal: seek the smallest/least costly subset of features for which classifier performance above a certain level

20 of 25 Results Compared GAs with exhaustive search, branch and bound, and (p,q)-search on sample problem –Branch and bound and (p, q)-search did not come close enough (within 1% error) to finding a set that performed optimally –Exhaustive search used 2 24 evaluations –GA found optimal or near-optimal solutions with evaluations, a huge savings over both exhaustive search and branch and bound GA exhibits slightly more than linear increase of time complexity with added features GA also outperforms other methods on a real problem with 30 features

21 of 25 Punch et al. 1993: Further research on feature selection and classification using genetic algorithms Approach: Use GAs for feature weighting for Knn classifier: classifier “dimension warping” Each chromosome is a vector of real-valued weights –Mapped exponentially to range [.01, 10) or 0 Also experimented with “hidden features” representing combination (multiplication) of 2 other features

22 of 25 Results Feature weighting can outperform simple selection, especially on large data set with noise –Best performance when feature weighting follows binary selection 14 days to compute results Parallel version: nearly linear speedup when strings passed to other processors for fitness evaluation

23 of 25 Recent work Minaei-Bidgoli, Kortemeyer, and Punch, 2004: Optimizing classification ensembles via a genetic algorithm for a web-based educational system –Incorporate GA feature weighting in a multiple-classifier system (vote system) –Results show over 10% improvement in accuracy of the classifier ensemble when GA optimization is used

24 of 25 MIR Application ACE project –Accept arbitrary type and number of features, arbitrary taxonomy, training and testing data, and a target classification time –Classify data using best means available; e.g., make use of multiple classifiers, feature weighting –Parallel GAs for feature weighting can improve solution time and quality

25 of 25 Conclusions GAs are powerful and interesting tools, but they are complex and not always well-understood Feature selection/weighting involves selecting from a huge space of possibilities, but it is a necessary task GAs are useful for tackling the feature weighting problem Faster machines, parallel computing, and better understanding of GA behavior can all benefit the feature weighting problem This is relevant to MIR: we want to do good and efficient classification!