Hybrid Ant Colony Optimization-Support Vector Machine using Weighted Ranking for Feature Selection and Classification

Dimension of dataset high Decreased Predictive performance More Computation Time F1 F2 F3 F4 F6 F5 F7 DATASET WITH ALL FEATURES DATASET WITH ONLY SELECTED FEATURES NOISY FEATURES Dimension of dataset low Increased Predictive performance Lesser Computation Time

Evaluate features using heuristic information Fast but inaccurate E.g. Statistical correlation Feature Selection Algorithms FiltersWrappers Evaluate features using learning algorithm Slow but accurate E.g. Ant Colony Optimization, Genetic Algorithm

 Uses a hybrid filter-wrapper based approach.  Wrapper method: Ant Colony Optimization  Filter method : Weighted Ranking of features 7/10/20164

Ant Colony Optimization (ACO) Food Source 4 PheromoneOptimal Path  ACO is an iterative process used for solving combinatorial optimization problems  Inspired by the foraging behavior of real ants and their inherent ability to find the shortest path from a food source to their nest.

 Features equivalent to cities in TSP.  Ants conduct a partial tour in contrast to TSP.  ACO depends on: Heuristic Information on a feature Experience gained by ants in the previous iterations.

Filter approach. Each feature was assigned a Weighted Ranking score. Final Rank Score(WR) Information Gain Score (IG) Chi- Square Score (CS) CFS score (CFS) Weight w1 Weight w2 Weight w3 Weighted Rank (WR) of feature f is calculated as: WR f = w1 * IG f + w2 * CS f + w3 * CFS f

Proposed Algorithm 1 Rank the features based on the weighted summation of their IG, CS and CFS scores. 2 Initialize Pheromone values on all the links τo Initialize number of ants Initialize other algorithm parameters 3 An ant selects the first feature depending upon a scoring system. Scoring System employed for 90% of the cases. For the rest 10%, feature selected randomly

Proposed Algorithm 4 To select the next feature choose between exploitation and exploration 5 Conduct the partial tour in the same manner until maximum number of a priori fixed subset size is reached Depute other ants to conduct their own partial tours. 6 After completion of the tours of all ants, evaluate the quality of the tours employing SVM

7 Increase the pheromone concentration of links of the best ant proportional to the quality of the tour (accuracy of the classifier). 8 Decrease the pheromone concentrations on all the links visited by the remaining ants. 9 Repeat steps 3 to 8 for a certain number of iterations. Proposed Algorithm

 A scoring system has been developed for the selection of the first feature, which depends on:  Total no. of links connecting a feature having a pheromone value better than a threshold.  The sum of all the pheromone values on links starting from that feature.  The fitness value of that particular feature.

 The subset is decreased progressively from 90% of the original feature set size to 10%.  The subset size which gives the maximum accuracy is selected.

 In order to test the proposed algorithm, an SVM learning algorithm was used.  After a solution is constructed, each ant passes its feature subset to the classifier and receives its accuracy.  This accuracy is used as a fitness function for selecting the best ant of that iteration. The classifier accuracy was evaluated using 10- fold cross validation.

 Any good classifier can be employed as a wrapper.  Any combination of good filter-methods can be used with a weighted scoring system of features.

 Time Consuming.  Solution: Parallel implementation of the algorithm

ParametersValues Number of ants100 Number of iterations50 Exploitation Probability Factor (q0) 0.7 Pheromone Update Strength ( φ) 0.25 Pheromone Decay Parameter ( ρ ) 0.98 Pheromone Importance Factor ( β ) 1 Information Gain weight0.3 Chi-Square weight0.3 CFS weight0.3

 In order to test the proposed algorithm, six datasets were obtained from the UCI (University of California, Irvine) machine learning repository. Dataset NameNo. of featuresNo. of classesClass label type No. of instances Wisconsin Breast Cancer 102REAL683 Hepatitis192REAL155 Lung Cancer563REAL32 Splice602REAL1000 Bupa Liver Disorder 62REAL345 Statlog Heart132REAL270

 The classifier accuracy was calculated for all the datasets with and without using Feature Selection and the results were compared. Dataset NameNo. of original features No. of Features in the subset % Accuracy without FS % Accuracy with FS Wisconsin Breast Cancer Hepatitis Lung Cancer Splice Bupa Liver Disorder Statlog Heart

 The selected subset of features of all the datasets Dataset NameSelected Features Wisconsin Breast Cancer2,3,4,7,9 Hepatitis12, 13, 14 Lung Cancer9, 14 Splice15, 16, 18, 19, 22, 23, 25, 26, 28, 29, 30, 31, 32, 33, 34, 35 Bupa Liver Disorder1,2 Statlog Heart3,7,9,12,13

 Malignant Tumor that develops from Breast cells  Most Common cause of death among women  To reduce deaths, early diagnosis is important  Early diagnosis is important to distinguish between benign and malignant tumors

 Biopsy- expensive and invasive.  Several machine learning techniques successfully used to predict Breast Cancer.  Objective- To assign patients to benign or malignant class

 Dataset obtained from UCI Machine Learning Repository.  Consists of 10 features, 699 instances.  Features describe characteristics of the cell nuclei present in the breast mass image.  An RBF kernel was found to outperform the polynomial and linear kernels for this task, with the value of C=100.  A feature subset of size 5 features obtained an accuracy of %.

#FEATURESDOMAIN 1.Sample code numberId Number 2.Clump Thickness Uniformity of Cell Size Uniformity of Cell Shape Marginal Adhesion Single Epithelial Cell Size Bare Nuclei Bland Chromatin Normal Nucleoli Mitoses Class2 for benign, 4 for malignant

AlgorithmAccuracy (%) NB+CHI NB+GAIN NB+RelieF NB+CFS RBF+GA KNN+CHIWSS NN+CHIWSS74.23 SVM+CHWSS76.29 ACO-SVM+WR

 Hybrid filter-wrapper technique.  Reduces complexity of Machine learning algorithms.  Easy implementation.  Increased Predictive Performance  Picks up the correlations between features.  Flexible