AISTATS 2010 Active Learning Challenge: A Fast Active Learning Algorithm Based on Parzen Window Classification L.Lan, H.Shi, Z.Wang, S.Vucetic Temple University

2 Introduction Pool-based Active Learning  Data labeling is expensive  Large amount of unlabeled data are available at low cost.  Goal is to label as few of the unlabeled examples and achieve as high accuracy as possible 2010 Active Learning Challenge  Provide an opportunity for practitioners to evaluate active learning algorithms within an unbiased setup  Data sets came from 6 various application domains Active Learning Algorithm Based on Parzen Window Classification

Challenge Data Sets Common Properties  Binary Classification  Class-Imbalanced Differences  Features  Concept 3 Data Set DomainFeat. Type Feat. Num. Sparsity % Missing % Label Train num. Train pos:neg Test num. AHandWritting Rec.mixed binary : BMarketingmixed binary : CChemo-informaticsmixed binary : DText classificationbinary binary : EEmbryologycontinuous binary : FEcologymixed1200binary : Active Learning Algorithm Based on Parzen Window Classification

4 Challenge Setup Given 1 positive seed example Repeat  Select which unlabeled examples to label  Train a classifier  Evaluate its accuracy (AUC: Area Under ROC Curve) Evaluate the active learning algorithm  using ALC (Area under the Learning Curve) Active Learning Algorithm Based on Parzen Window Classification

Algorithm Design Issues Querying Strategy  How many examples to label at each stage  What unlabeled examples to select Classification Algorithm  Simple vs. Powerful  Easy to implement vs. Involved Preprocessing and Feature Selection  Often the critical issue

6 Components of Our Approach Data preprocessing  Normalization Feature selection filtering  Pearson correlation; Kruskal-Wallis test Regularized Parzen Window Classifier  Parameter tuning by cross-validation Ensemble of classifiers  Classifiers differ by the selected features Active learning strategy  Uncertainty sampling + Clustering-based random sampling Active Learning Algorithm Based on Parzen Window Classification

7 Algorithm Details Data preprocessing  Missing value (did not address this issue)  Normalization (mean = 0 and std = 1 of all non-binary features) Feature selection filters  Pearson Correlation Test  Kruskal-Wallis Test  Calculated p-value for each feature  Selected M features with lowest p-values  Selected all features with p-value below 0.05 Active Learning Algorithm Based on Parzen Window Classification

Classification Model  Regularized Parzen Window Classifier (RPWC) ε is regularizing parameter (set to in our experiments) K is the Gaussian Kernel of form: where the σ represents the kernel size  RPWC easy to implement can learn highly nonlinear problems 8 Algorithm Details Active Learning Algorithm Based on Parzen Window Classification

9 Algorithm Details Active Learning Algorithm Based on Parzen Window Classification Classification Model  Ensemble of RPWC classifiers Base classifiers differ in features used  all features  p-value of Pearson correlation<0.05 (filter_data1)  10 features with smallest p-value of Pearson correlation<0.05 (filter_data2)  p-value of Kruskal-Wallis test<0.05 (filter_data3)  10 features with smallest p-value of Kruskal-Wallis test (filter_data4) Resulting ensemble classifier  Average of the 5 base RPWC  Base RPWC parameter tuning Examined 4 different values for kernel width σ: [M/9, M/3, M, 3M] Used leave-one-out cross-validation to select σ for each base classifier

10 Algorithm Details Active Learning Strategy Uncertainty sampling (EXPLOITATION)  Uncertainty score for example x is defined as score(x) = |p(y|x)  0.5|  Examples with the smallest score are selected  Advantage: Focuses on improving accuracy near decision boundary.  Disadvantage: Overlooks important underexplored regions Clustering-based random sampling (EXPLORATION)  Partition unlabeled data into k clusters  Select same number of random examples from each cluster  Advantage: Does not miss any important region  Disadvantage: Fails to focus on the uncertain regions Active Learning Algorithm Based on Parzen Window Classification

Algorithm Outline 11 Input: labeled set L, unlabeled set U Q  randomly select 20 unlabeled examples from U for t = 1 to 10 U  U – Q ; L  L + labeled(Q) for j = 1 to F F j  feature_filter(L) // feature selection filter C j  train_classifier(L, F j ) // train classifier C j from L using features F j ; determine model parameters by CV on L A j  accuracy(C j,L) // estimate accuracy of classifier C j by CV on L end for C avg  average(C j ) // build ensemble classifier C avg by averaging Q  (2|L|/3 of the most uncertain examples in U) Q  Q + (|L|/3 random examples chosen from randomly selected clusters of U) end for

Competition Results DATA SET Best ALC INTEL ALC (best team) TUCIS ALC TUCIS RANK A B C D E F Overall, our TUCIS Team ranked at 5 th place Active Learning Algorithm Based on Parzen Window Classification Official ALC Scores:

Competition Results DATA SET Best AUC INTEL AUC (best team) TUCIS AUC A B C D E F Active Learning Algorithm Based on Parzen Window Classification AUC Accuracy of the Final Classifier: Our final predictors are less accurate than the best challenge algorithms. This indicates that Parzen Window Classifiers are not the best choice for the challenge data sets.

Competition Results

Data Set One by one (alternative) Begin with 20 (submitted) A0.537 ± ± B0.267 ± ± C0.242 ± ± D0.576 ± ± E0.445 ± ± F0.750 ± ± Overall, querying beginning with 20 random examples is slightly better Active Learning Algorithm Based on Parzen Window Classification Post-Competition Experiments 1. Early Start  select the first 20 examples randomly one by one vs.  select the first 20 at once

16 Ensemble of 5 classifiers is the best overall choice Active Learning Algorithm Based on Parzen Window Classification Data Set One classifier (all features) One classifier (Pearson Corr)Two ClassifiersFive Classifiers A0.462± ± ± ±0.016 B0.260± ± ± ±0.023 C0.321± ± ± ±0.049 D0.670± ± ± ±0.050 E0.426± ± ± ±0.006 F0.620± ± ± ±0.025 Post-Competition Experiments 2. Comparison of Ensembles of Classifiers

17 Pre-clustering does not improve the performance Active Learning Algorithm Based on Parzen Window Classification Post-Competition Experiments 3. Comparison of 2 Querying strategies  2/3 uncertainty + 1/3 random vs.  preclustering DataSet2/3 + 1/3Preclustering A0.466± ±0.011 B0.273± ±0.057 C0.260± ±0.048 D0.600± ±0.036 E0.432± ±0.058 F0.757± ±0.062

Conclusions Our active learning algorithm  Uses ensemble of Parzen Window Classifiers  Uses feature selection filters  Combined uncertainty and random sampling  Has geometric sampling schedule Our team was ranked 5 th overall  The gap from the best performing algorithms was significant  Indicates PW classifiers are not appropriate for challenge sets  Building larger PW ensembles could improve performance Our exploration-exploitation querying approach was successful Active Learning Algorithm Based on Parzen Window Classification