A Combinatorial Fusion Method for Feature Mining Ye Tian, Gary Weiss, D. Frank Hsu, Qiang Ma Fordham University Presented by Gary Weiss

2 Introduction Feature construction/engineering often a critical step in the data mining process –Can be very time-consuming and may require a lot of manual effort Our approach is to use a combinatorial method to automatically construct new features –We refer to this as “feature fusion” –Geared toward helping to predict rare classes –For now it is restricted to numerical features, but can be extended to other features

3 How does this relate to MMIS? One MMIS category is local pattern analysis –How to efficiently identify quality knowledge from a single data source –Listed data preparation and selection as subtopics and also mentioned fusion We acknowledge that this work probably is not what most people think of as MMIS

4 How can we view this work as MMIS? Think of each feature as piece of information –Our fusion approach integrates these pieces Fusion itself is a proper topic for MMIS since it can also be used with multiple info sources –The fusion method we employ does not really care if the information (i.e., features) are from a single source As complexity of constructed features increases, each can be viewed as a classifier –Each fused feature is an information source –This view is bolstered by other work on data fusion that using ensembles to combine each fused feature

5 Description of the Method 1.A data set is a collection of records where each feature has a score –We assume numerical features 2.We then replace scores by ranks –Ordering of ranks determined by whether larger or small scores better predict class 3.Compute performance of each feature 4.Compute performance of feature combinations 5.Decide which combinations to evaluate/use

6 Step 1: A data set F1F2F3F4F5Class A B C D E F G H

7 Step 2: Scores replaced by Ranks F1F2F3F4F5 A12225 B21443 C33154 D44731 E66867 F78386 G H87572

8 Step 3: Compute Feature Performance Performance measures how well feature predicts minority class We sort rows by feature rank and measure performance on top n%, where n% belong to minority class In this case we evaluate on top 3 rows. Since 2 of 3 are minority (class=1), performance =.66 F2 RankClass B10 A21 C31 D40 G51 E60 H70 F80

9 Step 3 continued FeaturePerformance F10.67 F20.67 F30.67 F40.67 F50.00

10 Step 4: Compute Performance of Feature Combinations Let F6 be fused F1F2F3F4F5 Rank combination function is average of ranks Compute rank of F6 for each record Compute performance of F6 as in step 3 F6 Rank F6 Rank A2.4E6.6 B2.8F6.4 C3.2G5.0 D3.8H5.8

11 Step 5: What Combinations to Use? Given n features there are 2 n – 1 possible combinations –C(n,1) + C(n,2) … C(n.n) –This “fully exhaustive” fusion strategy is practical for many values of n We try other strategies in case not feasible –k-exhaustive strategy selects k best features and tries all combinations –k-fusion strategy uses all n features but fuses at most k features at once

12 Combinatorial Fusion Table Number Features k-fusion for values of k shown below

13 Combinatorial Fusion Algorithm Combinatorial strategy generates features –Performance metric determines which are best Used to determine which k features for k-fusion Also used to determine order of features to add We add a feature if it leads to a statistically significant improvement (p ≤.10) –As measured on validation data –This limits the number of features –But requires a lot of computation

14 Example Run of Algorithm AUC p-value Feature validtest(+ means added) {F1,F2,F3,F4,F5} F3F F1F F1F3

15 Description of Experiments We use Weka’s DT, 1-NN, and Naïve Bayes methods Analyze performance on 10 data sets –With and without fused features –Focus on AUC as the main metric More appropriate than accuracy especially with skewed data Use 3 combinatorial fusion strategies –2-fusion, 3-fusion, and 6-exhaustive

16 Results Summary Results over all 10 Data Sets Results over 4 Most Skewed Data Sets (< 10% Minority)

17 Discussion of Results No one of the 3 fusion schemes is clearly best The methods seem to help, but the biggest improvement is clearly with the DT method –May be explained by traditional DT methods having limited expressive power They can only consider 1 feature at a time Can never perfectly learn simple concepts like F1+F2 > 10, but can with feature-fusion –Bigger improvement for highly skewed data sets Identifying rare cases is difficult and may require looking at many features in parallel

18 Future Work More comprehensive experiments –More data sets, more skewed data sets, more combinatorial fusion strategies Use of heuristics to more intelligently choose fused features –Performance measure now used only to order –Use of diversity measures –Avoid building classifier to determine which fused features to add Handle non-numerical features

19 Conclusion Showed how a method from information fusion can be applied to feature construction Results encouraging but more study needed Extending the method should lead to further improvements

20 Questions?

21 Detailed Results: Accuracy

22 Detailed Results: AUC