Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK

Publications (580) Citations (4594) “CLASSIFIER ENSEMBLE DIVERSITY” Search on 10 Sep 2014

MULTIPLE CLASSIFIER SYSTEMS 30 INT JOINT CONF ON NEURAL NETWORKS (IJCNN) 22 PATTERN RECOGNITION 17 NEUROCOMPUTING 14 EXPERT SYSTEMS WITH APPLICATIONS 13 INFORMATION SCIENCES 12 APPLIED SOFT COMPUTING 11 PATTERN RECOGNITION LETTERS 10 INFORMATION FUSION 9 IEEE INT JOINT CONF ON NEURAL NETWORKS 9 KNOWLEDGE-BASED SYSTEMS 7 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 7 INT J OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE 6 MACHINE LEARNING 5 IEEE TRANSACTIONS ON NEURAL NETWORKS 5 JOURNAL OF MACHINE LEARNING RESEARCH 5 APPLIED INTELLIGENCE 4 INTELLIGENT DATA ANALYSIS 4 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 4 ADVANCES IN KNOWLEDGE DISCOVERY AND DATA MINING 4 NEURAL INFORMATION PROCESSING papers, 335 journals / conferences

MULTIPLE CLASSIFIER SYSTEMS 30 INT JOINT CONF ON NEURAL NETWORKS (IJCNN) 22 PATTERN RECOGNITION 17 NEUROCOMPUTING 14 EXPERT SYSTEMS WITH APPLICATIONS 13 INFORMATION SCIENCES 12 APPLIED SOFT COMPUTING 11 PATTERN RECOGNITION LETTERS 10 INFORMATION FUSION 9 IEEE INT JOINT CONF ON NEURAL NETWORKS 9 KNOWLEDGE-BASED SYSTEMS 7 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 7 INT J OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE 6 MACHINE LEARNING 5 IEEE TRANSACTIONS ON NEURAL NETWORKS 5 JOURNAL OF MACHINE LEARNING RESEARCH 5 APPLIED INTELLIGENCE 4 INTELLIGENT DATA ANALYSIS 4 IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 4 ADVANCES IN KNOWLEDGE DISCOVERY AND DATA MINING 4 NEURAL INFORMATION PROCESSING papers, 335 journals / conferences

Where in the world are we? China 140 UK 68 USA 63 Spain 55 Brazil 41 Canda 32 Poland 28 Iran 23 Italy France 11

Where in the world are we? China 140 UK 68 USA 63 Spain 55 Brazil 41 Canda 32 Poland 28 Iran 23 Italy France 11 Laurent HEUTTE Professor of Computer Science, University of Rouen, France

Are we still talking about diversity in classifier ensembles? Apparently yes...

That elusive diversity... classifier feature values (object description) classifier class label “combiner” Classifier ensemble

That elusive diversity... Classifier 2 Classifier 1 correct wrong correct wrong independent outputs ≠ independent errors hence, use ORACLE outputs Number of instances labelled correctly by classifier 1 and mislabelled by classifier 2

That elusive diversity... Classifier 2 Classifier 1 correct wrong correct wrong Q kappa correlation (rho) disagreement double fault...

That elusive diversity... SEVENTY SIX !!!

Do we need more “NEW” pairwise diversity measures? Looks like we don’t...

proposed by Margineantu and Dietterich in 1997 visualise individual accuracy and diversity in a 2-dimensional plot have been used to decide which ensemble members can be pruned without much harm to the overall performance Kappa-error diagrams

Adaboost 75.0% Bagging 77.0% Random subspace 80.9% Random oracle 83.3% Rotation Forest 84.7%  sonar data (UCI): 260 instances, 60 features, 2 classes, ensemble size L = 11 classifiers, base model – tree C4.5 Example Kuncheva L.I., A bound on kappa-error diagrams for analysis of classifier ensembles, IEEE Transactions on Knowledge and Data Engineering, 2013, 25 (3), (DOI: /TKDE ).

correctwrong C1 correct ab wrong cd C2 error kappa = (observed – chance)/(1-chance) Kappa-error diagrams

bound (tight) bound (tight) error kappa Kappa-error diagrams

error kappa Kappa-error diagrams – simulated ensembles L = 3

error kappa Kappa-error diagrams – real data L = 11

error kappa Real data: 77,422,500 pairs of classifiers room for improvement

Is there space for new classifier ensembles? Looks like yes...

Good and Bad diversity Diversity is not MONOTONICALLY related to ensemble accuracy

Good and Bad diversity 3 classifiers: A, B, C 15 objects, wrong vote, correct vote individual accuracy = 10/15 = P = ensemble accuracy independent classifiers P = 11/15 = identical classifiers P = 10/15 = dependent classifiers 1 P = 7/15 = dependent classifiers 2 P = 15/15 = ABCABC ABCABC ABCABC ABCABC MAJORITY VOTE

Good and Bad diversity 3 classifiers: A, B, C 15 objects, wrong vote, correct vote individual accuracy = 10/15 = P = ensemble accuracy independent classifiers P = 11/15 = identical classifiers P = 10/15 = dependent classifiers 1 P = 7/15 = dependent classifiers 2 P = 15/15 = ABCABC ABCABC ABCABC ABCABC MAJORITY VOTE Good diversity Bad diversity

Good and Bad diversity Data set Z Ensemble, L = 7 classifiers   Are these outputs diverse?

Good and Bad diversity Data set Z Ensemble, L = 7 classifiers    How about these?

Good and Bad diversity Data set Z Ensemble, L = 7 classifiers    3 vs 4... Can’t be more diverse, really...

Good and Bad diversity Data set Z Ensemble, L = 7 classifiers    MAJORITY VOTE  Good diversity

Good and Bad diversity Data set Z Ensemble, L = 7 classifiers    MAJORITY VOTE Bad diversity  

Good and Bad diversity maj  maj  Decomposition of the Majority Vote Error Individual error Subtract GOOD diversity Add BAD diversity Brown G., L.I. Kuncheva, "Good" and "bad" diversity in majority vote ensembles, Proc. Multiple Classifier Systems (MCS'10), Cairo, Egypt, LNCS 5997, 2010,

Good and Bad diversity     Note that diversity quantity is 3 in both cases

Ensemble Margin     POSITIVE NEGATIVE

Ensemble Margin Average margin However, nearly all diversity measures are functions of Average absolute margin or Average square margin Margin has no sign...

Ensemble Margin

Diversity is not MONOTONICALLY related to ensemble accuracy So, STOP LOOKING for a monotonic relationship!!!

Conclusions 1.Beware! Overflow of diversity measures! 2.In theory, there is some room for better classifier ensembles. 3.Diversity is not monotonically related to ensemble accuracy, hence larger diversity does not necessarily mean better accuracy. Directly engineered or heuristic? – up to you

36

37