1. Transductive Reliability Estimation for Kernel Based Classifiers 1 Department of Computer Science, University of Ioannina, Greece 2 Faculty of Computer and Information Science, University of Ljubljana, Slovenia Dimitris Tzikas 1, Matjaz Kukar 2, Aristidis Likas 1 tzikas@cs.uoi.grtzikas@cs.uoi.gr, matjaz.kukar@fri.uni-lj.si, arly@cs.uoi.grmatjaz.kukar@fri.uni-lj.siarly@cs.uoi.gr

2. Introduction We wish to assess the reliability of single example classifications of kernel-based classifiers Support Vector Machine (SVM) Relevance Vector Machine (RVM) Such assessment is useful in risk-sensitive applications Weighted combination of several classifiers Reliability measures can be obtained directly from the classifier outputs We propose the use of the transduction reliability methodology to kernel- based classifiers

3. Kernel Classifiers  Mapping function to the feature space:  Kernel function: inner product in the feature space:  Kernel Classifier:  Training: estimate w using training set D  Prefer sparse solutions: most w n →0  SVM and RVM differ in the training method.

4. Support Vector Machine (SVM)  SVM model (two-class)  Maximize margin from the separating hyperplane in feature space subject to  C is a hyperparameter to be prespecified

5. Reliability Measure for SVM  The points near the decision boundary have lower reliability.  Output |y svm (x)|: distance from the separating hyperplane (decision boundary).  Transform the outputs to probabilities by applying the sigmoid function:  Define reliability measure:

6. Relevance Vector Machine  RVM model (two-class):  Provides posterior probability for class C 1  RVM is a Bayesian linear model with hierarchical prior on weights w  The hierarchical prior enforces sparse solutions

7. Relevance Vector Machine  Compute by maximizing likelihood  Many  Compute w:  Incremental RVM:  Start from an empty model and a set of basis functions  Incrementally add (and delete) terms  Convenient for the transduction approach which requires retraining

8. RVM Reliability Measure  Compute reliability estimate for the decision of input x as:

9. Transductive Reliability Estimation (Kukar and Kononenko, ECML 2002)  The transductive methodology estimates reliability of individual classifications.  Measures stability of the classifier after small perturbation to the training set (the test example with the class label is added to the training set) retraining of the classifier  Assumption: For reliable decisions, this process should not lead to significant model changes.  The method can be applied to any classifier that outputs class posterior probabilities  Transduction requires retraining → incremental training methods are preferable

10. Transductive Reliability Estimation  Assume a classifier CL1 and a training set  Compute class posteriors p k and classify a test example.  Objective: Estimate reliability of decision  Transductive step  Add previous test example with the classification label to training set  Train a classifier CL2  Compute class posteriors q k and classify the test example.

11. Transductive Reliablility Estimation  Difference between the class posterior vectors p and q of CL1 and CL2 is an estimate of reliability.  Symmetric KL divergence:  Scale reliability values to [0, 1]:  Reliable estimations:  How do we select threshold T?

12. Selecting the Threshold  Use Leave-one-out to obtain classifications and reliability estimations TRE(x) for each example x  For a threshold T  We wish: D 1 to contain incorrectly classified examples D 2 to contain correctly classified examples  Select T that maximizes Information Gain check

13. Evaluation of reliability measures  Transduction has been evaluated on several classifiers: decision trees, Naïve Bayes  We applied the transduction approach to SVM and RVM SVM is retrained from scratch with same hyperparameters For RVM we considered both retraining from scratch and incremental retraining  Reliability measures: ΤRE SVM, TRE RVM and TRE RVM(inc).  TRE RVM(inc). is computationally efficient (50 – 100 times faster)  We compare direct measures RE SVM, RE RVM with transductive measures.

14. Evaluation of reliability measures  3 UCI medical datasets (RBF kernel)  1 bioinformatics (linear kernel) dataset (leukemia)  Cardiac Artery Disease (CAD) dataset (RBF kernel) Comparison with expert physicians  Evaluation of reliability estimation methods  Use Leave-one-out to decide for correct or incorrect classification of each example and compute the reliability estimates (RE(x), TRE(x)).  For each dataset and measure determine the threshold that maximizes the information gain  Use the maximum information gain to compare different reliability measures on each dataset

15. Evaluation on UCI Datasets  Max IG of TRE SVM is higher than RE SVM  Max IG of TRE RVM(inc) is higher than TRE RVM and RE RVM (except hepatitis dataset) Methodhepatitisnew- thyroid wdbcleukemia RE SVM 0.1060.0830.0360.054 TRE SVM 0.1200.0920.0470.073 RE RVM 0.1090.0680.0910.089 TRE RVM 0.1780.0620.0940.062 TRE RVM(inc) 0.1330.0720.1060.107

16. Application on CAD (comparison to physicians)  Coronary Artery Disease (CAD) dataset (University Clinical Centre, Ljubljana).  327 cases (228 positive, 99 negative)  Physicians estimate reliability by computing a posterior probability based on diagnostic tests and other information.  For posterior > 0.9 or < 0.1 diagnosis is assumed reliable.

17. Application on CAD PositiveNegative MethodReliable (%) Correct ( %) Errors (%) Reliable (%) Correct (%) Errors (%) Physicians 76 724 52 457 RE SVM 65 0 34 304 TRE SVM 78 762 65 578 RE RVM 63.4 630.4 60 546 TRE RVM 68.3 671.3 54 495 TRE RVM(inc) 69.4 690.4 61 547

18. Conclusions  We applied the transductive approach to kernel-based models  Support Vector Machine (SVM)  Relevance Vector Machine (RVM)  We compared direct and transductive reliability measures on several datasets  We also compared against physician’s performance on a real dataset for Diagnosis of Coronary Artery Disease (CAD)  The transductive approach seems to provide good estimates

19. Future work  Examine incremental training methods for SVM.  Define reliability measures based on the structural difference between the classifiers CL1 and CL2.  Use transduction to estimate ‘strangeness’ of an example in the typicalness framework for confidence estimation (Kukar, KIS 2006)