ACTIVE LEARNING USING CONFORMAL PREDICTORS: APPLICATION TO IMAGE CLASSIFICATION HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions L. Makili 1, J. Vega 2, S. Dormido-Canto 3 1 Universidade Katyavala Bwila. Benguela (Angola) 2 Asociación EURATOM/CIEMAT para Fusión. Madrid (Spain) 3 Universidad Nacional de Educación a Distancia (UNED). Madrid (Spain) 7th Workshop on Fusion Data Processing Validation and Analysis

Outline Introduction Concepts overview Experimental results Conclusions HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Motivation 5 – class classification problem – Classification of TJ – II Thomson Scattering images Classifier based on conformal predictors, using SVM as the underlying algorithm – Computationally intensive task Patterns of TSD images: (a) BKGND, (b) COFF, (c) ECRH, (d) NBI and (e) STRAY HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Goal To find out a minimal and good enough training dataset for classification purposes HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Active learning The learning algorithm must have some control over the data from which it learns It must be able to query an oracle, requesting for labels of data samples that seem to be most informative for the learning process Proper selection of samples implies better performances with fewer data HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions Settles, B. “Active Learning Literature Survey. Computer Sciences Technical Report 1648”, University of Wisconsin – Madison, Available at reports/2009/TR1648.pdfhttp://research.cs.wisc.edu/tech reports/2009/TR1648.pdf

Uncertainty sampling The learning algorithm selects new examples when their class membership is unclear Suitable for classifiers that besides making classification decisions, estimates certainty of these decisions HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions Lewis, D. and Gale, W., “A Sequential Algorithm for Training Text Classifiers”. In Proceedings of the ACM – SIGIR Conference on Research and Development in Information Retrieval, Croft, W. B. and van Rijbergen, C. J. (eds). New York: Springer – Verlag, 1994, pp. 3 – 12

Conformal prediction Permits complementation of predictions made by machine learning algorithms with some measures of reliability Besides the label predicted for a new object, it outputs two additional values – Confidence – Credibility HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions Vovk, V., Gammerman, A. and Shafer, G., Algorithmic Learning in a Random World, New York: Springer Science + Business Media, Inc., 2005

Conformal prediction Used as nonconformity scores the Lagrange multipliers computed during SVM training Extended to a multiclass framework in a one- vs-rest approach HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Active learning algorithm Inputs – Initial training set T, calibration set C, pool of candidate samples U – Selection treshold τ, batch size β Train an initial classifier on T While a stopping-criterion is not reached – Apply the current classifier to the pool of samples – Rank the samples in the pool using the uncertainty criterion – Select the top β examples whose certainty level fall under the selection threshold τ – Ask teacher to label the selected samples and add them to the training set – Train a new classifier on the expanded training set HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

(Un)Certainty criteria Credibility: Confidence: Query-by-transduction: Multicriteria: – Combination 1: – Combination 2: HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions Ho, S. and Wechsler, H., “Query by Transduction”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30 (9), 2008, pp – 1571

Setup 1149 samples divided into – Initial training: 5 samples – Pool: 794 samples – Calibration set: 150 samples – Test set: 200 samples Batch size: 25 Selection treshold : 0.4 Used with Qbt, Comb1 and Comb2 HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Setup For Each criterion in (Qbt, Comb1, Comb2) – For experiment = 1 : 10 Select test set Run active learning algorithm selecting 700 samples hybridly For NumTrn = 50 : 50 : 700 – Train CP classifier on the first NumTrn samples – Aplly classifier to the test set End for NumTrn – End For experiment End For Each HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Results HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Results HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

Conclusions Active learning was applied to the selection of a minimal and good enough training dataset for classification purposes It allows reaching higher success rates and confidence in predictions with fewer data points, compared to the random selection of the training set Combining multiple criteria we can balance the trade-off between success rate and confidence of prediction improvement HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions

ACTIVE LEARNING USING CONFORMAL PREDICTORS: APPLICATION TO IMAGE CLASSIFICATION THANK YOU 7th Workshop on Fusion Data Processing Validation and Analysis HypHyp Introduction HypHyp Conceptual overview HypHyp Experiments and results HypHyp Conclusions