klinické neurofyziologie Data mining EEG data: When models can be trusted? Pavel Kordík kordikp@fel.cvut.cz Miroslav Čepek Computational Intelligence Group Department of Computer Science and Engineering Faculty of Electrical Engineering Czech Technical University in Prague 55. Společný sjezd klinické neurofyziologie

FFT, Automated preprocessing Data mining model FFT, Automated preprocessing Raw EEG signals Feature extraction Manual annotation GAME neural network Training data set Data mining This presentation Model (classifier) Classification Annotated signal

Raw EEG signals Training data set Newborn sleep stage EEG dataset Institute for Care of Mother and Child, Prague FFT, statistics annotation: fp1_min fp1_max fp1_skewness fp1_kurtosis fp1_mean fp1_std fp1_mean_abs_first_derivation fp1_fft_abs_delta … fp2 … Wake Active Quiet -60.94 35.94 -0.31 2.56 -8.01 19.21 2.77 18.74 3.97 29.67 2759.77 … 1 -53.12 51.56 0.31 2.80 -9.52 19.41 2.93 14.06 4.12 15.61 2766.10 -59.38 50.00 0.15 2.85 -8.28 20.56 2.75 10.94 3.51 17.20 2888.78 Training data set

GAME Neural Network GAME models grow automatically using the training data set.

Model (classifier) Classifier Wake: output equals 1 for Wake class 0 for Active,Quiet Classifier for Active Classifier for Quiet Wake (0,1)

Models can be written as equation Wake=0.257*exp(1.236/(1+exp(7.718*exp(-4.307*c4_fft_rel_gamma+0.43)-1.985*ecg1_mean_abs_second_derivation+22.58))-2147483.8*fp1_fft_rel_beta1+0.14*png1_skewness)-0.269 Quiet=… Active=…

Recall of model on testing data annotation: fp1_min fp1_max fp1_skewness fp1_kurtosis fp1_mean fp1_std fp1_mean_abs_first_derivation fp1_fft_abs_delta … fp2 … Wake Active Quiet -60.94 35.94 -0.31 2.56 -8.01 19.21 2.77 18.74 3.97 29.67 2759.77 … 1 -53.12 51.56 0.31 2.80 -9.52 19.41 2.93 14.06 4.12 15.61 2766.10 -59.38 50.00 0.15 2.85 -8.28 20.56 2.75 10.94 3.51 17.20 2888.78 GAME classifier WAKE GAME classifier ACTIVE GAME classifier QUIET 0.02 0.99 0.2 Correct classification But: 0.22 0.21 0.1 Usual approach: Choose a classifier with maximum response 0.99 0. 99 0.99 ???????????

Where is the problem? Can we trust classifiers? Sometimes, their output is random (not based on training data) How to estimate credibility of the classifier? Our approach: Create ensembles of classifiers

Ensemble classifiers – what is it? The collection of classifiers is trained for the same task. Input variables Classifier WAKE 1 Classifier WAKE 2 Classifier WAKE 3 … Classifier WAKE n Output variable Output variable Majority or Multiply Ensemble output

Recall of ensembles on testing data annotation: fp1_min fp1_max fp1_skewness fp1_kurtosis fp1_mean fp1_std fp1_mean_abs_first_derivation fp1_fft_abs_delta … fp2 … Wake Active Quiet -60.94 35.94 -0.31 2.56 -8.01 19.21 2.77 18.74 3.97 29.67 2759.77 … 1 -53.12 51.56 0.31 2.80 -9.52 19.41 2.93 14.06 4.12 15.61 2766.10 -59.38 50.00 0.15 2.85 -8.28 20.56 2.75 10.94 3.51 17.20 2888.78 GAME classifier WAKE GAME classifier WAKE GAME classifier ACTIVE GAME classifier QUIET GAME classifier WAKE GAME classifier WAKE GAME classifier WAKE GAME classifier WAKE GAME classifier WAKE GAME classifier WAKE GAME classifier WAKE GAME classifier WAKE 4 GAME classifier ACTIVE 4 GAME classifier QUIET 4 {0.2, 0.6, 0.45, 0.8} {0.99, 0.98, 1, 0.99} {0.99, 0.01, 1, 1} MUL: 0.04 0.96 0.09 AVG: 0.4 0.99 0.8 STD: high low high

Fuzzy prediction Unclear, guess WAKE Definitely not ACTIVE Definitely Testing example 1 Testing example 2 Testing example 3 Testing example 4 Unclear, guess WAKE Definitely not ACTIVE Definitely not WAKE, QUIET Most probably ACTIVE Totally unclear, guess ACTIVE, probably not QUIET Clear WAKE, not ACTIVE,QUIET

WAKE AVG: MUL:

QUIET AVG: MUL:

ALL AVG: MUL:

Conclusion Credibility of classification Some examples cannot be correctly classified, based on information contained in training data We propose the approach, how to identify these examples FAKE GAME project demostration