1. Advanced Residual Analysis Techniques for Model Selection A.Murari 1, D.Mazon 2, J.Vega 3, P.Gaudio 4, M.Gelfusa 4, A.Grognu 5, I.Lupelli 4, M.Odstrcil 5 1 3 2 4 University of Rome “Tor Vergata” 5

2. The Scientific Method and Models Model building is a innate faculty of human beings because it allows handling information in a much more economic way. Model validation: process of assessing the quality of your model Model selection: process of selecting the best model, among many, to interpret the available data. A “subject value” approach is advocated: both model selection and model validation are “utility” based Model

3.  Model selection =No established and universal methodology available  Model Falsification Criterion (MFC) : Estimates the most appropriate model among a set of competing and independent ones Based both on the accuracy and the robustness of the candidate models Implements a form of falsification principle more than the ‘Occam razor’ A model is not penalised for its complexity but on the basis of its lack of robustness Model selection: introduction

4.  ROBUSTNESS: a model is not penalised for its complexity but more for its lack of robustness, i.e. the fact that its estimates degrade if errors in the parameters are made  small errors introduced on each model parameter  study of the repercussions on the global estimates The repercussions of the parameter errors are quantified with some sort of information theoretic quantity (Shannon entropy) calculated for the residuals MFC : THE BASIC PHYLOSOPHY Details in the paper A.Murari et al “Preliminary discussion on a new Model Selection Criterion, based on the statistics of the residuals and the falsification principle” Conference FDT2 (Frontiers in Diagnostic Technologies)

5. The correlation tests method  Hypothesis: the noise is random and additive  Consequence: the residuals of a perfect model should be randomly distributed  The model with the distribution of the residuals closer to a random one is to be preferred  A random distribution of numbers (residuals) maximizes the Shannon entropy

6.  Mathematical expression of the Model Falsification Criterion : r i the absolute value of the i-th residual p i r the quantised probability of the i-th residual r par,i calculated after varying each parameter one at time (± 10 %) p par,i the quantised probability of this new residual n par the number of model parameters 1<i<n where n is the total number of experimental points MFC 1 MFC 2 MFC : Matematics an example

7. A better model = smaller sum of residuals + higher entropy of residuals A better model = smaller change in case of small errors introduced on the various parameters The best among the candidate models is the one which presents the lowest value of the MFC indicator  Mathematical expression of the Model Falsification Criterion : MFC : MATHEMATICS

8.  A purely numerical equation :  exact solution + random noise of ± 10% = synthetic experimental data  Seven models created to fit the data NUMERICAL TESTS

16. Error of ±10% introduced on each parameter one at time MFC evaluated for each model Model classification obtained (classified in order of increasing MFC value) INTUITIVE CLASSIFICATION n Model 1 Model 7 Model 4 Model 2 Model 6 Model 3 Model 5 1 NUMERICAL TESTS

17. MFC AIC BIC B Model 1 41 -105 -549 Model 7 106 959 372 Model 4 120 889 435 Model 2 130 955 362 Model 6 309 1071 570 Model 3 493 1231 700 Model 5 5043 1634 1129  Various forms of MFC criteria seem to outperform traditional criteria in particular for extrapolation and for high levels of noise NUMERICAL TESTS: Results

18.  Electron temperature required to access the H-mode of confinement in tokamak plasmas :  Variables scanned over their respective interval (using 500 values)  Synthetic experimental data generated by adding a random noise of ±10%  Five models considered to test the indicator Bt R a n q N Min 2 0.8 0.2 1 2 Max 8 2 0.7 10 8 Scaling Laws: Numerical tests

19. Error of ±10% introduced on each parameter one at time MFC evaluated for each model Model classification obtained (classified in order of increasing MFC value) INTUITIVE CLASSIFICATION n Model 1 Model 2 Model 5 Model 3 Model 4 SCALING LAWS

20. INTUITIVE CLASSIFICATION MFC value n Model 1 Model 2 = 2.40*10 6 Model 2 Model 1 = 3.21*10 6 Model 5 Model 5 = 4.81*10 6 Model 3 Model 3 = 1.12*10 7 Model 4 Model 4 = 1.22*10 7 Results of the MFC criterion : MFC Classification Results of the MFC criterion: since the exponent of n e is very low, at realistic noise levels the MFC realises that the models containing this quantity are prone to overfitting and that models without this parameter are more robust.

21. VS  small dependence from the density  when affected by an error bigger MFC value  not a fundamental variable VSResiduals The MFC criterion also automatically penalises the major and minor radii from the scaling laws of individual devices because they do not vary over a significant range

22.  errors introduced = bounds of the 85% confidence interval theoretical models variables used models generated n Chankin Bt, q, R model 1 Kernel collisionless Kernel collisional Bt, q, R, n model 2 Rogister Scott Bt, n model 3 Shaing & Crume q, R, a, n model 4 none Bt, q, R, a, n model 5 none Bt, q, n model 6 linear regression lower upper lower bound central value upper bound n 3.05 3.25 3.44 ITPA DATABASE

23.  JET : Linear regression VS well-known theoretical models ITPA DATABASE

24.  JET : 469 shots MFC n Model 6 4.49*10 4 Model 3 4.49*10 4 Model 5 2.08*10 5 Model 2 2.18*10 5 Model 1 3.27*10 5 Model 4 7.88*10 5 ITPA DATABASE

25.  ASDEX : 48 shots MFC n Model 6 4265 Model 3 4664 Model 5 1.83*10 5 Model 1 2.22*10 5 Model 2 4.11*10 5 Model 4 1.05*10 6 ITPA DATABASE

26.  CMOD : 98 shots MFC n Model 6 2.13*10 5 Model 3 2.21*10 5 Model 2 1.06*10 9 Model 1 1.24*10 9 Model 4 1.17*10 10 Model 5 1.49*10 10 ITPA DATABASE

27.  JET + ASDEX + CMOD : 615 shots MFC n Model 4 2.20*10 5 Model 3 2.25*10 5 Model 6 2.37*10 5 Model 5 2.68*10 5 Model 2 2.72*10 5 Model 1 2.93*10 5 ITPA DATABASE

28.  Summary of the best results obtained :  JET :  ASDEX :  CMOD :  All the database : ITPA DATABASE: Summary The MFC determines that Te depends only on Bt, n and q but with not the same exponents at all Results are very different from the ones obtained with each independent tokamak

29.  Analyse of the best results obtained : ITPA: Comparison Exponents  Plasma radius, a and R, not evaluated as fundamental variables  Not the same exponents at all JETASDEXCMOD

30. -The MFC criterion has some potential advantages compared to traditional criteria particularly in the case of scaling laws and extrapolation -The application to the ITPA database has given some interesting results (See also talk by I.Lupelli) -Model selection: various alternative MFC criteria are being applied to the power threshold to reach the H mode of confinement Summary and Future developments