Presentation is loading. Please wait.

Presentation is loading. Please wait.

Advanced Residual Analysis Techniques for Model Selection A.Murari 1, D.Mazon 2, J.Vega 3, P.Gaudio 4, M.Gelfusa 4, A.Grognu 5, I.Lupelli 4, M.Odstrcil.

Similar presentations


Presentation on theme: "Advanced Residual Analysis Techniques for Model Selection A.Murari 1, D.Mazon 2, J.Vega 3, P.Gaudio 4, M.Gelfusa 4, A.Grognu 5, I.Lupelli 4, M.Odstrcil."— Presentation transcript:

1 Advanced Residual Analysis Techniques for Model Selection A.Murari 1, D.Mazon 2, J.Vega 3, P.Gaudio 4, M.Gelfusa 4, A.Grognu 5, I.Lupelli 4, M.Odstrcil 5 1 3 2 4 University of Rome “Tor Vergata” 5

2 The Scientific Method and Models Model building is a innate faculty of human beings because it allows handling information in a much more economic way. Model validation: process of assessing the quality of your model Model selection: process of selecting the best model, among many, to interpret the available data. A “subject value” approach is advocated: both model selection and model validation are “utility” based Model

3  Model selection =No established and universal methodology available  Model Falsification Criterion (MFC) : Estimates the most appropriate model among a set of competing and independent ones Based both on the accuracy and the robustness of the candidate models Implements a form of falsification principle more than the ‘Occam razor’ A model is not penalised for its complexity but on the basis of its lack of robustness Model selection: introduction

4  ROBUSTNESS: a model is not penalised for its complexity but more for its lack of robustness, i.e. the fact that its estimates degrade if errors in the parameters are made  small errors introduced on each model parameter  study of the repercussions on the global estimates The repercussions of the parameter errors are quantified with some sort of information theoretic quantity (Shannon entropy) calculated for the residuals MFC : THE BASIC PHYLOSOPHY Details in the paper A.Murari et al “Preliminary discussion on a new Model Selection Criterion, based on the statistics of the residuals and the falsification principle” Conference FDT2 (Frontiers in Diagnostic Technologies)

5 The correlation tests method  Hypothesis: the noise is random and additive  Consequence: the residuals of a perfect model should be randomly distributed  The model with the distribution of the residuals closer to a random one is to be preferred  A random distribution of numbers (residuals) maximizes the Shannon entropy

6  Mathematical expression of the Model Falsification Criterion : r i the absolute value of the i-th residual p i r the quantised probability of the i-th residual r par,i calculated after varying each parameter one at time (± 10 %) p par,i the quantised probability of this new residual n par the number of model parameters 1<i<n where n is the total number of experimental points MFC 1 MFC 2 MFC : Matematics an example

7 A better model = smaller sum of residuals + higher entropy of residuals A better model = smaller change in case of small errors introduced on the various parameters The best among the candidate models is the one which presents the lowest value of the MFC indicator  Mathematical expression of the Model Falsification Criterion : MFC : MATHEMATICS

8  A purely numerical equation :  exact solution + random noise of ± 10% = synthetic experimental data  Seven models created to fit the data NUMERICAL TESTS

9

10

11

12

13

14

15

16 Error of ±10% introduced on each parameter one at time MFC evaluated for each model Model classification obtained (classified in order of increasing MFC value) INTUITIVE CLASSIFICATION n Model 1 Model 7 Model 4 Model 2 Model 6 Model 3 Model 5 1 NUMERICAL TESTS

17 MFC AIC BIC B Model 1 41 -105 -549 Model 7 106 959 372 Model 4 120 889 435 Model 2 130 955 362 Model 6 309 1071 570 Model 3 493 1231 700 Model 5 5043 1634 1129  Various forms of MFC criteria seem to outperform traditional criteria in particular for extrapolation and for high levels of noise NUMERICAL TESTS: Results

18  Electron temperature required to access the H-mode of confinement in tokamak plasmas :  Variables scanned over their respective interval (using 500 values)  Synthetic experimental data generated by adding a random noise of ±10%  Five models considered to test the indicator Bt R a n q N Min 2 0.8 0.2 1 2 Max 8 2 0.7 10 8 Scaling Laws: Numerical tests

19 Error of ±10% introduced on each parameter one at time MFC evaluated for each model Model classification obtained (classified in order of increasing MFC value) INTUITIVE CLASSIFICATION n Model 1 Model 2 Model 5 Model 3 Model 4 SCALING LAWS

20 INTUITIVE CLASSIFICATION MFC value n Model 1 Model 2 = 2.40*10 6 Model 2 Model 1 = 3.21*10 6 Model 5 Model 5 = 4.81*10 6 Model 3 Model 3 = 1.12*10 7 Model 4 Model 4 = 1.22*10 7 Results of the MFC criterion : MFC Classification Results of the MFC criterion: since the exponent of n e is very low, at realistic noise levels the MFC realises that the models containing this quantity are prone to overfitting and that models without this parameter are more robust.

21 VS  small dependence from the density  when affected by an error bigger MFC value  not a fundamental variable VSResiduals The MFC criterion also automatically penalises the major and minor radii from the scaling laws of individual devices because they do not vary over a significant range

22  errors introduced = bounds of the 85% confidence interval theoretical models variables used models generated n Chankin Bt, q, R model 1 Kernel collisionless Kernel collisional Bt, q, R, n model 2 Rogister Scott Bt, n model 3 Shaing & Crume q, R, a, n model 4 none Bt, q, R, a, n model 5 none Bt, q, n model 6 linear regression lower upper lower bound central value upper bound n 3.05 3.25 3.44 ITPA DATABASE

23  JET : Linear regression VS well-known theoretical models ITPA DATABASE

24  JET : 469 shots MFC n Model 6 4.49*10 4 Model 3 4.49*10 4 Model 5 2.08*10 5 Model 2 2.18*10 5 Model 1 3.27*10 5 Model 4 7.88*10 5 ITPA DATABASE

25  ASDEX : 48 shots MFC n Model 6 4265 Model 3 4664 Model 5 1.83*10 5 Model 1 2.22*10 5 Model 2 4.11*10 5 Model 4 1.05*10 6 ITPA DATABASE

26  CMOD : 98 shots MFC n Model 6 2.13*10 5 Model 3 2.21*10 5 Model 2 1.06*10 9 Model 1 1.24*10 9 Model 4 1.17*10 10 Model 5 1.49*10 10 ITPA DATABASE

27  JET + ASDEX + CMOD : 615 shots MFC n Model 4 2.20*10 5 Model 3 2.25*10 5 Model 6 2.37*10 5 Model 5 2.68*10 5 Model 2 2.72*10 5 Model 1 2.93*10 5 ITPA DATABASE

28  Summary of the best results obtained :  JET :  ASDEX :  CMOD :  All the database : ITPA DATABASE: Summary The MFC determines that Te depends only on Bt, n and q but with not the same exponents at all Results are very different from the ones obtained with each independent tokamak

29  Analyse of the best results obtained : ITPA: Comparison Exponents  Plasma radius, a and R, not evaluated as fundamental variables  Not the same exponents at all JETASDEXCMOD

30 -The MFC criterion has some potential advantages compared to traditional criteria particularly in the case of scaling laws and extrapolation -The application to the ITPA database has given some interesting results (See also talk by I.Lupelli) -Model selection: various alternative MFC criteria are being applied to the power threshold to reach the H mode of confinement Summary and Future developments


Download ppt "Advanced Residual Analysis Techniques for Model Selection A.Murari 1, D.Mazon 2, J.Vega 3, P.Gaudio 4, M.Gelfusa 4, A.Grognu 5, I.Lupelli 4, M.Odstrcil."

Similar presentations


Ads by Google