1. Quantification of the non- parametric continuous BBNs with expert judgment Iwona Jagielska Msc. Applied Mathematics

2. Outline of the presentation I PART 1. Introduction II PART 2. Method of eliciting conditional rank correlations 3. Comparison of algorithms to calculate multivariate normal probabilities 4. Presentation of elicitation software UniExp III PART 5. Building the Maintenance Performance Model Model variables Dependence relation 6. Results 7. Conclusions and recommendations I PART 1. Introduction II PART 2. Method of eliciting conditional rank correlations 3. Comparison of algorithms to calculate multivariate normal probabilities 4. Presentation of elicitation software UniExp III PART 5. Building the Maintenance Performance Model Model variables Dependence relation 6. Results 7. Conclusions and recommendations

3. CATS – casual model for Air Transport Safety – motivation and purpose - three sectors of human performance ATC Model, Flight Crew Performance Model Maintenance Performance Model 1. Introduction

4. 2. Way of assessing dependence relations Conditional Rank correlations  Conditional probabilities of exceedence Why normal copula? - Advantages known relation between partial and rank correlation equal conditional and partial correlations possess zero independent property - Disadvantages no analytical form for multivariate cumulative distribution function P 1 = P ( F X4 (X 4 ) > q | F X3 (X 3 ) > q ) “Suppose that the variable X 3 was observed above its q th quantile. What is the probability that also X 4 will be observed above its q th quantile? “

5. 2. Way of assessing dependence relations To see the conditional probability as a function of rank correlation we integrate bivariate normal density over the given region. we can calculate relationship between rank correlation and conditional probability

6. P 2 = P ( F X4 (X 4 ) > q | F X3 (X 3 ) > q, F X2 (X 2 ) > q ) “ Suppose that not only variable X 3 but also X 2 was observed above their q th quantile. What is the probability that also X 4 will be observed above its q th quantile? ” To find the conditional probability we integrate trivariate normal density over the given region with covariance matrix. 2. Way of assessing dependence relations We assess the higher order conditional rank correlation in the similar way.

7. Proposed numerical integration methods: Algorithm I and II – by Genz - first we apply transformation to simplify integration region - later randomized quasi Monte Carlo method is used - different choice of quasi points - in algorithm I we specify number of points; algorithm II assign number of points, s.t. the requested accuracy is provided Algorithm III and IV - based on successive subdivisions of integration region, where each subdivision is used to provide a better approximation of the integrand - polynomial rule is used to approximate integrand on each subregions - error estimate – difference between two polynomial rules of different order - algorithm IV may involve some simplification routines (change of variables) 3.1. Algorithms to calculate multivariate normal probabilities probabilities

8. T Ai – time of calculation for algorithm i P Ai – probability obtained by algorithm i E Ai – estimated error of approximation provided by algorithm i 700, 1500 – number of quasi random point in Alg I; 10 -5 requested accuracy for Alg II 3.2. NumericalComparison 3.2. Numerical Comparison dimension = 4, determinant = 0.5271 dimension = 7, determinant = 0.489

9. Algorithms III and IV are unpractical for large scale applications since they require long time for numerical calculations - time for hypercube [0.5, inf] 7 is more than 700seconds - when the procedure of subdivision of integration region is applied, algorithm do not provide the total error In Algorithm I user needs to specify number of quasi random points used to calculation; there is no control of provided error of estimation; time of calculation depends on the number of points, not of covariance matrix Time of calculation for Algorithm II is sometimes grater than for Algorithm I; time depends on covariance matrix; number of quasi random points depends on requested accuracy of solution At this moment Algorithm II is used in the software UniExp as the most accurate one; Algorithm I also has future potencial for implementation. 3. NumericalComparison – brief summary 3. Numerical Comparison – brief summary

10. 4. Software elicitation tool - UniExp 1 Step – input of nodes and connections

11. 4. Software elicitation tool - UniExp 2 Step – elicitation of conditional rank correlations

12. 4. Software elicitation tool - UniExp Values of Rank Correlations can be found in RankCorrelationValues.txt file

13. 5. Maintenance Performance Model

14. Elicitation with single expert we asked – 4 questions about marginal distributions – classical method of expert judgment 7 questions about conditional probabilities of exceedance 5. Maintenance Performance Model – dependence relation All variables are negatively correlated with variable human error

15. 5. Maintenance Performance Model At the bottom of each histogram the expectation and standard derivation are shown. Unconditional expected value of human error is 0.266/10000

16. 6. Maintenance Performance Model - conditioning Number of years of experience = 3 expected value of human error increases 0.266/10000 -> 0.309/10000

17. 6. Maintenance Performance Model - conditioning Requiring at least 6 hours of sleep provides decrease of expected human error from 0.266/10000 to 0.152/10000 Moreover E(HE|WorkCond=1,Alert=6) = 0.248/10000 while E(HE|WorkCond=1)=0.398/10000

18. 7. Conclusions and recommendations Calculation of multivariate normal probabilities is not an easy task in case of high dimension; there is still need to develop more fast (and also accurate) algorithm for higher dimension Include Algorithm I in UniExp software; together with making UniExp to worked outside the Matlab environment Combining experts opinion to obtain better results Collect data describing to marginal distribution in Maintenance Performance Model Discover other possible influential factors in Maintenance Performance Model Any other propositions?

19. Questions ???

20. Additional Slides

21. A1. Covariance matrixes used in numerical tests Dimension 4 determinant = 0.5271 determinant = 0.1099

22. A1. Covariance matrixes used in numerical tests Dimension 7 determinant = 0.4890 determinant = 0.1102

23. A2. Determinant of covariance matrix as the measure of spread from distribution of spread from distribution Dimension 2 determinant = 1

24. A2. Determinant of covariance matrix as the measure of spread of distribution of spread of distribution Dimension 2 determinant = 0.51,  =0.714

25. A2. Determinant of covariance matrix as the measure of spread of distribution of spread of distribution Dimension 2 determinant = 0.0199,  =0.141

26. 3.2. NumericalComparison 3.2. Numerical Comparison Test 1 – identity covariance matrix dimension = 4 dimension = 7 T Ai – time of calculation for algorithm i P Ai – probability obtained by algorithm i E Ai – estimated error of approximation provided by algorithm i 700, 1500 – number of quasi random point in Alg I; 10 -5 requested accuracy for Alg II

27. Test 2 – covariance matrix with determinant  0.5 dimension = 4, determinant = 0.5271 dimension = 7, determinant = 0.489 3. NumericalComparison 3. Numerical Comparison

28. Test 3 – covariance matrix with determinant  0.1 dimension = 4, determinant = 0.1099 dimension = 7, determinant = 0.1102 3. NumericalComparison 3. Numerical Comparison

29. -Motivation – - part of CATS model - build to describe the causal factors influencing the maintenance crew Methodology - non-parametric BBNQuantification - Nodes – variables which can influence the human performance among the maintenance crew; marginal distribution – data or Classical Method (Expert Judgment) - Conditional rank correlations – obtained from experts through the dependence probabilities of exceedance 5. Maintenance Performance Model

30. 5. Maintenance Performance Model – model variables VariableDefinition Source of marginal distribution 1. Job 1. Job Trainings average number of training per year Expert judgment 2. Alertnessaverage number of hours an aircraft mechanic sleeps of per dayData 3. Communicationcurrent information transfer procedure in use, distinguishing: 1. only paper notes, 2. paper notes with oral feedback Expert judgment 4. Experienceaverage number of years a person worked as aircraft mechanicData 6. Aircraft Generationaircraft generation in scale from 1 to 4 where 4 is the most recent generation Data 5. Working Conditions average number of maintenance operations needed to be performed 1.out-side / 2. inside the hangar per 10,000 maintenance operations Expert Judgment 7. Human Error number of maintenance human errors that might lead to hazardous situations per 10,000 maintenance tasks Expert Judgment