Quantification of the non- parametric continuous BBNs with expert judgment Iwona Jagielska Msc. Applied Mathematics

Outline of the presentation I PART 1. Introduction II PART 2. Method of eliciting conditional rank correlations 3. Comparison of algorithms to calculate multivariate normal probabilities 4. Presentation of elicitation software UniExp III PART 5. Building the Maintenance Performance Model Model variables Dependence relation 6. Results 7. Conclusions and recommendations I PART 1. Introduction II PART 2. Method of eliciting conditional rank correlations 3. Comparison of algorithms to calculate multivariate normal probabilities 4. Presentation of elicitation software UniExp III PART 5. Building the Maintenance Performance Model Model variables Dependence relation 6. Results 7. Conclusions and recommendations

CATS – casual model for Air Transport Safety – motivation and purpose - three sectors of human performance ATC Model, Flight Crew Performance Model Maintenance Performance Model 1. Introduction

2. Way of assessing dependence relations Conditional Rank correlations  Conditional probabilities of exceedence Why normal copula? - Advantages known relation between partial and rank correlation equal conditional and partial correlations possess zero independent property - Disadvantages no analytical form for multivariate cumulative distribution function P 1 = P ( F X4 (X 4 ) > q | F X3 (X 3 ) > q ) “Suppose that the variable X 3 was observed above its q th quantile. What is the probability that also X 4 will be observed above its q th quantile? “

2. Way of assessing dependence relations To see the conditional probability as a function of rank correlation we integrate bivariate normal density over the given region. we can calculate relationship between rank correlation and conditional probability

P 2 = P ( F X4 (X 4 ) > q | F X3 (X 3 ) > q, F X2 (X 2 ) > q ) “ Suppose that not only variable X 3 but also X 2 was observed above their q th quantile. What is the probability that also X 4 will be observed above its q th quantile? ” To find the conditional probability we integrate trivariate normal density over the given region with covariance matrix. 2. Way of assessing dependence relations We assess the higher order conditional rank correlation in the similar way.

Proposed numerical integration methods: Algorithm I and II – by Genz - first we apply transformation to simplify integration region - later randomized quasi Monte Carlo method is used - different choice of quasi points - in algorithm I we specify number of points; algorithm II assign number of points, s.t. the requested accuracy is provided Algorithm III and IV - based on successive subdivisions of integration region, where each subdivision is used to provide a better approximation of the integrand - polynomial rule is used to approximate integrand on each subregions - error estimate – difference between two polynomial rules of different order - algorithm IV may involve some simplification routines (change of variables) 3.1. Algorithms to calculate multivariate normal probabilities probabilities

T Ai – time of calculation for algorithm i P Ai – probability obtained by algorithm i E Ai – estimated error of approximation provided by algorithm i 700, 1500 – number of quasi random point in Alg I; requested accuracy for Alg II 3.2. NumericalComparison 3.2. Numerical Comparison dimension = 4, determinant = dimension = 7, determinant = 0.489

Algorithms III and IV are unpractical for large scale applications since they require long time for numerical calculations - time for hypercube [0.5, inf] 7 is more than 700seconds - when the procedure of subdivision of integration region is applied, algorithm do not provide the total error In Algorithm I user needs to specify number of quasi random points used to calculation; there is no control of provided error of estimation; time of calculation depends on the number of points, not of covariance matrix Time of calculation for Algorithm II is sometimes grater than for Algorithm I; time depends on covariance matrix; number of quasi random points depends on requested accuracy of solution At this moment Algorithm II is used in the software UniExp as the most accurate one; Algorithm I also has future potencial for implementation. 3. NumericalComparison – brief summary 3. Numerical Comparison – brief summary

4. Software elicitation tool - UniExp 1 Step – input of nodes and connections

4. Software elicitation tool - UniExp 2 Step – elicitation of conditional rank correlations

4. Software elicitation tool - UniExp Values of Rank Correlations can be found in RankCorrelationValues.txt file

5. Maintenance Performance Model

Elicitation with single expert we asked – 4 questions about marginal distributions – classical method of expert judgment 7 questions about conditional probabilities of exceedance 5. Maintenance Performance Model – dependence relation All variables are negatively correlated with variable human error

5. Maintenance Performance Model At the bottom of each histogram the expectation and standard derivation are shown. Unconditional expected value of human error is 0.266/10000

6. Maintenance Performance Model - conditioning Number of years of experience = 3 expected value of human error increases 0.266/ > 0.309/10000

6. Maintenance Performance Model - conditioning Requiring at least 6 hours of sleep provides decrease of expected human error from 0.266/10000 to 0.152/10000 Moreover E(HE|WorkCond=1,Alert=6) = 0.248/10000 while E(HE|WorkCond=1)=0.398/10000

7. Conclusions and recommendations Calculation of multivariate normal probabilities is not an easy task in case of high dimension; there is still need to develop more fast (and also accurate) algorithm for higher dimension Include Algorithm I in UniExp software; together with making UniExp to worked outside the Matlab environment Combining experts opinion to obtain better results Collect data describing to marginal distribution in Maintenance Performance Model Discover other possible influential factors in Maintenance Performance Model Any other propositions?

Questions ???

Additional Slides

A1. Covariance matrixes used in numerical tests Dimension 4 determinant = determinant =

A1. Covariance matrixes used in numerical tests Dimension 7 determinant = determinant =

A2. Determinant of covariance matrix as the measure of spread from distribution of spread from distribution Dimension 2 determinant = 1

A2. Determinant of covariance matrix as the measure of spread of distribution of spread of distribution Dimension 2 determinant = 0.51,  =0.714

A2. Determinant of covariance matrix as the measure of spread of distribution of spread of distribution Dimension 2 determinant = ,  =0.141

3.2. NumericalComparison 3.2. Numerical Comparison Test 1 – identity covariance matrix dimension = 4 dimension = 7 T Ai – time of calculation for algorithm i P Ai – probability obtained by algorithm i E Ai – estimated error of approximation provided by algorithm i 700, 1500 – number of quasi random point in Alg I; requested accuracy for Alg II

Test 2 – covariance matrix with determinant  0.5 dimension = 4, determinant = dimension = 7, determinant = NumericalComparison 3. Numerical Comparison

Test 3 – covariance matrix with determinant  0.1 dimension = 4, determinant = dimension = 7, determinant = NumericalComparison 3. Numerical Comparison

-Motivation – - part of CATS model - build to describe the causal factors influencing the maintenance crew Methodology - non-parametric BBNQuantification - Nodes – variables which can influence the human performance among the maintenance crew; marginal distribution – data or Classical Method (Expert Judgment) - Conditional rank correlations – obtained from experts through the dependence probabilities of exceedance 5. Maintenance Performance Model

5. Maintenance Performance Model – model variables VariableDefinition Source of marginal distribution 1. Job 1. Job Trainings average number of training per year Expert judgment 2. Alertnessaverage number of hours an aircraft mechanic sleeps of per dayData 3. Communicationcurrent information transfer procedure in use, distinguishing: 1. only paper notes, 2. paper notes with oral feedback Expert judgment 4. Experienceaverage number of years a person worked as aircraft mechanicData 6. Aircraft Generationaircraft generation in scale from 1 to 4 where 4 is the most recent generation Data 5. Working Conditions average number of maintenance operations needed to be performed 1.out-side / 2. inside the hangar per 10,000 maintenance operations Expert Judgment 7. Human Error number of maintenance human errors that might lead to hazardous situations per 10,000 maintenance tasks Expert Judgment