11 Prior Distribution Elicitation for Generalized Linear and Piecewise-Linear Models Paul Garthwaite and Fadlalla Elfadaly Open University

22 Why piecewise-linear models? Initial motivation for this model came from the need to model ecologists’ opinion about the presence/absence of rare and endangered animals. (A good example where expert opinion is useful – the ecologists had sightings of rare species but the data was not from a sampling frame and hence hard to incorporate in a statistical analysis.) For most variables there was an optimum value for a species. E.g. too hot or too cold did not suit it; nor too wet or too dry, etc.

3

4 Sampling Model Logistic model: y = ln( p/(1-p)) = β 0 + β 1 x 1 + …+ β k x k. GLM: y = g(μ) = β 0 + β 1 x 1 + …+ β k x k. Strategy: Elicit quantiles of p or μ and transform the assessments to quantiles of y. Prior model: β ~ multivariate normal. Three software implementations of the method: Garthwaite (1998: Visual Basic) Kynn (2004: Pascal. Elicitor) Elfadaly, Jenkinson, Garthwaite and Laney (2007/9: JAVA) (The programs of Garthwaite and Kynn only handle logistic regression.)

55 Assessments at reference point Scene-setting questions determine the number of variables and factors, their ranges and also a reference point. The reference point is chosen by the expert and gives the origin of variables and the reference level of factors. For a continuous variable it is assumed that opinion about slopes on one side of the reference point is independent of opinion about slopes on the other side. With the methods of Garthwaite and Elfadaly et al., median, lower and upper quartiles of the response at the reference point are assessed.

66 Lower and upper quartiles have the advantage that they can be assessed by the method of bisection. L M U 25% 25% 25% 25% _________________________________

7 Elicitor is much more flexible. For assessing the median, some techniques that can be used with logistic regression are available to the expert: Visual aids such as a probability wheel can be used. Probabilities can be given by first stating a (large) sample size and then assessing the number in that sample with the characteristic of interest. Scales marked in odds or log-odds can also be used. For credible intervals, intervals other than 50% intervals can be specified and a form of fixed interval method is also advocated.

88 Median Assessments Medians are assessed for one covariate at a time. The expert is asked to assume that all other covariates are at their reference values and to consider how the response varies with the covariate of current interest. The expert clicks on a graph to draw a curve for covariates or a bar chart for factors. (This is a poor approach to designing experiments but has clear benefits when eliciting expert opinion.)

9

10 The number of knots does not seem crucial. Elicitor gives the option of fitting a linear or quadratic function to the medians. Garthwaite (1998) gave option of superimposing graphs to help improve the expert’s internal consistency across covariates. (In forming models we almost always adopt linear relationships as the building blocks. Elicited piecewise linear relationships could instead be used as the building blocks.)

11

12 Feedback Feedback is generally beneficial. Useful to display the median estimate at other design points, other than those points where all but one of the covariates are at their reference values. Mason (2008) used Elicitor to question an expert about non-random non-response in a longitudinal survey. Reference point was for best response-rate. Worst case setting of the covariates gave a response rate of only 1%. The expert revised his median assessments and the worst- case response-rate increased to 9%, which the expert still thought was too low. The response-rate rapidly diminishes as probabilities are multiplied. Intend adding this feedback option to the software.

13

14

15

16

17

18

19 Examples O’Leary et al. (2009a) give an example where two experts assessed the probability of presence/absence for the brush-tailed rock- wallaby using Elicitor. Only two covariates: (i) Aspect (northerly vs other) (ii) Slope (0 o - 90 o ). O’Leary et al. (2009b) also gives an example where presence/absence for this wallaby is assessed – this time by only one expert but using four different methods, with aspect as the only covariate. Data: presence at 41 sites and absence at 9 (rare species? pest?)

20 Assessments of the two experts (O’Leary et al., 2009a)

Classification rates of four methods (O’Leary et al., 2009b) MethodPredictedObserved present absent Elicitorpresent419 absent00 Map-methodpresent01 absent418 Questionnairepresent419 absent00 Classification tree present351 absent68

22 Kynn (2004) gives five case studies conducted during the development of Elicitor where ecologists used it to quantify their opinions about an endangered species. Two of the studies had sample data with which to evaluate models. Ground parrot 137 presences and 438 pseudo-absences. 80% of the data was used to fit models and 20% for testing. Two continuous covariates, a factor with three levels and a second factor with four levels. Three models were considered: (a)Assessed prior + data (b)“Relaxed” prior + data (relaxed: variances were multiplied by 10) (c) Classical logistic stepwise regression.

23 Classification rates for ground parrot (Kynn, 2005) Stepwise does best – presumably variable selection helps. It used just the two continuous variables. MethodPredictedObserved present absent Assessed prior + data present2816 absent148 Relaxed prior + data present2211 absent753 Frequentist stepwise present2710 absent254

24 Criteria for threshold: minimise

25 Stemmacantha (a thistle) 203 presences and 2741 absences. Same three models; 80% of the data for fitting & 20% for testing. Stemmacantha Ground Parrat

26 Classification rates for Stemmacantha (Kynn, 2005) Numbers are inconsistent, but there seems little to choose between the methods. MethodPredictedObserved present absent Assessed prior + data present3377 absent11457 Relaxed prior + data present3483 absent6461 Frequentist stepwise present3269 absent15468

27 Garthwaite (1998) and Garthwaite & Al-Awadhi (2006) also quantify the opinion of ecologists about rare species in Queensland. Central Government wanted State Government to estimate habitat distribution of rare and endangered species. Some sample data were gathered. The aim was to link the data, ecologists’ knowledge and a GIS database to relate the probability of presence/absence to a large number of covariates. Preliminary meeting with about a dozen ecologists indicated that non- linear relationships were needed to model their opinion (hence the piecewise linear models). Little bent-wing bat. (5 variables and 8 factors, giving 57 regression coefficients. Data: 42 presences in 375 sites.)

28 Plumed frogmouth. (7 variables, 3 factors; 58 parameters). Data: 31 presences in 324 sites. Powerful owl. (1 variable, 5 factors; 24 parameters). Data: 13 presences in 324 sites. Greater glider. (7 variables, 4 factors; 60 parameters). Data: 53 presences in 343 sites. Common bent-wing bat. (4 variables, 7 factors; 59 parameters). Data: 13 presences in 375 sites.

29 Various prior distributions were fitted to compensate for systematic biases in the expert’s assessments. 1.(β 0, β 1,…, β k ) multivariate normal. 2.β 0 diffuse, (β 1,…, β k ) ~ MVN(b, Σ). 3.θ, β 0 diffuse, (β 1,…, β k ) ~ MVN(θb, θ 2 Σ). 4.γ, θ, β 0 diffuse, (β 1,…, β k ) ~ MVN(θb, γΣ). Cross-validation: Repeatedly using 80% of the data for fitting and 20% for testing. Squared error loss was used to measure performance.

30 Little bent- wing bat Common b-w bat Plumed frogmouth Powerful Owl Greater glider Prior Prior Prior Prior Stepwise logistic Regression Prior: no data Prior 3 (constant term given diffuse prior and all coefficients multiplied by a constant) is the best for each animal – noticeably better for the plumed frogmouth. The prior with no data is comparable with stepwise regression except for the greater glider. There is quite limited data.

31 A second example: Air pollution in (Khaldiya) Kuwait City Khaldiya had a mobile laboratory station to monitor pollution for one year. Focus is on the probability of pollutants exceeding harmful threshold level. There are two permanent fixed laboratory stations: 5 km north-east and 5 km south-west of Khaldiya. Aim is to use the data and the opinion of two scientists to relate Khaldiya pollution to the permanent laboratories. Pollutants: SO 2, NO 2 and n-CH 4 (non-methane). Scientists quantified their opinions separately. Variables: pollution levels at the permanent labs, temperature, wind speed, humidity, height of the inversion line.

32 Expert A/ SO 2 Expert A/ NO 2 Expert B/ NO 2 Expert A/ n-CH 4 Expert B/ n-CH 4 Prior Prior Prior Prior Stepwise logistic Regression Prior: no data Non-methane: priors seem poor as priors + no data do much worse than other methods; stepwise logistic does better than using expert B’s prior but not expert A’s, especially with Prior 2. For SO 2 and NO 2, the prior’s seem better and prior + data does better than stepwise logistic regression. Prior 2 is perhaps the best.

33 (Not Kuwait City)

34 A medical application The UK National Health Service (NHS) initiated a study to estimate the benefits of current bowel cancer services in England and examine costs and benefits of alternative developments in service provision. ScHARR developed a treatment pathway model that gave the possible sequences of presentation, diagnosis, treatment and outcomes that could be followed by a patient with suspected colorectal (bowel) cancer. Available information supplied most of the required numbers but expert opinion filled in gaps. The resulting report states, “Owing to a lack of empirical evidence in a number of areas, several of the model parameter and details of the model structure were elicited from experts.”

35 For two quantities there were covariates. For these, the new version of the software was used to quantify consultants’ opinions. Choice of diagnostic test had level of fitness as a covariate. Choice of adjuvant chemotherapy had five covariates (mostly factors): age, tumor location, disease status, perforation/obstruction, and fitness for cytotoxic therapy. Results were validated where possible. Commenting on assessments about adjuvant chemotherapy the YHEC- ScHARR report notes that “The [pathways] model uses expert 1’s responses as part of a generalised linear model and is validated by expert 2’s responses.” The use of elicitation in the study is reported in Garthwaite, Chilcott, Jenkinson & Tappenden (2008).

36 Al-Awadhi & Garthwaite (2006). Computational statistics, 21, Garthwaite (1998). Quantifying expert opinion for modelling habitat distributions. Sustainable Forest Management Tech. Report, Queensland Depart. Natural Resources. Garthwaite & Al-Awadhi (2006). Tech. Report 06/07. Dept. Statistics, Open University. Garthwaite, Chilcott, Jenkinson & Tappenden (2008). Int. J. Technology assessment in Health Care, 24, Kynn (2005). Eliciting expert knowledge for Bayesian logistic regression in species habitat modelling in natural resources. PhD thesis. Queensland University of Technology. Mason (2008). Methodological developments for combining data. O’Leary, Choy, Kynn, Denham, Martin, Mengersem & Murray (2009a). Environmetrics, 20, O’Leary, Mengersem, Murray & Choy (2009b). Comparison of four expert elicitation methods. 18 th World IMACS/MODSIM Congress.

37

38