Integrating Ethics into Graduate Training in the Environment Sciences Series Unit 3: Ethical Aspects of Data Analysis AUTHOR: KLAUS KELLER and LOUISE MILITCH Department of Geosciences The Pennsylvania State University With input from Nancy Tuana, Ken Davis, Jim Shortle, Michelle Stickler, Don Brown, and Erich Schienke

–What are potential ethical questions arising in data analysis? –What are the “rules of the game”? –Do research publications follow these rules? –Where to go for guidance? 2 Guiding Questions

What are potential ethical questions arising in data analysis? What are the impacts of potential errors in the data analysis result on the outcome of decision? –Type I error –Type II error –Overconfident projections –Biased projections How to deal with the illusion of objectivity? How to communicate potential overconfidence? How to formulate the null-hypothesis? When is the analysis “done” and ready for submission? What to do if the data are insufficient for a formal and robust hypothesis test? 3

What can go wrong while testing an hypothesis? Type 1 error: –Effect is noise but we assign significant connection. –Null-hypothesis is rejected, when it is actually true –“False positive”. –Scientists typically design statistical tests with a low probability of a type 1 error (e.g., “p < 0.05”). Type 2 error: –Effect is real, but we do not assign a significant connection. –Null-hypothesis is accepted, when it is actually false. –“False negative”. Optimal (or Bayesian) decision theory –Design the strategy based on the relative costs of Type I and Type II errors. –Example: A hurricane is predicted to arrive in Miami with p=0.2. Should you take action? –Maximize the utility of the decision consistent with your posterior. 4

–What are potential ethical questions arising in data analysis? –What are the “rules of the game”? –Do research publications follow these rules? –Where to go for guidance? 5 Guiding Questions

American Statistical Association Ethical Guidelines for Statistical Practice “Statisticians should: present their findings and interpretations honestly and objectively; avoid untrue, deceptive, or undocumented statements; disclose any financial or other interests that may affect, or appear to affect, their professional statements.” 6

American Statistical Association Ethical Guidelines for Statistical Practice “ Statisticians should: delineate the boundaries of the inquiry as well as the boundaries of the statistical inferences which can be derived from it; emphasize that statistical analysis may be an essential component of an inquiry and should be acknowledged in the same manner as other essential components; be prepared to document data sources used in an inquiry, known inaccuracies in the data, and steps taken to correct or refine the data, statistical procedures applied to the data, and the assumptions required for their application; make the data available for analysis by other responsible parties with appropriate safeguards for privacy concerns; recognize that the selection of a statistical procedure may to some extent be a matter of judgment and that other statisticians may select alternative procedures; direct any criticism of a statistical inquiry to the inquiry itself and not to the individuals conducting it”. 7

–What are potential ethical questions arising in data analysis? –What are the “rules of the game”? –Do research publications follow these rules? –Where to go for guidance? 8 Guiding Questions

What is overconfidence? Estimates with artificially tight confidence bounds are overconfident. Overconfidence in subjective assessments and model predictions is common. 9 Error in the recommended values for the electron mass Henrion and Fischhoff (1986) Year of publication

What are key sources of overconfidence? Neglecting autocorrelation effects. Undersampling the unresolved variability (i.e., out-of-range projections). Assuming unimodal probability density functions. Neglecting model representation errors. Considering only a subset of the parametric uncertainty. Neglecting structural model uncertainty. 10 Are current climate projections overconfident?

11 Morita et al (2001) The fact that the range of CO 2 emission projections have widened over time is consistent with the hypothesis that previous projections have been overconfident. “The 40 scenarios cover the full range of GHG [..] emissions consistent with the underlying range of driving forces from scenario literature” [Nakicenovic et al, 2000, p.46].

A proof of concept: Probabilistic hindcasts and projections of CO 2 emissions through Bayesian data assimilation. Assimilates historic observations of population size, economic output, and CO 2 emissions over the past three centuries. Logistic population model interacts with a Solow economic growth model and a simple logistic model of technological change. We account for model error, as well as autocorrelation, and estimate the full, nonconvex pdf of the joint (17 dimensional) parameter pdf. This parsimonious model shows considerable hindcasting skill. Keller et al. (2007)

Past climate projections are likely overconfident as published CO 2 emission scenarios miss potentially important tails of fully probabilistic projections.

Pictures: Rahmstorf (1997, left) and Stocker et al (2001, right, modified) The North Atlantic meridional overturning circulation (MOC) may collapse in a threshold response to anthropogenic forcing. Why are projections of threshold responses vulnerable to overconfident forcing estimates?

When might overconfidence result in biased decision-analyses? 15 Designing risk management strategies in the face of threshold responses requires sound probabilistic information. Overconfident climate projection may underestimate the risks of low-probability high impact events.

–What are potential ethical questions arising in data analysis? –What are the “rules of the game”? –Do research publications follow these rules? –Where to go for guidance? 16 Guiding Questions

Where to go for guidance? ASA Ethical Guidelines for Statistical Practice, published by the American Statistical Association: The Online Ethics Center for Engineering and Science: Your mentors and peers. 17

Discussion Questions / Checklist Should one submit a manuscript that may well be wrong and that could detrimentally affect the policy process? –How do you define “detrimental”? When and how is it appropriate to exclude “data outliers”? Are the potential sources of biases clearly flagged Is the sensitivity to the choice of analyzed data sufficiently explained? Does the discussion adopt a specific value judgment about what is a “significant” result? Are there ethical issues in performing a “classic” “p<0.05” hypothesis test? 18

Reading Materials Keller, K., Miltich, L.I., Robinson, A. and Tol, R.S.J.: 2007, 'How overconfident are current projections of carbon dioxide emissions?' Working Paper Series, Research Unit Sustainability and Global Change, Hamburg University. FNU-124, Berger, J. O., and D. A. Berry Statistical-Analysis and the Illusion of Objectivity. American Scientist 76 (2): Cohen, J The Earth Is Round (P-Less-Than.05). American Psychologist 49 (12): Lipton, P Testing hypotheses: Prediction and prejudice. Science 307 (5707):