C O M P U T A T I O N A L R E S E A R C H D I V I S I O N You need how many runs?! Michael F. Wehner Lawrence Berkeley National Laboratory
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs should we make? The answer to this question has always been: As many as you can afford. A more quantitative reply is possible if the question is more specific. How many realizations are necessary to know the mean value of a field to within a specified tolerance and statistical certainty? Or How many realizations are necessary to know that differences between models are statistically significant?
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs? Q. What is the minimum number of realizations (n) required to estimate model mean output within a specified tolerance (E) and statistical confidence ( )? A. For a Gaussian distributed random variable: s 2 =sample variance, =population variance N=Number of available realizations Z and are properties of the Gaussian function and
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs? The number of runs required depends on: Which fields are deemed important. How well defined they need to be. Statistical certainty Tolerance What scale is needed. Temporal Spatial The internal variability of the model.
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Chickens and Eggs But how can we use this formula to estimate ensemble size before we perform the integrations? How to estimate ensemble variance? If N=20, =95% then 0.58s 2 < 2 <2.1s 2 The answer lies in postulating ergodicity of the climate system. For example, the modeled system is considered ergodic if the inter-realization variance of the mean of each decade from an ensemble of transient runs is statistically identically to the variance of the decadal mean from a long stationary control run. If the model is ergodic, we can use the control run sample variance estimate in the equation for n.
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Is the modeled climate ergodic on decadal time scales? Nine transient runs (20c3m) N=9; 0.45s 2 < 2 <3.7s 2 500 years of control run 2 (picntrl) N=50; 0.69s 2 < 2 <1.5s 2 F-test at 90% confidence No significant difference
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Is the modeled climate ergodic? Decadal mean annual surface air temperature E=0.5K, =95% Centered pattern correlation = 0.95 Control Run
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Is the modeled climate ergodic? Decadal mean annual precipitation E = 10% of the mean value, =95% Centered pattern correlation = 0.96 Control Run
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Strong seasonal dependence Decadal mean seasonal surface air temperature E=0.5K, =95% DJFJJA
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Strong seasonal dependence Decadal mean seasonal precipitation E = 10% of the mean value, =95% DJFJJA
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N What about interannual time scales? Pretty hopeless to determine an annual or seasonal mean at these accuracies for single gridpoints. Either relax the accuracy or spatially average.
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N What about interannual time scales?
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N
Other considerations. Double these estimates to perform differences between scenarios to the same accuracies. Extreme events ~10 to estimate 20 year return value of annual daily extrema Pair control runs with transient runs A clever way to account for drift and/or initialize. Doubles the number of runs. The variability of the new model may be different than the current model. PCM variability is considerably larger than CCSM3.0
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs? In the absence of a clearly defined set of specifications: The final answer …
C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Remains As many as you can!