1. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N You need how many runs?! Michael F. Wehner Lawrence Berkeley National Laboratory mfwehner@lbl.gov

2. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs should we make?  The answer to this question has always been:  As many as you can afford.  A more quantitative reply is possible if the question is more specific.  How many realizations are necessary to know the mean value of a field to within a specified tolerance and statistical certainty?  Or  How many realizations are necessary to know that differences between models are statistically significant?

3. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs?  Q. What is the minimum number of realizations (n) required to estimate model mean output within a specified tolerance (E) and statistical confidence (  )?  A. For a Gaussian distributed random variable:  s 2 =sample variance,   =population variance  N=Number of available realizations  Z and  are properties of the Gaussian function and 

4. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs?  The number of runs required depends on:  Which fields are deemed important.  How well defined they need to be. Statistical certainty Tolerance  What scale is needed. Temporal Spatial  The internal variability of the model.

5. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Chickens and Eggs  But how can we use this formula to estimate ensemble size before we perform the integrations?  How to estimate ensemble variance?  If N=20,  =95% then 0.58s 2 <  2 <2.1s 2  The answer lies in postulating ergodicity of the climate system.  For example, the modeled system is considered ergodic if the inter-realization variance of the mean of each decade from an ensemble of transient runs is statistically identically to the variance of the decadal mean from a long stationary control run.  If the model is ergodic, we can use the control run sample variance estimate in the equation for n.

6. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Is the modeled climate ergodic on decadal time scales?  Nine transient runs (20c3m)  N=9; 0.45s 2 <  2 <3.7s 2  500 years of control run 2 (picntrl)  N=50; 0.69s 2 <  2 <1.5s 2  F-test at 90% confidence  No significant difference

7. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Is the modeled climate ergodic?  Decadal mean annual surface air temperature  E=0.5K,  =95%  Centered pattern correlation = 0.95 Control Run

8. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Is the modeled climate ergodic?  Decadal mean annual precipitation  E = 10% of the mean value,  =95%  Centered pattern correlation = 0.96 Control Run

9. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Strong seasonal dependence  Decadal mean seasonal surface air temperature  E=0.5K,  =95% DJFJJA

10. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N Strong seasonal dependence  Decadal mean seasonal precipitation  E = 10% of the mean value,  =95% DJFJJA

11. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N What about interannual time scales?  Pretty hopeless to determine an annual or seasonal mean at these accuracies for single gridpoints.  Either relax the accuracy or spatially average.

12. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N What about interannual time scales?

13. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N

16. Other considerations.  Double these estimates to perform differences between scenarios to the same accuracies.  Extreme events  ~10 to estimate 20 year return value of annual daily extrema  Pair control runs with transient runs  A clever way to account for drift and/or initialize.  Doubles the number of runs.  The variability of the new model may be different than the current model.  PCM variability is considerably larger than CCSM3.0

17. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N How many runs?  In the absence of a clearly defined set of specifications:  The final answer …

18. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N  Remains  As many as you can!