1. Statistics, data, and deterministic models NRCSE

2. Some issues in model assessment Spatiotemporal misalignment Grid boxes vs observations Types of error Measurement error and bias Model error Approximation error Manipulate data or model output? Two case studies: SARMAP – kriging MODELS-3 – Bayesian melding Other uses of Bayesian hierarchical models

3. Assessing the SARMAP model 60 days of hourly observations at 32 sites in Sacramento region Hourly model runs for three “episodes”

4. Task Estimate from data the ozone level at x’s in a grid square. Use sum to estimate integral over grid square. Issues: Transformation Diurnal cycle Temporal dependence Spatial dependence Space-time interaction

5. Transformation Heterogeneous variability–mean and variance positively related Square root transformation All modeling now on square root scale– approximately normal

6. Diurnal cycle

7. Temporal dependence

8. Spatial dependence

9. Estimating a grid square average Estimate using (not averages of squares of kriging estimates on the square root scale)

10. Looking at an episode

11. Afternoon comparison

12. Nighttime comparison

13. A Bayesian approach SARMAP study spatially data rich If spatially sparse data, how estimate grid squares? P =  Z + M + A +  O = (  )Z + B + E P = process model output O = observations Z = truth Calculate (Z | P,O) for prediction Calculate (O | P,  = 1, M = S = A = 0) for model assessment

14. CASTNet and Models-3 CASTNet is a dry deposition network Models-3 sophisticated air quality model Average fluxes on 36x36 km 2 grid Weekly data and hourly output

15. Estimated model bias The multiplicative bias  is taken spatially constant (= 0.5). The additive bias E(M+A+  ) is spatially distributed.

16. Assessing model fit Predict CASTNet observation O i from posterior mean of prediction using Models-3 output P i and remaining observations O -I. Average length of 90% credible intervals is 7 ppb Average length using only Models-3 is 3.5 ppb

17. Crossvalidation

18. Combining deterministic models Ensemble forecasts: Combining weather forecasts using different estimates of the current state of the atmosphere Combining weather forecasts using different forecast models Used as a way to estimate model uncertainty

19. Wind forecasts Eight models with same forecast model and different initial conditions.

20. Bayesian model averaging The forecast f k of a weather quantity y has density g k (y|f k ) Combine using weights w k, summing to 1: Weights are empirically determined by each forecast’s performance in the training period. Training period is a sliding window (so reestimate parameters and weights over time)

21. Calibration assessment If a forecast is well calibrated, the rank of each observation (e.g. max wind speed) should be uniformly distributed over the different forecasts.

22. Forecast 95th %ile median

23. The Bayesian hierarchical approach Three levels of modelling: Data model: f(data | process, parameters) Process model: f(process | parameters) Parameter model: f(parameters) Use Bayes’ theorem to compute posterior f(process, parameters | data)

24. Some applications Data assimilation Satellite tracking Precipitation measurement Combination of data on different scales Image analysis Agricultural field trials

25. Application to Models-3 where  (I) are samples from the posterior distribution of  MEILNCINFLMI CASTNet0.153.290.903.140.571.02 Models-30.333.335.329.590.521.04 Adjusted Models-3 0.122.881.093.120.441.01

26. Predictions

27. NCAR-GSP (IMAGe)