1. Tim Smyth and Jamie Shutler Assessment of analysis and forecast skill Assessment using satellite data

2. Rationale – what are we trying to achieve ? Original approaches and web portal. New metrics and approaches –Annual SST comparisons –Annual chl-a comparisons –Weekly chl-a comparisons ( 2 examples) Conclusions. Overview

3. Rationale Sat in too many conferences, squinting from the back, where models and satellite data were said to be “in good agreement”: –Highly dissatisfactory as is subjective; –Authors frequently play with scales to suit; –Not necessarily comparing like with like –SOLUTION: come up with comparison web site …

5. Advantages Immediately shows up where discrepancy between satellite and model; Gives potential for models to be improved over shorter time-cycles (need to discuss this at the workshop); The shame factor! Disadvantages Not necessarily comparing like with like; Only surface measurements Problems with cloud coverage and averaging; “What is truth?”

6. Some immediate results On short time-scales: poor agreement; –Spring bloom timing is crucial: time of dramatic changes. On seasonal time-scales: better agreement; (see Allen et al. (2008)); –Not surprising as our models are based on trying to predict things like seasonal succession. There is some model skill in predicting the likelihood of blooms. However still need to produce more useful metrics to judge different aspects of the model …

7. New metrics Towards determining reasons for differences: Principal component analysis (or EOF) of time series. Kappa coefficient –Method traditionally employed to look at land change. –Measure of ‘difference’ between two sets of data. Receiver operator curves (ROC). –Used extensively in pattern recognition. –Employed in precipitation analyses (Met Office). –Performance graphing technique. Wavelet decomposition method. –Employed in precipitation analyses (Met Office). –Spatial decomposition. –Provides a measure of skill and mean squared error.

8. New metrics General approach: Metrics are automatically generated every week. Comparing weekly composite data. All data mapped to same scale and domain. Common cloud masking (of composite). Satellite data spatially averaged to approximate spatial scale of model data. Two sets of results: Annual analyses (February to October) Weekly analyses

9. Annual : SST metric results Visual observations: spatially and temporally similar Satellite PC1 Satellite PC2 model PC1 model PC2 Satellite temporal weighting Model temporal weighting

10. Annual : SST metric results Visual observations: spatially and temporally similar Satellite PC1 Satellite PC2 model PC1 model PC2 Satellite temporal weighting Model temporal weighting

11. Annual : chlorophyll metric results Visual observations: spatially and temporally different Satellite PC1 Satellite PC2 model PC1 model PC2 Satellite temporal weighting Model temporal weighting

12. Weekly : 5 July chlorophyll metric results Satellite MRCS model %age difference 5 th July 2008 Visual observations: model data appear to contain a large bias some structures appear similar (changes in gradient) Same geophysical scale

13. Weekly : 5 July chlorophyll metric results Satellite MRCS model %age difference 5 th July 2008 Visual observations: model data appear to contain a large bias some structures appear similar (changes in gradient). Same geophysical scale Similar gradients (with bias)

14. Weekly : 5 July chlorophyll metric results Satellite MRCS model %age difference 5 th July 2008 Visual observations: model data appear to contain a large bias. some structures appear similar (changes in gradient). Same geophysical scale Similar gradients (with bias)

15. Weekly : 5 July chlorophyll metric results Kappa MSE Skill score ROC

16. Weekly : 5 July chlorophyll metric results Kappa ROC MSE Skill score Data diverge At ~0.5mg m -3 Errors at all scales across range of chlorophyll -ve skill score for high chlorophyll, particularly at lower spatial scales Better than random performance

17. Weekly : 11 October chlorophyll metric results Satellite MRCS model %age difference 11 th October 2008 Visual observations: mixture of similar and dissimilar structures model data inverse of satellite data in some regions.

18. Weekly : 11 October chlorophyll metric results Satellite MRCS model %age difference 11 th October 2008 Visual observations: mixture of similar and dissimilar structures model data inverse of satellite data in some regions. opposite responses

19. Weekly : 11 October chlorophyll metric results Satellite MRCS model %age difference 11 th October 2008 Visual observations: mixture of similar and dissimilar structures model data inverse of satellite data in some regions. Feature in model data not apparent in satellite

20. Weekly : 11 October chlorophyll metric results Kappa ROC MSE Skill score

21. Weekly : 11 October chlorophyll metric results Data diverge At ~0.1mg m -3 Performance closer to random Errors at all scales Across range of chlorophyll -ve skill score for high chlorophyll, at low spatial scales ROC Kappa MSE Skill score

22. Conclusions Presented operational framework for evaluating MRCS vs satellite data Apparent need to use multiple metric approach. Seasonal signal captured well by model SST. Chlorophyll comparisons show the biggest differences. Performance of model chlorophyll varies (when compared with satellite data). However, satellite chlorophyll signal in winter months is likely to be incorrect or biased (algorithm performance reduces during this time). Both datasets have uncertainties. Still not comparing like with like (e.g. temporal differences in input data, possible that binning the satellite data to match model data will improve results).