1. 3D-Var Revisited and Quality Control of Surface Temperature Data Xiaolei Zou Department of Meteorology Florida State University zou@met.fsu.edu June 11, 2009

2. Outline 3D-Var Formulation 3D-Var Formulation Statistical Formulation Statistical Formulation Analysis Analysis Practical Applications Practical Applications Part I: Motivation Motivation EOF analysis EOF analysis QC for T s QC for T s Part II:

3. Part I 3D-Var Revisited

4. Facts 1)All background fields, observation operators and observations have errors. 2) There is no truth. Errors in background, observation operator and observations can only be estimated approximately. Produce the best analysis by combining all available information. The Goal

5. Questions 1) What is the measure of the best analysis? 2) How to combine all available information?

6. Variational Formulation A scalar cost function is defined: where

7. Statistical Formulation Available information PDF Write the PDFs for all three sources of information as: Joint PDF: PDF of the a posteriori state of information

8. The Bayes Theorem The marginal PDF of the a posteriori state of information: is the PDF of the a posteriori state of information in model space.

9. Application of Bayes Theorem to Data Assimilation Data assimilation derives some features of the PDF, which is the a posteriori state of information in model space. The maximum likelihood estimate ~ analysis The covariance matrix of this estimate ~ analysis error covariance A

10. Assuming All Errors Are Gaussian, The PDF for y obs : The PDF x b : The PDF for H(x 0 ):

11. Bayes Estimate Under Gaussian Assumptions

12. Maximum Likelihood Estimate MaximizingMinimizing The PDF of the a posteriori state of information in model space: Statistical EstimateVariational Calculus

13. Gaussian and Non-Gaussian signals The signals are sampled at 10000 points. PDFs are constructed at an interval of

14. Gaussian and Non-Gaussian signals

15. 3D-Var & 3D-Var Analysis The 3D-Var data assimilation solves a general inverse problem using the maximum likelihood estimate under the assumptions that all errors are Gaussian. The 3D-Var analysis is the maximum likelihood estimate if all errors are Gaussian.

16. Zero Gradient: A necessary Condition a linear operator a nonlinear operator H

17. Analytical Expression of Solution with a Linear Model H is linear:

18. Analytical Expression of Solution with an Approximate Linear Model

19. Analysis Error When linear approximation is valid, When linear approximation is valid, the a posteriori PDF is approximately Gaussian, with the analysis as its mean and the following covariance matrix:

20. 3D-Var Analysis A -1 is referred to as an information content matrix. When the analysis error is small, the value of ||A -1 || is large, the information content is large. The information content of the 3D-Var analysis is greater than the information content in either the background field or the observations that were assimilated.

21. 3D-Var Practice Develop System Decision on variables and resolutions Estimate of background error covariance Assimilate Data Decision on observations to be assimilated Understanding of the observations Estimate of observation errors Comparison between observations and background Development of the observation operator Estimate of model errors Obtain Solution Minimization (preconditioning, scaling) Advanced computing (parallelization, data intensive computing platforms)

22. What does 3D-Var data assimilation involve? Choice of analysis variable What data to assimilation? Which model to use? What background to start with? How to estimate elements in B? Where to find their values? How to quantify it? Model SpaceObserved Space 3D-Var analysis +

23. What need to be done before and after conducting 3D-Var experiments? 3D-Var Input Data Output Analysis Quality Control Diagnosis of Analysis

24. What need to be done before and after conducting 3D-Var experiments? Quality Control Knowing the data Knowing the major difference between data and background field Remove errorneous data Eliminate data that render errors non-Gaussian Diagnosis of 3D-Var analyses Check the convergence Examine the analysis increments Estimate analysis errors Assess forecast impact Provide physical and dynamical explanations to the numerical results one obtains

25. When Working with Real-Data, The key things are Knowing the data before inputting them Knowing the data before inputting them into a 3D-Var system by a careful QC! into a 3D-Var system by a careful QC! Kowing the system after a 3D-Var Kowing the system after a 3D-Var experiment by a careful analysis of experiment by a careful analysis of the 3D-Var results! the 3D-Var results!

26. Examing 3D-Var Results

27. Analysis - obs one-week average results qp uv

28. Differences between model and obs. before and after a 3D-Var experiment p b - p obs and p a -p obs

29. 29 Inferred from calculated and

30. Part II Quality Control of Surface Temperature Data

31. 31 Motivations Surface data are abundant Very little surface data are assimilated in operational systems Surface data are important to thunderstorm prediction Challenges Existing data assimilation systems have short or no memory of surface data Diurnal cycle dominants the variability of surface variability and is not described with sufficient accuracy in large-scale analysis which is used as background in mesoscale forecast Background errors are non-Gaussian

32. 32 A Total of 3197 Surface Stations The number of missing data at each station in January 2008 is indicated by color bar.

33. 33 Improving Surface Data Assimilation Key steps: 1)Inclusion of more surface data 2)Improved QC 3)Vertical interpolation based on the atmospheric structures within the boundary layer Surface layer Mixed layer 3) Incorporation of dynamic constraint

34. 34 EOF Modes for T s Constructed from Station Observations First Second Third Fourth Fifth Sixth

35. 35 EOF Modes for T s Constructed from Station Observations (cont.) Seventh Eighth Ninth Tenth

36. 36 Explained Variances Surface Data (blue) NCEP analysis (red)

37. 37 Principal Components (PCs)

38. 38 Principal Components (PCs)

39. 39 Dominant Oscillations in January 2008 Period (unit: day) Obs. NCEP EOF mode Period (unit: hour) Longer-period oscillation Diurnal oscillation Shorter-period oscillation

40. 40 Diurnal Oscillation

41. 41 Longer-Period Oscillations

42. 42 Diurnal Oscillation and Longer-Period Oscillations Phase difference Amplitude difference

43. 43 PC Differences between Surface Data and NCEP Analysis Second Third FourthFifth Sixth Time (unit: day) Blue line: First Week Red line: Last Week

44. 44 Frequency Distributions of Diurnal Cycle Modes First Week Third Fourth Fifth Sixth Frequency Second T obs -T NCEP (unit: K) Frequency Last WeekFirst Week Last Week Fourth Sixth T obs -T NCEP (unit: K) January 2008 Second Fourth Sixth Third Fifth

45. 45 Frequency Distributions (modes 2-6) First WeekLast Week Entire Month Frequency T obs -T NCEP (unit: K) Frequency Sum of Modes 2-6

46. 46 Statistical Measures MeanVariance KurtosisSkewness

47. 47 QC Procedure Step 1: 1)Historical extremum check  T > The average of NCEP analysis of each station pluses (minuses) 15-times its variance 2)Temporal consistency check  T > 50 ℃ in 24-hours interval 3)Bi-weight check Z-score > 3 4)Spatial consistency check T > The average of linear fit to highly correlated stations pluses (minuses) 4-times its variance

48. 48 QC Procedure (cont.) Step 2: The Z-score of the difference between station observation and background field must less than 4 Step 3: The Z-score of the difference between station observation and background field excluding the contribution from diurnal cycle must less than 2

49. 49 Step 2 Step 3 Background Obs. Background

50. 50 Frequency Distribution before and after QC First WeekLast Week Entire Month T obs -T NCEP (unit: K) Frequency

51. 51 Frequency Distribution with and without Contribution from Modes 2-6 First WeekLast Week Entire Month T obs -T NCEP (unit: K) Frequency

52. 52 Correlations and RMS Differences of the PCs before and after QC

53. 53 Data Number Removed at Each Station Step One Step Two Step Three All Three QC Steps

54. 54 Percentage of Data Removed by QC Step 1 Steps 1-2 Steps1- 3

55. 55 Percentage of Data Removed by QC Time (day)

56. 56 Variation of the Statistical Measures with QC Steps Mean (unit: K) Std. (unit: K) Skewness Kurtosis Step 1Step 2Step 3 Step 1Step 2Step 3 Ori.No DC Ori.No DC

57. 57 Time Evolution of Standard Deviation before and after QC Std. (K) Time (unit: day)

58. 58 Time-Zone Dependence of Diurnal Oscillation Time (unit: hour) Temperature Weekly mean T s at seven surface stations selected within different time zones: Zone 1: 55.03E, 36.42N Zone 2: 65.68E, 40.55N Zone 3: 82.78E, 41.23N Zone 4: 98.9E, 40.0N Zone 5: 110.05, 41.03 Zone 6: 128.15E, 40.89N Zone 7: 141.17E, 39.7N Surface Obs. NCEP Analysis

59. 59 Average Time at Which T s Reached the Maximum in the First Week of January 2008 Time (UTC)

60. 60 Global Diurnal Cycle NCEPECMWF (ERA-Interim) Surface Observations Time (UTC) Temperature (K) Time (UTC) January 1-7, 2008

61. Summary Diurnal cycle dominates the temporal variability of surface data Large-scale analysis contains a significant phase error (~10-85 degrees) of the diurnal cycle A three-step QC procedure is developed to identify outliers in surface-station temperature data which have a non-Gaussian frequency distribution

62. More details can be found in Qin, Z.-K., X. Zou, G. Li and X.-L. Ma, 2009: Quality control of surface temperature data with non-Gaussian background errors. Quart. J. Roy. Meteor. Soc., Submitted. Zou, X. and Qin, Z.-K., 2009: Diurnal cycle in global analysis. J. Geo. Letter., to be submitted.