1. RTOFS Monitoring and Evaluation Metrics Avichal Mehra MMAB/EMC/NCEP/NWS

2. Focus on global and basin-wide domains Focus on global and basin-wide domains Adopt standard definitions and community- wide formats Adopt standard definitions and community- wide formats A good starting point – GODAE validations A good starting point – GODAE validations ( Oceanography, Vol. 22, No. 3: 128-143) ( Oceanography, Vol. 22, No. 3: 128-143) -- based on EU MERSEA Integrated project (2004-2008). (2004-2008). Methodology

3. Methodology Use real time data Use real time data Design diagnostics based on established science Design diagnostics based on established science Evaluate system performance and product quality Evaluate system performance and product quality Uniformity of assessment across operational centers Uniformity of assessment across operational centers Facilitate inter-comparison of operational systems (Global and Basin-wide) Facilitate inter-comparison of operational systems (Global and Basin-wide)

4. According to Le Provost et al (2002) (reference in notes), three types of metrics are considered: “ Consistency: verifying that the system outputs are consistent with the current knowledge of the ocean circulation and climatologies Quality: quantifying the differences between the system “best results” (analysis) and the sea truth, as estimated from observations, preferably using independent observations (not assimilated). Performance (or accuracy): quantifying the short term forecast capacity of each system, i.e. answering the questions “do we perform better than persistency? better than climatology?… “ (adapted from Le Provost et al, presentation at GOADE, 2004) Types of Metrics

5. CLASS 1: 2D horizontal fields; T, S, U, V, and W CLASS 1: 2D horizontal fields; T, S, U, V, and W at fixed depths; SSH, MLD. CLASS 2: (T,S,U,V) vertical sections/moorings. CLASS 2: (T,S,U,V) vertical sections/moorings. CLASS 3: Integrated quantities (transports etc.). CLASS 3: Integrated quantities (transports etc.). CLASS 4: Test performance of analysis and forecasts. CLASS 4: Test performance of analysis and forecasts. Classes of products

6. Next: Examples presented here for the Atlantic Basin RTOFS system. The same approach will be applied to the future operational Global RTOFS system. Also leverage US Navy’s ongoing evaluation of its Global HYCOM system (Hurlburt et al., 2009; Smedstad et al., 2009).

7. CONSISTENCY METRICS Class 1: Mean potential temperatures at 200m depth compared to NCEP climatology Class 2: Comparisons with WOCE sections Class 2: Vertical velocity sections compared with other models (example: at 27 N) Class 3: Transports at various sections (examples follow)

8. Consistency, class 1: Mean potential temperature at 200m depth for RTOFS-Atlantic (shown at right) maintain the large scale patterns in the climatology shown on the left.

9. Consistency, class 2: Potential Temperature at A20 WOCE section: A meridional section across the Atlantic (as shown in the inset). This should be compared to the section on the next slide.

10. Consistency, class 2: The 18 degree tropical mode water pool and bottom temperatures (2 degrees) in the deep (~ 4000 m depth) are maintained in RTOFS-Atlantic.

11. Consistency, class 2: Salinity at A20 WOCE section: A meridional section across the Atlantic (as shown in the inset). This should be compared to sections on the next slide.

12. Consistency, class 2: Intermediate and the deep salinity distribution in RTOFS-Atlantic compares well with WOCE.

13. TOPAZ MERTOPFOAMHYCOM-US MERCATOR FOAMUS-HYCOM Consistency, class 2: Results from four other models showing the location and strength of DWBC at 27 N: mean (top) and standard deviations (bottom). This should be compared to the section on the next slide.

14. Consistency, class 2: Meridional section from RTOFS Atlantic showing location and strength of DWBC at 27 N. RTOFS results compare well with those of other models.

15. Consistency, class 3: Location of Sections for comparing net transports with historical and real time measurements

16. Consistency, class 3: Transports across Florida to Bahamas from RTOFS-Atlantic (in blue) compared to cable data (in red). The black line is a 25 hr filter. Model transports correlate well with cable data.

17. QUALITY METRICS Class 1: Surface Salinity Fronts near rivers (example Mississippi River ) Class 2: Comparisons of T,S profiles with CTD data Some examples are shown comparing potential temperature and salinity. Class 2: Mean Path of the Gulf Stream based on SSH (with no data assimilation).

18. Nowcast for 20100405 Quality, Class 1: Surface Salinity map for RTOFS-Atlantic (right panel) compared to surface salinity map near the mouth of Mississippi river based on conductivity sensors and current meters data (left panel) collected from moorings near the LATEX coast in 1982 (Estuaries, Wiseman & Kelly, 1994). The offshore salinity front in RTOFS is weaker than the one observed and is located closer to the coast.

19. Location: Gulf Stream region POTENTIAL TEMP SALINITY Quality, Performance, class 2: RTOFS-Atlantic (prod, para) compared to a CTD profile (obs) and climatology (clim). RTOFS para is with SSH assimilation.

20. Location: Near Azores (eastern Atlantic) POTENTIAL TEMPSALINITY Quality, Performance, class 2: RTOFS-Atlantic (prod, para) compared to a CTD profile (obs) and climatology (clim). RTOFS para is with SSH assimilation.

21. Location: Sargasso Sea (middle Atlantic) POTENTIAL TEMP SALINITY Quality, Performance, class 2: RTOFS-Atlantic (prod, para) compared to a CTD profile (obs) and climatology (clim). RTOFS para is with SSH assimilation.

22. Quality, class 2: The above is from a free RTOFS Atlantic run without any data assimilation.

23. PERFORMANCE METRICS Class 1: Gulf Stream North Wall location Class 4: SST bias, mean and anomaly correlation Class 4: SST Forecast, Nowcast anomaly correlation and error

24. Performance, class 1: North Wall of the Gulf Stream from RTOFS-Atlantic (in magenta), Navy Analysis (in black) superposed on model SSH. The Navy analysis is based on remotely sensed SST data.

25. Performance, class 4: SST error statistics from RTOFS Atlantic using in-situ data which was not assimilated. These can be used to compare and contrast different operational systems.

26. Performance, class 4: SST Forecast, Nowcast anomaly correlation and error from RTOFS Atlantic. In this simulation, an error in the assimilation of SST was removed on day 100.