1. CLIMATE SERVICE DIVISION / OCWWS / NWS L3MTO QC and Accuracy Marina Timofeyeva Contributors: Annette Hollingshead, Dave Unger and Andrea Bair

2. CLIMATE SERVICE DIVISION / OCWWS / NWS CPC: CD forecast RHQ: Collecting trouble tickets WFO: 1.Ensure link to the product 2.Secondary QC of the product 3.Local value added 4.Customer services and outreach LFPS: 1. 1.Make forecast 2. 2.Primary QC 3.Fix troubles WS: 1. 1.Post product on web 2.Fix Troubles General QC Flow Archive

3. CLIMATE SERVICE DIVISION / OCWWS / NWS L3MTO Primary QC Interrogate CPC site and check for new forecast data 1. National 2. Alaska & Hawaii Parse data: extract mean & standard deviation Import data into Splus Run Splus scripts Export data 1. 1.Parse Splus output 2. 2.Create XML formatted files Run QC scripts Push data to web team Pass Troubleshoot and fix problems Fail Early am ET (~ 7am- Noon) 6-8 hours 45 minutes

4. CLIMATE SERVICE DIVISION / OCWWS / NWS Secondary QC (WFOs) Erroneous Data Check Approve data Web & Functionality Error Check Conspicuous Data Check Send trouble ticket to WS Pass Fail Send trouble ticket to Regional HQ Start Fail Pass Section A Section B Section C Send trouble ticket to Regional HQ Send trouble ticket to Regional HQ Other Discrepancy: Detailed QC

5. CLIMATE SERVICE DIVISION / OCWWS / NWS WFO Technical Support Team   Who are they? Regional representatives available to help answer any L3MTO questions   What is their role? Answer customer and partner questions and inquiries that NWS personnel are unable to address (see Guideline doc 3.2.3) Provide technical expertise & guidance: - interpreting L3MTO - comparing L3MTO to CPC 3-Month Temperature Outlook   How to find your regional rep? Page 13 in the L3MTO Guidelines document

6. CLIMATE SERVICE DIVISION / OCWWS / NWS Web problem QC failure Station metadata L3MTO Frustration Content and methodology Contact Regional Technical Support Team WFO Quick Guide on Getting Help with L3MTO Check CSSA, if error report to Regional CSPM and RCPM l3mto.cms@noaa.gov Trouble Ticket Procedure

7. CLIMATE SERVICE DIVISION / OCWWS / NWS WFO L3MTO Problem Reporting Protocol ( ) Detailed QC Secondary QC Web QC Add Stations MetadataMethodology Web Team Annette Hollingshead 1) Add Station (if not in CMS): report to CSPM; 2) Metadata Errors: - If CSSA is up-to-date, then submit change to CSPM - If CSSA is NOT up-to-date submit change to CSPM and RCPM who will expedite CSSA change TT collectors Marina Timofeyeva Issue Trouble Ticket Report to CSPM / RCPM Technical Support Team Technical Support Teams Protocol in Guidelines document Compare to CSSADiscussion Inconsistencies, MethodologyTechnical Issues Buffer issues

8. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC http://ww2.wrh.noaa.gov/hsd/climate/L3MTO/ARC/index.php Section A allows checking short leads L3MTO “sanity” with current climate trends Section B allows checking L3MTO agreement with some of CPC tools Section C allows checking L3MTO presentation and methodology correctness Section D presents L3MTO long term performance evaluation

9. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

10. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

11. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

12. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

13. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC Season: MJJ A CPC forecast of 49% (or greater) BELOW NORMAL probability is required to produce a significant* forecast for BELOW Normal category at the station. This automatically reduces the forecast for ABOVE Normal category to 28.3% or less. A CPC forecast of 36% (or greater) ABOVE NORMAL probability is required to produce a significant* forecast for ABOVE Normal category at the station. This automatically reduces the forecast for BELOW Normal category to 28.3% or less.

14. CLIMATE SERVICE DIVISION / OCWWS / NWS L3MTO Verification L3MTO forecast archive Additional scores to coordinate with NWS verification practices Extended selection choices for years and leads Univ. of AZ verification tool expanded to include:

15. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC ARC section D verification intended for WFO use, it shows results of 3 verification methods: Heidke Skill Score verifies 3 categorical forecast (pie chart format), indicates % improvement over climatology Reliability Diagram verifies long term performance of individual %-tile of POE Continuous Rank Probability Skill Score verifies accuracy, skill and resolution of POE, the most stringent score of the 3

16. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC DJFJJA Most stations have correct forecasts 30% (or more) of the time, and have little variability between the 13 leads For all years analyzed together, shorter leads in the winter seasons have higher scores than the same in the warm seasons Larger spread between stations’ performance in the warm seasons than cold

17. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

18. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

19. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC For each station and forecast target season a reliability diagram was created to analyze each percentile’s performance. Confidence limits around the perfect forecast line, were drawn from a binomial distribution at 95% confidence. The forecast was compared with 1971-2000 climatology performance

20. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

21. CLIMATE SERVICE DIVISION / OCWWS / NWS QC Tool: L3MTO Analysis Resource Center - ARC

22. CLIMATE SERVICE DIVISION / OCWWS / NWS Verification Summary Using multiple verification scores in climate forecast evaluation provides the most comprehensive assessment of long term forecast performance Verification analysis identified locations and 3-month periods with satisfactory forecast performance, and their spatial and temporal variability Overall, for the majority of stations and seasons, the L3MTO indicates higher skill than the 1971-2000 climatology

23. CLIMATE SERVICE DIVISION / OCWWS / NWS Verification Summary Although L3MTO in general proves to be more skillful than 1971-2000 climatology, it is still important to identify station/season combinations where L3MTO does not have long term skill. At such stations/seasons a Verification Filter will be applied. The verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level that depends on number of independent forecast made for the stations. If the HSS is greater than passing, then the forecast is reported, if less, then 1971-2000 climatology is reported.

24. CLIMATE SERVICE DIVISION / OCWWS / NWS Verification Filter 5. CPC produces CD forecasts monthly 6. CD outlook together with bi- variate statistics reported from the regression analysis, station forecast is produced 7. Verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level. 8. Data distribution and display are provided at the L3MTO web site (6) Station Forecast Produced (7) Verification Filter (5) CPC Produces the CD Forecast Monthly Routine: Sufficient Score? Produce Forecast Data (8) Distribution and Display Yes Provide Climatology Data No

25. CLIMATE SERVICE DIVISION / OCWWS / NWS Additional Verification Information

26. CLIMATE SERVICE DIVISION / OCWWS / NWS Additional Verification Information

27. CLIMATE SERVICE DIVISION / OCWWS / NWS Additional Verification Information