How to Measure the Progress and Value of NMME? Issues to Address: Need to define how to measure the value of NMME. Besides CPC, who are other users of NMME? What is the process to engage users? Presentation from Colorado Basin River Forecast Center (A. Wood) Need to develop a longer-term strategy to measure the progress and value.

How to Measure the Progress and Value of NMME? To answer these questions need to define target audience(s)! I would argue there are three dominant audiences each with its own metrics relative to progress and value of NMME: 1. Science research community: New multi-model ensemble and correction methods Improved global models 2. Forecast skill community: Skill metrics such as RMSE, ACC, Briar-Skill Score, reliability diagrams, ROC, RPSS, etc. 3. Customers who will make use of the forecasts: Do NMME forecasts allow me to make decisions that will ?: Make money Save lives, avoid disasters Build more robust infrastructure Many other examples

How to Measure the Progress and Value of NMME? Experience suggests that the more difficult part of the task for climate scientists will be audience 3, the customers (stakeholders, users). Many reasons for this: 1. Customers are heterogeneous. One size may not fit all. 2. Customers have different backgrounds and aptitudes. 3. Forecast products we might think are great may not be very useful for particular users. How to overcome these barriers: Find out what customers wants/needs are and what knowledge gaps exist for them to use forecasts. Provide educational support and simple, flexible web-based tools. Conduct a regular (annual) survey: NWS already doing this. A survey provides regular feedback from customers on strengths and weaknesses of product (NMME).

Understanding User Needs for Climate Information NWS Customer Survey Excerpted analysis from: "Customer Satisfaction with NOAA’s National Weather Service Climate Products and Services”, Meyers, J., M. Timofeyeva, M. Hawkins, and S. Dixon. NOAA CPC CDPW, Raleigh, North Caroline, October Customer perception of accuracy and reliability was found to have the higher impact on their overall rating of CPC. Improving the customer’s understanding of the accuracy of products by providing verification information, clarification of the information content of products, and when possible, improvements in skill should be priorities. 2. Respondents using NWS climate products for personal reasons are most satisfied; the more technical the user (i.e. those in academia, financial markets) the more critical the response.

Understanding User Needs for Climate Information NWS Customer Survey 3. NWS will continue regular climate services customer satisfaction surveys to ensure ongoing improvement in products and services based on customer input. 4. Results and analysis will motivate development of new products and services (more than 85% of respondents want additional local climate products). 5. Survey helped identify critical “high-impact” areas for priority improvement of climate services.

Understanding User Needs for Climate Information NWS Customer Survey 1433 Responses (6 NWS Regions and international users) Breakdown of users by sector: Public: 47% Academia: 10% Government (non-NOAA): 10% NOAA (non-NWS): 3% NWS: 9% Commercial: 8% Other: 12%

Customer Metrics for Progress and Value of NMME 1. Web-analytics Number of views, size of data downloads. 2. Memorandum of understanding, web acknowledgement of use of product, use of NMME/NWS/NCS logo on products. 3. Written acknowledgement of use of NMME in annual reports, inputs to NWS survey, papers, etc. 4. NWS survey on customer satisfaction indices. Progress can be tracked by relative satisfaction from year to year. These can be evaluated on a regular (annual) basis to assess progress.

Who are potential customers for NMME products? Groups that currently use CPC’s ISI-SI forecasts. This likely includes: 1. Other parts of NOAA: RISA’s, River Forecast Centers, NIDIS, CTB. 2. Other federal agencies: DOI, Army Corps of Engineers, USAID. 3. State and local emergency planning agencies. 4. Reservoir managers and farmers. 5. Private companies: agriculture, insurance. 6. Humanitarian Relief Agencies: IFRC, WFP, FEWS-NET. 7. Developing world governments. 8. WMO: Global Framework for Climate Service, GPC. 9. Groups working in the developing world to support sectoral projects that use climate information: IRI, IASCLIP.