1. CFI GROUP WORLDWIDE ANN ARBOR BEIJING LONDON MADRID MILAN PARIS SHANGHAI STOCKHOLM NASA Earth Observing System Data and Information System Customer Satisfaction Results November 29, 2011

2. 2 © CFI Group 2011 Today’s Discussion Background Overview Key Results Detailed Analysis Summary

3. 3 © CFI Group 2011 Background

4. 4 © CFI Group 2011 Project Background Objectives Measure customer satisfaction with the NASA Earth Observing System Data and Information System at a national level and for each Data Center –Alaska Satellite Facility Distributed Active Archive Center –Crustal Dynamics Data Information System –Global Hydrology Resource Center –Goddard Earth Sciences Data and Information Services Center –Land Processes Distributed Active Archive Center –MODAPS Level-1 Atmospheres Archive and Distribution System –NASA Langley Atmospheric Science Data Center –National Snow and Ice Data Center Distributed Active Archive Center –Oak Ridge National Laboratory Distributed Active Archive Center –Ocean Biology Processing Group –Physical Oceanography Distributed Active Archive Center Jet Propulsion Laboratories (JPL) –Socioeconomic Data and Applications Center Assess the trends in satisfaction with NASA EOSDIS specifically in the following key areas: –Product Search –Product Selection and Order –Delivery –Product Quality –Product Documentation –Customer Support Identify the key areas that NASA can leverage across the Data Centers to continuously improve its service to its users

5. 5 © CFI Group 2011 Project Background Measurement timetable Finalized questionnaireAugust 1, 2011 Data collection via web Sending invitations spanned the first two weeks. Sending reminders spanned the last two weeks. The survey was in the field for a longer time this year for resending invitations. September 12, 2011 – October 18, 2011 Topline resultsOctober 26, 2011 Results briefingNovember 29, 2011

6. 6 © CFI Group 2011 Project Background Data collection Respondents 3,996 responses were received 3,996 responses were used for modeling E-mail addresses from lists associated with some of the data centers were included to reach the large number of users who may have accessed data via anonymous ftp. Those who answered for more than one data center: Two: 103 Three: 14 Four: 2

7. 7 © CFI Group 2011 Project Background Respondent information For which specific areas do you need or use Earth science data and services? * Multi-select question; Answer choices added in 2010 and 2011; Language to question was changed slightly in 2009; Modeling was asked as a separate question prior to 2008 Demographics (when comparable) remain fairly consistent with 2010.

8. 8 © CFI Group 2011 Project Background Respondent information * Questionnaire was modified in 2009-2011; Prior to 2010 WIST also included EDG. WIST became available in 2005. EDG was decommissioned Feb. 2008 when all data could be accessed through WIST. Demographics (when comparable) remain fairly consistent with 2010.

9. 9 © CFI Group 2011 Overview Key Results

10. 10 © CFI Group 2011 NASA EOSDIS Customer satisfaction remains steady Ideal How close does [DAAC] come to the ideal organization? Overall satisfaction How satisfied are you with the data products and services provided by [DAAC]? Expectations To what extent have data products and services provided by [DAAC] fallen short of or exceeded expectations? ACSI 76 82 73 78 2005 2007 75 80 73 2008 77 81 74 7571 79 73 75 2004 (+/-) 0.9(+/-) 0.7(+/-) 0.6(+/-) 0.5 N=1016N=1263N=2291N=2601 2009 77 81 73 75 (+/-) 0.4 N=3842 72 78 71 74 (+/-) 0.5 N=2857 2006 2010 77 81 74 75 (+/-) 0.4 N=4390 2011 77 81 74 75 (+/-) 0.4 N=3996

11. 11 © CFI Group 2011 NASA EOSDIS Benchmarks Strong performance continues … ACSI (Overall) is updated on a quarterly basis, with specific industries/sectors measured annually. Federal Government (Overall) is updated on an annual basis and data collection is done in Q3. Quarterly scores are based on a calendar timeframe: Q1- Jan through March; Q2 – April through June; Q3 – July through Sept.; Q4 – Oct. through Dec. 75 77 65 76 304050607080 News & Information Sites (Public Sector) 2011 NASA EOSDIS-Aggregate 2011 Federal Government (Overall) 2010 ACSI (Overall) Q2 2011

12. 12 © CFI Group 2011 NASA EOSDIS Model Product Search/Selection/Documentation most critical The performance of each component on a 0 to 100 scale. Component scores are made up of the weighted average of the corresponding survey questions. Scores The change in target variable that results from a five point change in a component score. For example, a 5-point gain in Product Search would yield a 0.9-point improvement in Satisfaction. Impacts Customer Satisfaction Index Future Use Recommend Sample Size: 3996 77 89 87 Customer Support 86 1.7 Product Search 75 0.9 Product Quality 78 0.4 Product Documentation 76 0.9 Product Selection and Order 77 1.1 3.2 3.8 Delivery 81 0.4

13. 13 © CFI Group 2011 NASA EOSDIS 2008 – 2011 Scores hold steady; no change more than one point =Significant Difference vs. 2010 (+/-) 0.4 (+/-) 0.9 (+/-) 0.5 (+/-) 0.6 77 86 81 78 77 76 75 77 86 80 77 76 77 85 81 77 76 77 75 77 84 81 74 77 75 Customer Satisfaction Index Customer Support Delivery Product Quality Product Selection and Order Product Documentation Product Search 2011201020092008

14. 14 © CFI Group 2011 Areas of Opportunity for NASA EOSDIS Remain consistent year over year Top Improvement Priority Product Search (75) Product Selection and Order (77) Product Documentation (76)

15. 15 © CFI Group 2011 Detailed Analysis

16. 16 © CFI Group 2011 Score Comparison Same CSI inside and outside the USA Respondents inside and outside the USA have the same Satisfaction with EOSDIS (77). USA customers rated Delivery and Customer Support higher than those outside USA. Respondents inside and outside the USA have the same Satisfaction with EOSDIS (77). USA customers rated Delivery and Customer Support higher than those outside USA. 71% of respondents are outside of the USA in 2011 vs. 73% in 2010.

17. 17 © CFI Group 2011 CSI by Data Centers – 2008-2011 Three data centers show significant score changes (+/-) 1.7 (+/-) 1.9 (+/-) 2.3 (+/-) 2.9 (+/-) 2.4 (+/-) 3.2 (+/-) 0.6 (+/-) 1.1 (+/-) 1.2 (+/-) 1.7 (+/-) 2.3 (+/-) 2.6 77 78 83 80 76 78 76 81 75 82 71 75 74 79 80 79 76 77 82 78 80 69 76 75 80 77 79 75 77 81 77 78 70 77 75 88 77 78 76 75 76 80 75 79 70 ASDC-LaRC ASF SAR DAAC CDDIS GES DISC GHRC LP DAAC MODAPS/LAADS NSIDC DAAC OBPG/Ocean Color ORNL DAAC/FLUXNET PO.DAAC-JPL SEDAC 2011201020092008 =Significant Difference vs. 2010 CDDIS (83) and PO.DAAC- JPL (82) have highest satisfaction.

18. 18 © CFI Group 2011 Product Search Remains a key driver of satisfaction and is top priority 60% used data center’s or data-specific specialized search, online holdings or datapool (49% in 2010) 14% used WIST to search for data and products (17% in 2010) 15% selected Internet Search Tool (16% in 2010) Impact=0.9 75 77 75 74 76 78 75 74 75 78 74 75 77 74 Product Search How well the search results met your needs Ease of finding data Ease of using search capability 2011201020092008 =Significant Difference vs. 2010

19. 19 © CFI Group 2011 Product Search Score Comparison By method for most recent search (+/-) 0.6 (+/-) 3.0 (+/-) 5.6 (+/-) 1.4 (+/-) 1.3 (+/-) 3.1 76 73 69 75 77 78 77 74 69 76 72 78 77 74 70 75 77 76 75 70 68 75 76 Data center’s or data-specific specialized search, online holdings or datapool Direct interaction with user services personnel Global Change Master Directory Internet search tool Reverb/Warehouse Inventory Search Tool (WIST) Other 2011201020092008 How did you search for the data products or services you were seeking? 60% 3% 1% 15% 14% 3%

20. 20 © CFI Group 2011 Product Search Scores by Data Center; variation in the trends (+/-) 2.2 (+/-) 2.9 (+/-) 4.5 (+/-) 3.2 (+/-) 3.3 (+/-) 0.7 (+/-) 1.4 (+/-) 1.6 (+/-) 2.2 (+/-) 2.0 (+/-) 2.6 (+/-) 3.2 76 75 77 81 80 74 78 71 79 74 76 69 76 74 75 79 76 75 77 75 81 77 69 77 76 78 71 77 74 77 75 80 75 78 67 77 73 85 78 77 75 74 80 72 75 66 ASDC-LaRC ASF SAR DAAC CDDIS GES DISC GHRC LP DAAC MODAPS/LAADS NSIDC DAAC OBPG/Ocean Color ORNL DAAC/FLUXNET PO.DAAC-JPL SEDAC 2011201020092008 =Significant Difference vs. 2010 GES DISC (81) and GHRC (80) rate Product Search highest.

21. 21 © CFI Group 2011 Product Selection and Order Also a top opportunity for improvements 93% of respondents said that they are finding what they want in terms of type, format, time series, etc. (94% in 2010) Impact=1.1 77 78 77 75 77 78 77 75 76 77 75 77 78 76 75 Product Selection and Order Ease of requesting or ordering data products Ease of selecting data products Description of data products 2011201020092008 Did you use a sub-setting tool? 32% said No 45% said Yes, by geographic area 3% said Yes, by geophysical parameter 17% said Yes, by both geographic area and geophysical parameter 3% said Yes, by band 1% said Yes, by channel

22. 22 © CFI Group 2011 Product Selection and Order Scores by Data Center 77 76 83 79 82 76 78 74 80 76 81 71 75 77 80 82 76 78 76 81 79 80 70 76 73 77 74 77 78 81 76 78 71 76 72 84 79 76 75 81 75 79 67 ASDC-LaRC ASF SAR DAAC CDDIS GES DISC GHRC LP DAAC MODAPS/LAADS NSIDC DAAC OBPG/Ocean Color ORNL DAAC/FLUXNET PO.DAAC-JPL SEDAC (+/-) 2.0 (+/-) 3.0 (+/-) 3.3 (+/-) 0.7 (+/-) 1.3 (+/-) 1.5 (+/-) 2.1 (+/-) 2.0 (+/-) 2.5 (+/-) 3.2 2011201020092008 =Significant Difference vs. 2010 CDDIS (83) and GHRC (82) rate Product Selection and Order highest.

23. 23 © CFI Group 2011 Product Documentation Data product description remains most sought after What documentation did you use or were you looking for? Data product description 78% Product format 66% Science algorithm 45% Instrument specifications 42% Tools 37% Science applications 28% Production code 10% Impact=0.9 CSI for those whose documentation was not found is 68 vs. those who got it delivered with the data (79) or online (78). Was the documentation… Delivered with the data (17% vs. 18% in ‘10) Available online (76% vs. 75% in ‘10) Not found (7% vs. 7% in ‘10) 76 77 76 75 74 Product Documentation Data documentation helped you use the data Overall quality of the document 2011201020092008

24. 24 © CFI Group 2011 Product Documentation Scores by data center (+/-) 2.3 (+/-) 2.9 (+/-) 5.2 (+/-) 3.3 (+/-) 3.9 (+/-) 0.8 (+/-) 1.6 (+/-) 2.6 (+/-) 2.0 (+/-) 3.1 (+/-) 3.4 75 79 78 77 75 74 78 75 78 76 75 74 79 78 80 76 75 76 80 79 80 72 78 77 82 75 80 76 77 76 81 72 76 72 86 76 77 74 72 77 71 79 73 ASDC-LaRC ASF SAR DAAC CDDIS GES DISC GHRC LP DAAC MODAPS/LAADS NSIDC DAAC OBPG/Ocean Color ORNL DAAC/FLUXNET PO.DAAC-JPL SEDAC 2011201020092008 =Significant Difference vs. 2010 5 data centers rate Product Documentation 78 or 79.

25. 25 © CFI Group 2011 Customer Support Maintain great performance 91% (88% in 2010) were able to get help on first request. These respondents continue to have a significantly higher CSI (81) than those who did not (66). Impact=1.7 Did you request assistance from the Data Center’s user services staff during the past year? No=76%. Of those who said yes, 80% used email, 2% used the phone, and 10% used both phone and email. 86 88 87 86 85 86 87 86 85 84 85 87 86 85 83 84 86 84 83 82 Customer Support Professionalism Technical knowledge Accuracy of information provided Helpfulness in selecting data or products Timeliness of response Helpfulness in correcting a problem 2011201020092008

26. 26 © CFI Group 2011 Product Quality Preferences somewhat in line with what provided ~Multiple responses allowed In 2010, 57% said products were provided in HDF-EOS and HDF and 42% said they were their preferred method. GeoTIFF is most preferred format, while HDF-EOS/HDF is format in which products were provided the most. Only 8% of products provided in GIS although nearly one-quarter prefer that format.

27. 27 © CFI Group 2011 Product Quality One-point gain from last year Impact=0.4 78 77 74 Product Quality Ease of using the data product in the delivered format 2011201020092008 =Significant Difference vs. 2010

28. 28 © CFI Group 2011 Delivery Timeliness and Delivery up one point Over half said their data came from MODIS (same as 2010); 32% said ASTER (28% in 2010) Impact=0.4 MODIS (Atmosphere): 22% MODIS (Cryosphere): 8% MODIS (Land): 52% MODIS (Ocean): 18% *Question is multi-select 81 82 80 82 79 81 82 79 81 83 79 Delivery Convenience of delivery method Timeliness of delivery method 2011201020092008 =Significant Difference vs. 2010

29. 29 © CFI Group 2011 Delivery Methods for receiving … How long did it take to receive your data products? 23% immediate retrieve CSI=81 23% less than 1 hour CSI=78 26% less than a day CSI=76 22% 1-3 days CSI=77 4% 4-7 days CSI=74 2% more than 7 days CSI=66 67% said FTP was their preferred method in 2010 FTP immediate retrieval from online holdings is most preferred but retrieved after order is most used.

30. 30 © CFI Group 2011 Summary

31. 31 © CFI Group 2011 Summary  Satisfaction with NASA EOSDIS has held at 77 for four years. NASA continues to meet data users needs.  As would be expected with consistent satisfaction scores, there were very few changes in drivers’ scores. Half of the drivers had no change and any changes in satisfaction drivers were no more than one point. Delivery and Product Quality improved one point, while Product Search was down one point. There was no change in Customer Support, Product Selection and Order, and Product Documentation. However, due to the large sample size a one-point change is statistically significant at a 90% confidence level.  While scores are solid, there are opportunities to improve. Product Search, Selection and Order, and Documentation continue to be the top priorities. Work to refine and improve the search capabilities and functionality. Continue to clarify descriptions of data products, and make language easy to understand. As some respondents are still having difficulty locating documentation, continue to work on both providing accessible and clear documentation that is readily available for users.

32. 32 © CFI Group 2011 Summary  Customer Support remains the top scoring area. As it is also the highest impact area, it is important to maintain the great level of service and support already being provided. Share the importance and impact of customer support with those providing it, to help boost awareness. Look for areas of best practice among top performing data centers, to ensure all centers are providing high levels of service.  GeoTIFF and GIS appear to have a higher preference by customers than what was provided. NASA should explore offering more data products in these formats.

33. 33 © CFI Group 2011 Appendix

34. 34 © CFI Group 2011 Customers over multiple years Who have answered the survey multiple years … No significant differences were seen between 2010 and 2011 for those who have answered the survey over the last four years. For those answering the survey over multiple years, score movement is mixed. (Difference refers to 2011 vs. 2010)

35. 35 © CFI Group 2011 Customers over the past three years Who answered the survey in 2009, 2010 and 2011 For those answering the survey in 2009, 2010 and 2011, there are no statistically significant score differences. (Difference refers to 2011 vs. 2010)

36. 36 © CFI Group 2011 Customers over the past two years Who answered the survey in 2010 and 2011 For those answering the survey in 2011 and 2010, there are a number of statistically significant positive score differences. (Difference refers to 2011 vs. 2010)

37. 37 © CFI Group 2011 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x 1 x 3 x 4 x 5 x 6 y1y1 y2y2 y3y3 y 3 y 2 y 1 11 22  x i xit i , for i=1,2,3 t=1,2 y jyjj  1, for j=1,2,3    111221  x 2 The Math Behind the Numbers A discussion for a later date…or following this presentation for those who are interested.

38. 38 © CFI Group 2011 A Note About Score Calculation Attributes (questions on the survey) are typically answered on a 1-10 scale –Social science research shows 7-10 response categories are optimal –Customers are familiar with a 10 point scale Before being reported, scores are transformed from a 1-10 to a 0-100 scale –The transformation is strictly algebraic; e.g. –The 0-100 scale simplifies reporting: Often no need to report many, if any, decimal places 0-100 scale is useful as a management tool

39. 39 © CFI Group 2011 Deriving Impacts Remember high school algebra? The general formula for a line is: y = mx + b The basic idea is that x is a “cause” and y is an “effect”, and m represents the slope of the line – summarizing the relationship between x & y CFI Group uses a sophisticated variation of the advanced statistical tool, Partial Least Squares (PLS) Regression, to determine impacts when many different causes (i.e., quality components) simultaneously effect an outcome (e.g., Customer Satisfaction)