CFI GROUP WORLDWIDE ANN ARBOR ATLANTA BEIJING LONDON MADRID MILAN PARIS SHANGHAI STOCKHOLM REPRESENTATIVE OFFICES BUENOS AIRES KUALA LUMPUR PORTO ALEGRE NASA Earth Observing System Data and Information Systems Customer Satisfaction Results November 1, 2006

2 © CFI Group 2006 Today’s Discussion Background Overview Key Results Detailed Analysis Summary

3 © CFI Group 2006 Background

4 © CFI Group 2006 Project Background Objectives Measure customer satisfaction with the NASA Earth Observing System Data and Information System at a national level and for each Data Center –Alaska Satellite Facility (ASF) –Goddard Space Flight Center Earth Sciences Distributed Active Archive Center (GES DAAC) –Global Hydrology Resource Center (GHRC) –NASA Langley Atmospheric Science Data Center (LaRC) DAAC –Land Processes Distributed Active Archive Center (LP DAAC) –National Snow and Ice Data Center (NSIDC) –Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC) –Physical Oceanography Distributed Active Archive Center (PO DAAC) –Socioeconomic Data and Applications Center (SEDAC) Assess the satisfaction with NASA EOSDIS specifically in the following key areas: –Product Search –Product Selection and Order –Delivery –Product Quality and Documentation –Customer Support Identify the key areas that NASA can leverage across the Data Centers to better service its users

5 © CFI Group 2006 Project Background Measurement timetable Finalized questionnaireAugust 24, 2006 Data collection via webAugust 29, 2006 – September 20, 2006 Topline resultsSeptember 28, 2006 Results briefingNovember 1, 2006

6 © CFI Group 2006 Project Background Data collection Respondents A total of 2,857 responses were received:

7 © CFI Group 2006 Project Background Respondent information Q8. For which disciplines do you need or use Earth science data? (n=2,857)* *Multi-select

8 © CFI Group 2006 Project Background Respondent information Demographics remain fairly consistent with *Multi-select

9 © CFI Group 2006 Overview Key Results

10 © CFI Group 2006 NASA EOSDIS Customer satisfaction results How well does the Data Center compares with an ideal provider of scientific data, products and services 72 Overall satisfaction with the products and services provided by the Data Center ATTRIBUTES How well the products and services provided by the Data Center meet expectations ACSI

11 © CFI Group 2006 NASA EOSDIS Benchmarks Continues to score well … “Generally pleased with the system. Given the variety and volume of data it is hard to imagine doing much better.” “Your services are excellent to the betterment for the world community as a whole.”

12 © CFI Group 2006 Customer Satisfaction Index Future Use Recommend NASA EOSDIS Model Customer Support and Product Search/Selection most critical Sample Size: 2, The performance of each component on a 0 to 100 scale. Component scores are made up of the weighted average of the corresponding survey questions. Scores Customer Support Delivery Product Documentation Product Search Product Selection and Order The change in target variable that results from a five point change in a component score. For example, a 5-point gain in Product Search would yield a 0.9-point improvement in Satisfaction. Impacts Product Quality

13 © CFI Group 2006 NASA EOSDIS 2006 vs Significant declines from 2005 =Significant Difference vs. 2005

14 © CFI Group 2006 Areas of Opportunity for NASA EOSDIS Focus on the search, selection and ordering for improvement Also important to keep an eye on Customer Support (82). High impact area – further declines will affect overall customer satisfaction.

15 © CFI Group 2006 Detailed Analysis

16 © CFI Group 2006 Score Comparison Higher satisfaction persists outside of the USA Respondents outside the USA continue to have a higher overall Satisfaction score with EOSDIS (79 outside vs. 75 USA in 2005), though gap has lessened. 64% of respondents are outside of the USA in 2006 vs. 67% in 2005.

17 © CFI Group 2006 CSI by Data Centers All Data Centers register declines in satisfaction n=45 n=305 n=808 n=909 n=273 n=51 =Significant Difference vs n=359 n=29 n=92 n=30 n=419 n=138 n=165 n=65 n=223 n=96 n=78 n=35

18 © CFI Group 2006 Product Search Key driver of satisfaction 65% used EOS Data Gateway to search for data and products (63% in 2005) Note: All score decreases are statistically sig. Impact=0.9 “What I would like to see is simple more dynamic data search and order with plenty of (popular) pre- order preprocessing capabilities.” “Initially, I did not find the data search tools very intuitive and I did not find the explanation of how to carry out a search very helpful. However, I persevered and now find the search tool easy to use.”

19 © CFI Group 2006 Product Search Score Comparison By method for most recent search Q12. How did you search for the data products or services you were seeking? (n=2,857) 111 indicated other; 93 said direct interaction (did not rate product search questions) n=191 n=534 n=64 n=1,864 n=104 n=225 n=33 n=802

20 © CFI Group 2006 Product Search Scores by Data Center =Significant Difference vs. 2005

21 © CFI Group 2006 Product Selection and Order Also a top opportunity for improvement Impact=0.7 Note: All score decreases are statistically sig. 92% said that they are finding what they want in terms of type, format, time series, etc. Q16. Please think about your most recent request/order/download from the Data Center. Did you use a subsetting tool? (n=2,857) 30% said No, 59% said Yes, by geographic area and 11% said Yes, by geophysical parameter. “All data should be available free of cost for scientific research.”

22 © CFI Group 2006 Product Selection and Order Scores by Data Center =Significant Difference vs. 2005

23 © CFI Group 2006 Customer Support While high scoring, keep a close eye on … Impact=1.6 =Significant Difference vs Only 59% were aware that the Data Center has a user services office for assistance with placing orders Q37. Did you request assistance from the Data Center’s user services staff during your most recent search or order? (n=2,857) Yes=18%, No=82% 86% were able to get help on first request. These respondents have a significantly higher CSI (79) than those who did not (59).

24 © CFI Group 2006 Delivery Methods for receiving … How long did it take to receive your data products? 22% immediate retrieve 32% less than a day (44% in 2005) 26% 1-2 days (30% in 2005) 12% 3-7 days (15% in 2005) 5% 8-14 days (6% in 2005) 3% more than 14 days (5% in 2005) 65% said FTP was their preferred method in 2005; 22% said download from web.

25 © CFI Group 2006 Delivery Impact=0.5 Note: All score decreases are statistically sig. 62% said their data came from MODIS; 30% said ASTER

26 © CFI Group 2006 Product Documentation Visibility of documentation is important What documentation did you use or were you looking for?* Data product description 68% Product format 58% Science algorithm 44% Instrument specifications 39% Tools 36% Science Applications 33% Production code 12% Note: Questions reworded, not comparable with 2005 Impact=0.4 *Multi-select Q33. Was the documentation (n=2,844)... Delivered with the data (18% vs. 28% in ‘05), Available online (70% vs. 63% in ‘05), Not found (12% vs. 9% in ‘05). “More clear and comprehensive documentation about the data.” CSI for those whose documentation was not found is 62 vs. those who got it delivered with the data (76) or online (75).

27 © CFI Group 2006 Product Quality Preferences in line with actual for the most part In 2005, 9% said products were provided in GeoTIFF and 25% who said it was their preferred method.

28 © CFI Group 2006 Product Quality Impact=0.1 *Attributes not included in model * * “Create tutorials that would make it easier for non- experts … to use the data in different applications.”

29 © CFI Group 2006 Summary

30 © CFI Group 2006 Summary  While lower than last year, NASA EOSDIS still performs on par or better than government benchmarks  Product Search, Selection and Order continue to be the top opportunities for improvement Utilize additional research (i.e., focus groups, usability studies) to understand/address challenges with finding data and search capabilities Communicate reasons for charging for data  While Customer Support scores well, further declines will affect satisfaction. Monitor this area. High impact for those who require it  Most Data Centers with large number of respondents show significant declines (e.g., PO DAAC, GES DAAC, LP DAAC, ORNL DAAC) Focus improvement efforts here first

31 © CFI Group 2006 Appendix

32 © CFI Group 2006 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x 1 x 3 x 4 x 5 x 6 y1y1 y2y2 y3y3 y 3 y 2 y 1 11 22  x ixit i , for i=1,2,3 t=1,2 y jyjj  1, for j=1,2,3     x 2 The Math Behind the Numbers A discussion for a later date…or following this presentation for those who are interested.

33 © CFI Group 2006 A Note About Score Calculation Attributes (questions on the survey) are typically answered on a 1-10 scale –Social science research shows 7-10 response categories are optimal –Customers are familiar with a 10 point scale Before being reported, scores are transformed from a 1-10 to a scale –The transformation is strictly algebraic; e.g. –The scale simplifies reporting: Often no need to report many, if any, decimal places scale is useful as a management tool

34 © CFI Group 2006 Deriving Impacts Remember high school algebra? The general formula for a line is: y = mx + b The basic idea is that x is a “cause” and y is an “effect”, and m represents the slope of the line – summarizing the relationship between x & y CFI Group uses a sophisticated variation of the advanced statistical tool, Partial Least Squares (PLS) Regression, to determine impacts when many different causes (i.e., quality components) simultaneously effect an outcome (e.g., Customer Satisfaction)

35 © CFI Group 2006 Who is CFI Group? Founded in 1988; headquartered in Ann Arbor, Michigan Principals are among the world experts in measuring constituent/stakeholder satisfaction and how to improve it 14 offices, 150 full-time consultants and researchers worldwide The CFI System is the source for the American Customer Satisfaction Index (ACSI), a national measure of customer satisfaction compiled by the National Quality Research Center at the University of Michigan Business School

36 © CFI Group 2006 Unique Features of the American Customer Satisfaction Index (ACSI) The only uniform measure of customer satisfaction in the U.S. economy, covering sectors accounting for about 66% of GDP Measures the quality of economic output on a quarterly basis; complementary to productivity measures and indicative of consumer spending Uses multiple-item indicators to assess drivers of satisfaction Meets the objective of explaining desired outcomes Allows for comparison across agencies Illustrates how customer satisfaction is embedded in a system of cause and effect relationships

37 © CFI Group 2006 ACSI National, Sector and Industry Scores: Q – Q Hotels 77Limited- Service Restaurants 63Newspapers 73Motion Pictures 69Network/Cable TV News 74Computer Software 70Fixed Line Telephone Service 66Wireless Telephone Service 70Cellular Telephones 63Cable & Satellite TV 72Energy Utilities 74Supermarkets 69Gasoline Stations 75Department & Discount Stores 74Specialty Retail Stores 76Health & Personal Care Stores 75Banks 75Life Insurance 68Health Insurance 78Property & Casualty Insurance 65Airlines 71U.S. Postal Service 83Express Delivery 65.9Local Government 71.3Federal Government Accommodation & Food Services 75.8 Information 68.6 Utilities 72.4 Finance & Insurance 73.9 Transportation & Warehousing 72.3 Public Administration/ Government Retail 78Auctions 76Brokerage 77 Travel E-Commerce Hospitals Health Care & Social Assistance 74.1 Source: ACSI 74.4 Manufacturing/ Durable Goods 80.1 E-Business Personal Computers 80Electronics (TV/VCR/DVD) 81Major Appliances 81Automobiles & Light Vehicles 73News & Information 76Portals 79Search Engines Manufacturing/ Nondurable Goods Food Manufacturing 82Pet Food 83Soft Drinks 82Breweries 79Cigarettes 81Apparel 77Athletic Shoes 83Personal Care & Cleaning Products Retail Trade 72.4