Using Customer Feedback to Improve Data Delivery Colleen Blessing U.S. Dept. of Energy The Hague Sept. 2005

Two Customer Feedback Efforts The American Customer Satisfaction Index Survey (ACSI) Survey of customer satisfaction Run by the University of Michigan Used by many government agencies Uniform and independent measure Website Usability Testing Evaluating a new web design

ACSI Methodology Includes 15 standard questions Content, navigation, search, site performance EIA added 11 custom questions demographics, suggestions for improvement Invitation to survey tagged on L2 pages Survey began August 2004 Received 4650 responses by July 2005

Concerns about the Methodology Does ACSI capture a representative sample of EIA’s customers? Sample could be skewed because: Pop up blocking technologies prevent customers from seeing the survey. Repeat customers only respond to the survey once. Some EIA customers under deadlines are reluctant to respond to a web survey. The ACSI invitation is not compatible with all EIA web pages. Customers’ browsing techniques – search and bookmarking.

ACSI Results—Overall Scores The overall score is an index rather than a % satisfaction score. Search and Navigation are consistently rated lowest. Content is consistently rate high. Respondents indicate high likelihood to return and to recommend the site to others. Similar results to those from Statistics Canada’s survey – gaps between satisfaction and importance are largest for search and ease of finding information.

Countries with the most ACSI Respondents (#) More than one-third of ACSI respondents are from outside the United States (n=4650 through July 2005) ACSI Respondents by Country Countries with the most ACSI Respondents (#) Canada (234) France (60) UK (186) Brazil (57) Mexico (76) Italy (53) Australia (75) Germany (49) India (66) Venezuela (44)

Main Customer Groups = Business/Industry and Research/Consulting (n=4650 through July 2005) Distribution among sectors is similar for U.S. and International respondents

Most ACSI respondents are familiar with the energy industry (n=4650 through July 2005) More than 80% of respondents were at least somewhat familiar with the energy industry Cross tabs indicated that international respondents were less familiar with the energy industry Familiarity with the energy industry and energy terms

Most respondents visit EIA’s website regularly but 30% are first-time users (n=4650 through July 2005) Nearly two-thirds of respondents visit EIA’s website at least monthly 30% of respondents were first-time users Cross tabs indicated that more international respondents were first time users How frequently do you visit this site?

A New ACSI Question: How do you navigate the site A New ACSI Question: How do you navigate the site? (n=1538 from March to July 2005)

Who Has Trouble on our Site? Customers less successful finding information are: International users Those unfamiliar with the energy industry First-time visitors People who used the search box Respondents’ perceptions of data quality are highly correlated with their overall experience at the website.

Respondents who were unsuccessful in finding information were less satisfied with EIA’s website (n=3325 through March 28, 2005) “What is your overall satisfaction with this site?”

Respondents who did not find information rated the quality of EIA’s information much lower (n=3325 through March 28, 2005) “Please rate the quality of information on this site.”

Redesigning the Website: Testing ACSI results showed we needed to improve navigation and information architecture. Needed to help first-time and non-energy-expert users Usability testing on new designs Paper tests first Tests in the lab 20 common user questions Turned off search function - Tested 36 people

Results of the Usability Testing Much higher finding rate 2001=40% 2002=53% 2005=79% corp=66% What worked well Hints, hot box, publications list, less clutter, major link names, more consistency on L2 What didn’t work very well Global navigation, stovepiping, inconsistencies on L2, glossary at top of page, frustration not having search function

New EIA Home Page