Preliminary Evaluation Results: A Discussion with the Advisory Committee on Citizen-Friendly Reporting August 30, 2005 Human Services Research Institute Madeleine Kimmich Mike Nikolenko

Evaluation Background All three state agencies selected a website as their preferred method to communicate performance information HSRI developed the survey tool to help evaluate the three sites Survey questions were based on the GASB criteria for the effective communication of performance measures

Our Core Evaluation Question: How well each project’s presentation of performance measure information is likely to engage and help interested citizens? 1. Is the interface user-friendly? (Understandability) 2. Is the information they choose to present easy to understand and meaningful? (Comparability/Timeliness/Consistency/ Reliability) 3. How can users find out if the information is available? (Accessibility)

The Survey Tool One set of questions for each agency Questions address each of the 6 GASB criteria:  Accessibility (#1-2)  Understandability (#3-5)  Comparability (#6)  Timeliness (#7)  Consistency (#8a)  Reliability (#8b)

The Survey Tool, Continued The survey tool evolved over time:  Turning one survey into three  Eliminating some questions to reduce length  Adding in demographic information

Methodological Issues Lack of focus groups Very low response rate Change in the method of survey distribution Targeting respondents See Table 1 and Figure 1

Results: Accessibility Most respondents felt it was easy to open the web pages and navigate the sites. See Tables 2 and 3 Respondent Comment (DHS): “Adobe is a poor format for web pages as the Acrobat reader had to load and this takes time.”

Results: Understandability Most respondents felt the measures presented on the sites were important. Most respondents also said that the text was clear and helpful. See Tables 4 and 5 and Figure 2 Respondent Comment (DHS): “The website helps tremendously to provide the public with a snapshot of the organization.”

Results: Comparability Many respondents felt there was a clear frame of reference for the data and that historical data was offered for comparison (DHS and DCBS). DMV respondents differed on these questions. See Tables 6 and 7 Respondent Comment (DMV): “The only thing missing was the history of previous customer surveys.”

Results: Timeliness Respondents reported knowing how recently data was collected and most felt this was recent enough to be useful. See Tables 8 and 9 Respondent Comment (DCBS): “Very timely and clear…I really liked the few sentences in the “on track” box.”

Results: Consistency and Reliability Respondents differed when asked if they knew the origin of website data. A majority of respondents trusted the accuracy of the data presented. See Tables 10 and 11

Results: Other Findings A majority of respondents said: -There is a convenient way to provide feedback on the website -They learned something about the agency from the site -The site did not make them feel differently about the agency See Tables and Figure 3 Respondent Comment (DMV): “The survey shows that the agency is willing to receive input from customers so they can improve in areas needing work.”

Conclusions All agency websites performed well  Just say “no” to PDF files! Text descriptions on the sites were helpful  It is preferable to layer information People are not used to critical examination of data  This is part of educating interested citizens

Conclusions, Continued Educate citizens about the frame of reference  What is the agency’s purpose? It is viable to use websites to engage the public