Developing Effective Performance Measures: A Skill-Building Workshop David J. Bernstein, Ph.D., Senior Study Director, Westat Sponsored by the Government Evaluation Topical Interest Group 2014 American Evaluation Association Conference, October 17, 2014

Questions to Be Addressed 1. What are the types of performance measures, and how are performance measures used? 2. What is a logic model, and what are some examples of performance measurement use models? 3. What are the obstacles to the institutionalization of performance information? 4. How can we move towards solutions to encourage appropriate use of performance measures? 5. What are government managers’ perceptions of performance measurement quality? 2

Questions to Be Addressed 1. What are the types of performance measures, and how are performance measures used? 3

Types of Performance Measures  Inputs: Measures of the resources to provide services.  Activities: Measures of the services being provided.  Outputs: Measures of the quantity of services provided or the quantity of a service meeting quality requirements.  Outcomes: Measures that address the short-term, intermediate, or long-term results of a program or process on conditions for those receiving a service.  Customer Satisfaction: Measures of stakeholder satisfaction with processes and results.  Impacts: Measures that assess conditions in the lives of program recipients. 4

Government Processes That Incorporate Performance Measures  Strategic Planning  Performance and Annual Reports  Citizen/Customer Surveys  Program Performance Monitoring  Contractor Monitoring 5  Budgeting  Benchmarking  Personnel Evaluation  Policy Analysis  Performance Auditing  Evaluation Source: Bernstein, 2000

6 Performance Measurement Use Epstein’s Conception of PM Use (1988) 1. Improved decision making: day-to-day program management, resource allocation decisions, and support for strategic planning and public policy alternatives. 2.Program monitoring and service performance: tracking resources, intended and unintended program impacts, and monitoring the need for more comprehensive analysis (program evaluations, audits). 3.Performance measurement reporting: report on government decisions and alternatives considered, use of public funds and program results, and to provide information to various stakeholders.

Opportunities: A Utilization-Focus Performance measurement is but one facet of program evaluation, and it can be well served by the evaluation profession's institutional memory about enhancing utilization of strong performance data. Kathy Newcomer, The George Washington University 7 Source: Newcomer (1997)

Patton and Epstein on “Use” PattonEpstein Primary Uses of Evaluation Uses of Performance Measurement 1.Judge Merit and Worth1. Improve Public Accountability/ Communication 2.Improve Programs2. Improve Service Performance 3.Knowledge Generation3. Improve Decision Making 8 Patton, 2012 Epstein, 1988

Intended/Unintended Users and Uses Unintended Uses Intended Uses Unintended Users Unintended Trouble Unintended Success Intended Users Need More Technical Assistance and Training 9 Utilization -focused

Questions to Be Addressed 2. What is a logic model, and what are some examples of performance measurement use models? 10

11 Generic Logic Model W.K Kellogg. (2004). W.K. Kellogg Foundation logic model development guide. Battle Creek, MI: W.K Kellogg. Strategic Goal Objective 1 Performance Measure Objective 2 Performance Measure Performance Goal/Target

12 National Center for Mobility Management Logic Model Money # of staff Staff time (WYs) # of volunteers % of $ spent on project management Program operations Communication and outreach Technical assistance (TA) Training Communication Regional liaisons Information briefs Training courses CTAPs Training portal Promising practices MMIP CCAP Support # of participants # of downloads # of web hits # of Tweets/ Retweets # of courses # of information briefs completed # of trainings # of best practices # of inquiries Average response time to inquiries Increase in user satisfaction Increase in fiscal efficiency % improvement in response time % of practices considered rated as “best practices” % of users satisfied with MMIP data % of participants rating conference as promoting mobility management What resources are used? What are the program activities? What are the direct products of activities? What changes to services, conditions, processes? ?

13 Continuous Improvement Models: Prince William County, VA Marshall, M. (1998). Local government performance measurement: a view of the possibilities, pain and promise. Prince William County, Virginia: Office of Executive Management.

14 Continuous Improvement Models: GPRA Source: U.S. General Accounting Office, Executive Guide: Effectively Implementing the Government Performance and Results Act, June 1996 Government Performance and Results Act of 1993

Obstacles to PM Use 15

Shifting from Obstacles to Solutions 16 PROGRESS

17 Appropriate Use of PMs: Solutions

Government Managers’ Perception of PM Quality Clear/Comprehensible/Understandable: Measures should be easily understood. Results-Oriented: Measures should focus on results or outcomes to be achieved. Useful: Measures should be useful, particularly to staff for monitoring program activities. Valid, Verifiable, Accurate: Measures should be accurate and valid, capturing information so that it can be independently verified. Accessible, Easy to Capture, Measurable: Measures should be readily available and easy to get a hold of when needed. Comparable: Measures should allow for comparisons with other entities or historical results. Source: Bernstein,

References Bernstein, D.J. (2000). Local government performance measurement use: Assessing system quality and effects. George Washington University, Washington, DC. Available from UMI-ProQuest, =10020:22372:US&track=DxWeb&dlnow=1&rpath=http%3A//dissexpress.umi.com/dxw eb%3F%26query%3DAU%2528david%2520joseph%2520bernstein%2529%26page% 3D1. =10020:22372:US&track=DxWeb&dlnow=1&rpath=http%3A//dissexpress.umi.com/dxw eb%3F%26query%3DAU%2528david%2520joseph%2520bernstein%2529%26page% 3D1 Bernstein, D.J. (1998). The theoretical and potential relationship of performance measurement and program evaluation. Presented at the 1998 American Evaluation Association Conference, Chicago, IL. Epstein, P. (1988). Using performance measurement in government: A guide to improving decisions, performance, and accountability. New York, NY: National Civic League Press. Newcomer, K. (1997). “Using performance measurement to improve programs.” In K. Newcomer, Ed., Using performance measurement to improve public and nonprofit programs, New directions for evaluation, Number 75 (pp. 5-14). San Francisco, CA: Jossey-Bass. Patton, MQ. (2012). Essentials of utilization-focused evaluation. Los Angeles, CA: Sage Publications. 19

Contact Information 20 For a copy of this PowerPoint or other information, contact: David J. Bernstein, Ph.D. Senior Study Director, Westat 1600 Research Blvd, RA1292 Rockville, MD