1 Performance Measurement Community Literacy March 19, 2007 Harry P. Hatry The Urban Institute Washington DC

2 Key Distinctions Performance Measurement vs. Program Evaluation Performance Measurement vs. Performance Management

3 PROGRAMPERFORMANCE EVALUATIONSMONITORING Frequency:IrregularRegular, Continuing Coverage:Done on only Covers most a few programsprograms Depth ofSeeks reasons for Only tells “the Information:poor performance score”, not WHY CostHigh for each studyCost spread out UtilityMajor DecisionsContinuous program Improvement

4 Performance Measurement Information Plus Use of that Information to Improve Services Produces Performance Management

5 Outcome Sequence Chart Fed/State Funds Provided Organization Develops Improvement Plan Teachers Implement Changes to Instructional Practice in Classroom Students participate in regular classroom Instruction Students Demonstrate Improved Performance Students Successfully Complete Education Requirements Students Enrolled in Post Secondary Education and/or Employed “Intermediate” OutcomesOutput “End” Outcomes

6 Outcome Sequence Chart Fed/State Funds Provided Organization Develops Improvement Plan Teachers Implement Changes to Instructional Practice in Classroom Students participate in regular classroom Instruction Children with Disabilities Demonstrate Improved Performance Children Successfully Complete General Education Requirements Students Enrolled in Post Secondary Education and/or Employed “Intermediate” OutcomesOutput “End” Outcomes # Schools with SEA approved plans #/% of students participating in regular instruction #/% of students enrolled in post secondary education, and/or employed #/% of students who complete education requirements #/% of students demonstrating improved performance Quantity of funds provided #/% teachers reporting changes

7 Suggested Added Outcome Indicators #/% of students whose test scores showed one year’s gain. [End Outcome ] #/% of students reporting satisfaction with the assistance they received. [End Outcome] # of students that volunteered for the assistance [Intermediate Outcome] % of students enrolling divided by the number of eligibles. [Intermediate Outcome]

8 TYPICAL SERVICE QUALITY CHARACTERISTICS 1.TIMELINESS/WAIT TIMES 2.PLEASANTNESS/FRIENDLINESS 3. CONVENIENCE/ACCESSIBILITY  OF HOURS OF OPERATION  CUSTOMER CAN REACH SOMEONE TO TALK TO 4. AWARENESS OF PROGRAM SERVICES 5. CLARITY OF INFORMATION/REGULATIONS 6. STAFF/TEACHER HELPFULNESS/KNOWLEDGE 7. OVERALL CUSTOMER SATISFACTION

9 Categories of Data Sources and Collection Procedures Agency Records Administered Tests Customer Surveys Trained Observer Procedures Expert Judgments Focus Groups

10 Sample Outcome Information From Customer Surveys Ratings of overall satisfaction Ratings of specific service quality characteristics Ratings of results of the service Whether actions/behavior sought by the program occurred Extent of service use Awareness of services Reasons for dissatisfaction or non-use Suggestions for improvements

11 1. Provide frequent, timely information to programs and their staffs. 2. Set targets each year. 3. Disaggregate outcome data by customer and service characteristics. 4. Do regular, basic, analysis of the data, such as comparisons. 5. Seek explanations for unexpected outcomes. Making Performance Information Really Useful

12 Percent of Student That Reported The Program’s Assistance Had Helped Them Improve Their Reading N Very or Somewhat Helpful Target Difference (Percentage Points) All Clients % 60% -10 Gender Females % 60% -30 Males % 60% 4 Beginning Reading Level Lowest % 0 Second % 60% -5 Third % 60% -16 Highest 60 33% 60% -27 Faculty A % 60% -7 B 30 67% 60% 7 C % 60% -27 D % 60% -2

13 Which Hospital Would You Choose?

14 Compare the Latest Outcome Data: 1.To previous performance 2.To targets set by the organization 3.Among categories of customers 4.Among facilities 5.By type and amount of service 6.To results in other communities Types of Comparison

15 Making Performance Information Really Useful (Continued) 6.Hold “How Are We Doing?” sessions after each performance report. 7.Prepare “Service Improvement Action Plans” for areas with low performance. 8.Provide recognition rewards. 9.Identify successful practices.

16

17 Website: Outcome Indicators Project A joint project of the Urban Institute and The Center for What Works The Outcome Indicators Project provides a framework for tracking nonprofit performance. It suggests candidate outcomes and outcome indicators to assist nonprofit organizations that seek to develop new outcome monitoring processes or improve their existing systems. This website contains three primary elements: 1. Building a Common Outcome Framework to Measure Nonprofit Performance 2. Outcomes and Performance Indicators for 14 Specific Program Areas 3. Nonprofit Taxonomy of Outcomes Adult Education and Family Literacy Employment Training Advocacy Health Risk Reduction Affordable HousingPerforming Arts Assisted LivingPrisoner Re-entry Business Assistance Transitional Housing Community Organizing Youth Mentoring Emergency Shelter Youth Tutoring

18 Crocodiles May Get You But in the End It Should be Very Worthwhile For Student Literacy