Performance Measures AmeriCorps Project Director Training Saratoga, NY October 7 th – 9 th, 2013

 Theory of Change  Activity  Performance Measure Basics Overview  Activity/Questions AGENDA

Theory of Change

INTRODUCTION Theory of Change Perspective Community Problem/ Need Specific Intervention Intended Outcome  Looks at cause and effect relationships  Identifies specific interventions to achieve the desired result  Shifts thinking from “what are we doing” to focus on “what do we want to achieve”

Overview of Prohibited Activities Theory of Change Elements Community Problem/ Need Specific Intervention Intended Outcome STATISTICS documenting the need EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship

Prohibited Activities Everyday Life Example  I have strep throat (problem)  I will take antibiotics (intervention)  I will get better (outcome) Strep ThroatAntibioticsBe Healthy EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship

Prohibited Activities Everyday Life Example cont. EVIDENCE Penicillin Amoxicillin ×Tetracycline EVIDENCE Penicillin Amoxicillin ×Tetracycline  There are many antibiotics out there  Which antibiotic do I take for strep?  Look at evidence to make the choice Strep ThroatAntibioticsBe Healthy

Overview of Prohibited Activities Theory of Change Elements Community Problem/ Need Specific Intervention Intended Outcome STATISTICS documenting the need EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship Community Problem/Need is the specific issue your project, with its specific intervention (service activity), is designed to address. What is the extent and severity of this need in the community?

Overview of Prohibited Activities Theory of Change Elements - Community Need Community Problem/ Need Data documenting problem/need should answer these questions:  SCOPE: Who and how many are directly affected? How severe is this?  SIGNIFICANCE: What makes this a compelling need? Is it likely to become worse? What will happen if we do nothing?  CAUSE(S): Why does the need exist? How is it perpetuated?

Overview of Prohibited Activities Theory of Change Elements - Statistics STATISTICS documenting the need Community Problem/ Need  The best data comes from reputable primary sources like government agencies, institutions, and universities that do their own research that is:  Locally relevant  Up-to-date  Note: New reports are not primary sources but could be used as back-up to primary evidence. Data documenting problems/needs should be statistic that document the extent and severity of the problem.

Overview of Prohibited Activities Theory of Change Elements - Outcome Community Problem/ Need Specific Intervention Intended Outcome STATISTICS documenting the need EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship What change are you hoping to make related to the identified need?

Overview of Prohibited Activities Theory of Change Elements - Outcome Intended Outcome Healthy Futures Program Example Possible outcomes to measure:  Increased knowledge of what is “healthy food”  More frequent choice of healthy foods to eat  More frequent involvement in physical activity or exercise  Improved physical condition Identifying the Intended Outcome.

Overview of Prohibited Activities Theory of Change Elements - Intervention Community Problem/ Need Specific Intervention Intended Outcome STATISTICS documenting the need EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome?

Overview of Prohibited Activities Theory of Change Elements - Intervention EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship Specific Intervention  Design – who does what with whom?  Dosage  Frequency – how many sessions a week?  Intensity – length of each session.  Duration – how many total weeks of sessions? Describe the design and dosage of your intervention (service activity):

Overview of Prohibited Activities Theory of Change Elements - Intervention Specific Intervention  Design: national service participants implement the Shape Up curriculum with economically disadvantaged urban girls ages to increase physical activity (30 minutes/session) and educate them on healthy eating  Frequency: twice a week afterschool  Intensity: 60 minutes per session  Duration: 12 weeks Healthy Futures Program Example

Overview of Prohibited Activities Theory of Change Elements - Evidence Specific Intervention Intended Outcome EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship  Evidence demonstrates that the proposed intervention (design, dosage) is likely to lead to the outcome.  Service activities that are supported by research to demonstrate a high likelihood that these activities will address the need, and result in intended changes.

Overview of Prohibited Activities Theory of Change Elements - Evidence EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship Evidence provides a reality check for theory of change  New programs: What is the recommended design (specific program activities) and dosage (frequency, intensity, and duration) to achieve an intended outcome?  For existing programs: Is there sufficient evidence for the intervention to continue its use? Based on the evidence, are there modifications to the intervention that should be considered to have greater impact? Do you need to choose a new intervention? EVIDENCE: Information that supports your choice. It is NOT enough to just say “we believe” our intervention is “likely” to be successful.

Overview of Prohibited Activities Theory of Change Elements - Evidence EVIDENCE Guides choice of intervention Supports cause-effect relationship EVIDENCE Guides choice of intervention Supports cause-effect relationship  Your past performance measurement outcome data  Results from an impact evaluation of your program  Research that documents the outcomes of similar programs  Impact evaluations that document outcomes of similar programs Possible sources of evidence:

Overview of Prohibited Activities Testing Your Theory of Change  PLAUSIBLE: Does the logic of the model seem correct: “if we do these things, will we get the results we expect?”  FEASIBLE: Are resources sufficient to implement the chosen intervention?  MEANINGFUL: Are intended outcomes important? Is the magnitude of expected change worth the effort? Is your Theory of Change:

Overview of Prohibited Activities Theory of Change Summary  A theory of change identifies cause/effect  The three elements of a theory of change; community problem/need, intervention, intended outcome are supported by data and evidence  Data documenting community need should show scope, significance, and causes Summary of Key Points:

Case Study

Performance Measure Overview

Overview of Prohibited Activities Performance Measures Accountability to funders and stakeholders  Tell your story, justify funding Recognizing progress; measuring change  Reliable information collected in a systematic way Program improvement  Spot and correct problems  Strengthen the intervention  Determine where to allocate limited resources Why Measure Performance?

Overview of Prohibited Activities Performance Measures 1.Strategy and Planning  Use for planning and program development o Determine what you hope to achieve at different time periods  Develop site applications and Memoranda of Understanding templates (If working with sites) o Consider how partnerships will work to achieve outputs and outcomes 2.Site Recruitment and Selection  Define expectations  Assess site applications; determine which sites are best able to implement intervention Using Performance Measurement in Program Management

Overview of Prohibited Activities Performance Measures 3.Program Implementation  In conversations with staff, sites, and participants at trainings and meetings o Reiterate need, intervention, outcomes to ensure mutual understanding  Site monitoring  Continuous program improvement 4.Using Results/Data and Reporting  Progress reports  Marketing and promotion  CELEBRATE SUCCESS!!! Using Performance Measurement in Program Management continued

Overview of Prohibited Activities Performance Measures – Outputs and Outcomes OUTPUTS  Amount of services being provided o People served o Products created o Programs developed OUTCOMES  Reflects the changes or benefits that occur  Can reflect changes in individuals, organizations, communities, or environment  Address changes in attitudes/beliefs, knowledge/skill, behavior, or conditions

Overview of Prohibited Activities Performance Measures – Outcomes Attitude/ Belief Increase interest in School Knowledge/ Skill Improve math ability Behavior Increase school attendance Condition Successful completion of High School Outcome Example: EDUCATION

Overview of Prohibited Activities Performance Measures - Alignment Aligning Theory of Change with Performance Measures

Overview of Prohibited Activities Performance Measures - Alignment Alignment: Logical flow and connection between the different elements. Alignment helps you:  Strengthen your theory of change  Articulate you performance measures  Measure more accurately  Cleary report change

Overview of Prohibited Activities Performance Measures - Alignment Example: Aligned Output-Outcome with Performance Measure Focus Area Education Output Number of children that completed participation in CNCS- supported early childhood education programs. (ED21) Outcome Number of children demonstrating gains in school readiness in terms of social and/or emotional development.(ED23)

Overview of Prohibited Activities Performance Measures – Impact Evaluation Differences Between Impact Evaluations and Performance Measures Key Areas Performance Measurement Impact Evaluation Causality Assumes causality; does not “prove” theory of change Seeks to show causality and “prove” theory of change Implementation Tracks outputs and outcomes on a regular outgoing basis May occur from time to time, but not on a regular ongoing basis Time Focus Shorter term focus; what can be measured within one year Longer term focus Design Balance rigor with practicality Uses most rigorous evaluation design

Overview of Prohibited Activities Performance Measures- Summary  Performance measurement is a systematic process of measuring progress (outputs and outcomes)  Performance measurement does not seek to “prove” a theory of change, but can provide snapshots.  Impact evaluation can determine if results occurred because of the intervention.  Strong performance measures align with the theory of change (need, intervention and outcome) Key Points

Overview of Prohibited Activities Performance Measures- Summary CNCS Priorities and Performance Measures: Program-Specific Notices of Funding Opportunities and Application Instructions: Additional Resources

Project Director Training Saratoga, NY October 7 th – 9 th, 2013 Questions?