Evaluation Plans & Performance Indicators Office of Research, Evaluation, and Policy Studies Marcella M. Reca Zipp November 30, 2010

 Necessity of evaluation plan  Types of evaluation plans  Components of an evaluation plan  Performance indicators  Reporting requirements  Sample evaluation plans

Purpose of Evaluation Plan  Cohesive approach to conducting evaluation and using results  Explains what, when, how, why, who  Documents the evaluation process  Ensures implementation fidelity  Promotes a participatory approach Source: University of Toronto

3 Levels of Evaluation  Project-Level Evaluation Context Implementation Outcome  Cluster Evaluation  Program and Policymaking Evaluation

Project-Level Evaluation  Context Needs, assets, and resources of community Political atmosphere, social and environmental strengths/weaknesses  Implementation Critical components/activities of project Aspects that are strengths and weaknesses How do the components connect to goals and outcomes  Outcome Critical outcomes you are trying to achieve Impact on clients, community, etc. Unexpected impact

Cluster Evaluation  Determines how well the collection of projects fulfills the objective of systemic change.  Not a substitute for project-level evaluation.  Looks across a group of projects to identify common themes.  Information reported in aggregate form to granting organization. EVALUATOR PROJECT APROJECT BPROJECT C

Program and Policy Making Evaluation  Macro form of evaluation.  Utilizes information gathered from both project-level and cluster evaluation to make effective decisions about program funding and support.  Supports communities in creating policy change at local, state, and federal levels.

Elements of an Evaluation  Introduction  Project Objectives  Logic Model  Partnership Roles and Responsibilities  Intervention Programming/Research  Methodology/Data Collection  Instrumentation Measures

Introduction  Provides background information for the evaluation, identifies purpose and goal, sets the course on the evaluation road map. Evaluation purpose and goals  What does the evaluation strive to achieve? Evaluation team  Who is the evaluation coordinator?  Who are the members of the evaluation team?

Logic Model  Graphic depiction of the program description.  Links needs, objectives, activities, measurements.  Provides scope of program.  Ensures systematic decisions are made about what will be measured.  Identifies and organizes indicators.

Objectives  PIMO method  The number of objectives will be determined by purpose (i.e., intervention, treatment, prevention).  Interrelated to your projected activities (i.e., education, service, research).  Feasible to collect, will provide accurate results.

Partnership Roles and Responsibilities  Project partners are expected to provide certain, unique expertise to your project activities, either in direct service function or as advisory units.  Identify each partner in their role and responsibility in terms of involvement for your project.  Partner-cited activities must be evaluated formatively and summatively.

Intervention Programming  Identify one or more intervention strategies used to support project activities and anticipated outcomes.  Cite if the program is on the federal evidence-based initiative list: EBIEBI  For market-available programs that require training and certification of direct service providers, provide a timetable for acquiring training before intervention can be used.

Performance Indicators  Visible, measurable signs of program performance.  Relevant, understandable and useful.  Reflect program objectives, logic model and evaluation questions.  Define success  Reasonable expectations of program performance. Source: University of Toronto

Performance Indicators cont.  Other terms – industry jargon Key Performance Indicator (KPI) Performance metric Performance standard Balanced Scorecard Quality indicators  All are different words for the same thing: measure performance.

Data Collection  What methods will be used?  How often will data be collected?  Who will collect the data?  Validity and reliability of data sources Baseline data Outcomes-based triangulation Quality assurance Design (experimental, quasi- experimental, etc.)

Instrumentation Measures  Tools for data collection Only collect the information you need Easy to administer and use Pilot test tools before use in the evaluation  Human Subjects Considerations IRB, school board approval  Data management and storage Confidentiality and data quality

Tips & Helpful Hints  Be realistic In your assessment of resources In your timeline  Seek help  Use templates, tables, or guides that may be provided in the RFP or model after past funded proposals.

Reporting and Dissemination  Dissemination How will you disseminate findings? Who is responsible? How, where, when will findings be used?  Reporting Formative reports – quarterly, biannually Summative reports – final report/end of project Project deliverables

Sample RFP Evaluation Plan  Two examples of an evaluation plan within an RFP General, limited specifications Complex, very detailed

Evaluation Resources  CDC:  University of Toronto:  W.K. Kellogg Foundation:  Connell, J.P., Kubisch, A.C., Schorr, L.B., Weiss, C.H. (1995). New Approaches to Evaluating Community Initiatives, New York, NY: Aspen Institute.  Shadish, W.R., Cook, T.D., Leviton, L.C. (1991). Foundations of Program Evaluation. Newbury Park, CA: Sage Publications.  Taylor-Powell, E., Steele, S., Douglas, M. (1996). Planning a Program Evaluation. Madison, WI: University of Wisconsin Cooperative Extension.