Evaluability Assessment, Formative & Summative Evaluation Laura C. Leviton, Ph.D. Senior Advisor for Evaluation

■ Summative evaluation: Designing and using evaluation to judge merit ■ Formative evaluation: Designing and using evaluation to improve intervention ■ Evaluability assessment: Assessing whether the intervention is ready to be managed for results what changes are needed to do so whether evaluation would contribute to improved performance Definitions

Resources Wholey, Hatry & Newcomer, Handbook of Practical Program Evaluation, Wiley 2010 Leviton et al., Evaluability assessment to improve public health. In Annual Review of Public Health, Volume 31: Leviton, Kettel Khan & Dawkins, New Directions in Evaluation, No. 125, January 2010 Chapter 3 has templates, procedures

Evaluations are Often Handed to Us “Here, evaluate this.”

Some of the Interventions Just Aren’t Very Good “Here, evaluate this.”

Why? Perhaps 95% of interventions are not fully developed at the time of evaluation. Also, ongoing problems with measurement, design and analysis. Wilson and Lipsey, 2001 review of 319 meta-analyses: Proportion of effect sizes associated with study features Variance accounted for: Study methods = features of the intervention Biggest sources: research design, operationalizing the dependent variable, sampling error

Don’t Rush to Summative Evaluation It’s just not cost-effective. Good outcome studies are the culmination of careful work Start with formative evaluation, or better yet, Evaluability assessment.

Steps in Evaluability Assessment 1. Involve intended users of evaluation information 2. Clarify the intended intervention 3. Explore intervention reality 4. Reach agreement on needed changes in activities or goals 5. Explore alternative evaluation designs 6. Agree on evaluation priorities and intended uses of information.

Too Linear – It Works Like This:

Early Steps

Middle Steps

Logic Models InputsOutputsOutcomes (Impact) Resources & Staff Activities, Outreach to Target Group Products, Target Group Participation Achieve Short Term Objectives Achieve Intermediate Objectives Achieve Long Term Objectives

Theory of Change GoalsStrategiesTarget GroupIf-Then Statements Short Term Outcomes Long Term Outcomes StatementsHow goal will be accomplished Define groupIf (activity) then outcome MeasurementWas strategy achieved? Were participants in the target group? Did the activity result in the outcome? Was short term objective achieved? Was long term objective achieved? Data SourcesE.g., program records E.g., survey of group

Later Steps

What Now? Intervention development Data collection to inform improvements When to do summative (outcome) studies: Logic model or TOC is sharpened and agreed to The model looks like the reality, and vice versa It’s plausible to achieve the outcome(s) Formative evaluation indicates intermediate steps are being accomplished.

Measurement Nick Barber 15 th February 2013

Evaluation of Technologies Nick Barber 15 th February 2013

Presentation title set in header 18 When do we measure the effectiveness of this system?

Presentation title set in header 19 Structured approach to IT evaluation (after Cornford) TechnicalStakeholdersInstitution Structure Process Most IT evaluations here Outcome

Evaluation of Technologies Nick Barber 15 th February 2013