What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference: Evaluation and Influence Sydney Sept. 2011

The journey What is the true impact of evaluation?

The evidnece The evidence is confusing Do we add value ?

Changes organisations - Williams Develops learning environment - Preskill Evaluation willingness and capacity leads to increased program success -Clinton Influences individual, interpersonal & the collective- Marks & Henry Empowers –Fetterman; Evaluation attributes impact on organisations- Appleton Changes organisations - Williams Develops learning environment - Preskill Evaluation willingness and capacity leads to increased program success -Clinton Influences individual, interpersonal & the collective- Marks & Henry Empowers –Fetterman; Evaluation attributes impact on organisations- Appleton

Evaluation research methods What's our standard?

Our impact KNOW THY IMPACT!

Click to edit Master title style This study

The programs & the evaluation 5yr evaluation of a district-wide plan in NZ, aimed at mobilizing a community to prevent the onset of Type II Diabetes. *10 action areas: Over a 150 different initiatives over a 4 year period Nelson Marlborough Nutrition & Physical Activity: Collaborate as a community to reverse the tide of obesity. *5 Action Areas: Over a 100 different initiatives over a 3 year period EVALUATION METHODS:  An adapted form of the CDC Evaluation Framework Standard dimensions  Multiple forms of data collection :reports, surveys, case studies interviews, observations and internal data  Key information from 292 initiatives was coded using scoring rubric EVALUATION METHODS:  An adapted form of the CDC Evaluation Framework Standard dimensions  Multiple forms of data collection :reports, surveys, case studies interviews, observations and internal data  Key information from 292 initiatives was coded using scoring rubric

A research lens and the evaluation THIS STUDY:  Analysed year by year and over 4 years.  Determine relationships: Factor analysis, Path models THIS STUDY:  Analysed year by year and over 4 years.  Determine relationships: Factor analysis, Path models Scrubbed the data Determined the strength & validity of the dimension Identified outcomes Looked for causal relationships Scrubbed the data Determined the strength & validity of the dimension Identified outcomes Looked for causal relationships

MEASURES Program KPI Adaptation Degree of implementation Outcome Progress Sustainability Process Organisational development Collaboration Evaluation Program KPI Adaptation Degree of implementation Outcome Progress Sustainability Process Organisational development Collaboration Evaluation STANDARD SETTING SCOREEXEMPLAR KPI 1-3 Only 30% achieved. Only ‘easy’ KPIs indicator achieved 4-7 Achieved 40% mix of ‘hard and easy’ indicators % achieved mixture of levels of difficulty

THE MODEL

So what does this mean? SUSTAINABILITY: KPI  ADAPTATION --IVE  DOI  OD  COLLABORATION  EVALUATION  SUSTAINABILITY: KPI  ADAPTATION --IVE  DOI  OD  COLLABORATION  EVALUATION  PROGRESS: KPI  ADAPTATION  DOI  OD  COLLABORATION –IVE  EVALUATION  PROGRESS: KPI  ADAPTATION  DOI  OD  COLLABORATION –IVE  EVALUATION 

The outcome variables SUSTAINABILITY- WEIGHTING VARIES ACCORDING TO: TIME: DOI, COLLABORATION PROGRAM:OD, COLLABORATION, EVALUATION SUSTAINABILITY- WEIGHTING VARIES ACCORDING TO: TIME: DOI, COLLABORATION PROGRAM:OD, COLLABORATION, EVALUATION PROGRESS- WEIGHTING VARIES ACCORDING TO: TIME:ADAPTATION,OD, DOI PROGRAM: OD, COLLABORATION PROGRESS- WEIGHTING VARIES ACCORDING TO: TIME:ADAPTATION,OD, DOI PROGRAM: OD, COLLABORATION

EVALUATION IS AN EQUATION Program Outcome Imp Process Dosage Intervention Adaptation Evaluation influence CONTEXT

Final word EVALUATION IS AN INTERVENTION EVALUATION STANDARD METHODS EVALUATION STANDARD METHODS EVALUATION ADDS VALUE EVALUATORS - KNOW THY IMPACT

Thank you