Jodi Drisko, MSPH Parametrix Group, LLC

Objectives Increase knowledge of how evaluation fits in with other agency priorities Increase understanding of how to prioritize program activities for evaluation Define measures for effort and impact

Evaluations—Essential Shifts for Public Health EFFORT RESULTS RESULTS ACCOUNTABILITY ACCOUNTABILITY RESOURCES

Effectiveness in Public Health Planning Doing Assessment Strategies PlanImplement Monitoring Evaluation

Evaluations—Why? My funder requires it Everyone is doing it I like collecting data I want to improve my program I need to demonstrate effectiveness Help get more money for future (sustainability, program expansion) It will make my boss happy It fits with PH accreditation

How evaluation links to other priorities ES 9 of the 10 essential services Public health accreditation Quality improvement CDPHE-MCH performance monitoring Other - - -

Successful Evaluation Successful evaluation depends on:  Adequate Program Theory  Literature  Best practices  Adequate Program Design and Implementation  Appropriate Evaluation Design and Measurement

Linking planning and evaluation Start with the end in mind Evaluation starts at the planning stage What will you measure Prioritize Monitoring (objectives, implementation, outcomes) Tie in to performance management and QI Larger outcomes (using existing data whenever possible) Policy, Systems, Environmental changes Tracking -> moving from widget counting to what difference you are making

Prioritizing Can’t measure everything Easy to measure PRAMS, birth cert Fairly easy to collect data or track Most impactful aspects Implementation tracking What are the “make or break” pieces? Outcomes – which are realistic and attainable within certain timeframe?

Where do you start?

Quantity vs. Quality: OUR EFFORTS ARE NOT EQUALLY EFFECTIVE HOW MUCH did we do? HOW WELL did we do it? Process Measures OR Outputs Outcome measures

WHAT RESULTS did we produce? BEST quality RESULTS? Effort vs. Impact: NOT ALL WE DO PRODUCES CHANGE THAT IS MEASURABLE, INTENDED AND FOR THE BETTER

How much did we work? How well did we do our work? What results did we get? Did we get the BEST quality results? EFFORT (process) IMPACT (outcomes) NOT ALL WE DO GETS (INTENDED) RESULTS AND ALL RESULTS ARE NOT EQUALLY EFFECTIVE QUANTITYQUALITY

QUANTITYQUALITY RELATIVE OF MEASURES RELATIVE IMPORTANCE OF MEASURES MOST MOST Important EFFORT (process) IMPACT (outcomes)

12 34 QUANTITYQUALITY RELATIVE CONTROL OF MEASURES LEAST LEAST Control MOST MOST Control EFFORT (process) IMPACT (outcomes)

QUANTITYQUALITY Example: Teen Pregnancy Prevention Program OUTCOME: Reduction in too early sexual activity Indicators: 30% decrease in reported risk behaviors, 0 teen births 40 hours of class for 100 students 8 hours of mentoring per student 90% of students performed “correct” role play 85% of students liked program a lot 80% of students performed significantly better on post-test than pre-test 25% decline in reported risk behaviors, 1 teen pregnancy in targeted population EFFORT (process) IMPACT (outcomes)

Discussion – Ask the experts Formal evaluation Contractor Internal evaluator and epi/data assistance Informal evaluation