1. Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH MCH EPI Program Team Leader Division of Reproductive Health, CDC Carolyn B. Slack, MS, RN Director, MCH Division Columbus Health Department

2. Framework for Program Evaluation in Public Health n Adopted: MMWR 48(RR11) Sept 99 n Summarizes the essential elements n Provides framework n Clarifies the steps n Reviews standards n Addresses misconceptions

3. Shifts Essential Shifts in Public Health Planning and Policy EFFORT RESULTS RESULTS ACCOUNTABILITY ACCOUNTABILITY BUDGETS

4. Premises of Reinventing Government n What gets measured gets done. n If you don't measure results, you can't tell success from failure. n If you can't see success, you can't reward it. n If you can't reward success, you're probably rewarding failure. n If you can't see success, you can't learn from it. n If you can't recognize failure, you can't correct it. n If you can demonstrate success, you can win public support. Osborne and Gaebler (1992: chapter 5, "Results- Oriented Government")

5. Being Effective in Public Health

6. Program Evaluation Steps Engage Stakeholders Describe the Program Focus the Evaluation Design Gather Credible Evidence Ensure Use and Share Lessons Learned Justify Conclusions Standards: Utility Feasibility Propriety Accuracy

7. Step 1: Engage Stakeholders Fostering input, participation, and power- sharing among those persons who have an investment n Those involved in program operations n Those served or affected by the program n Primary users of the evaluation

8. Step 2: Describing the Program n Need n Expected effects n Activities n Resources n Stage of development n Context n Logic model Scrutinizing the features of the program being evaluated, including its purpose and place in a larger context.

9. Columbus, OH Columbus Health Department Perinatal Program

10. Program Map Self ReferralOutreach VisitPregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit No Show Outreach Waiting List Referral

11. Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Counseling

12. Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Counseling

13. Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

14. Step 3: Focusing the Evaluation Design Purpose: n Gain insight n Change practice n Assess impact n Inquiry affects participants Components: n Users n Uses n Questions n Methods n Agreements Planning in advance where the evaluation is headed and what steps will be taken; iterative process.

15. Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

16. Step 4: Gathering Credible Evidence Compiling information that stakeholders perceive as trustworthy and relevant for answering their questions. n Indicators n Sources of evidence n Quality of information n Quantity of evidence n Logistics

17. Step 5: Justifying Conclusions Making claims regarding the program that are warranted on the basis of the data. n Stakeholder standards n Analysis and synthesis of findings n Interpretation of the facts n Judgments n Recommendations

18. Step 6: Ensuring Use and Sharing Lessons Learned Ensuring that stakeholders are aware, findings are used in decisions, and participants have a beneficial experience. n Designed for use n Preparation for findings n Feedback from all parties n Follow-up of users n Dissemination process n Additional Uses

19. Program Evaluation Steps Engage Stakeholders Describe the Program Focus the Evaluation Design Gather Credible Evidence Ensure Use and Share Lessons Learned Justify Conclusions Standards: Utility Feasibility Propriety Accuracy

20. Measuring Outcomes: Potential Approaches in Evaluating Public Health

21. n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

22. Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

23. What is the Intervention? Bloom, 1968. The evaluation of primary prevention programs in Greenfield and Miller, Comprehensive Mental Health: The Challenge of Evaluation, University of Wisconsin Press. “We are generally asked to evaluate an outcome of an undefined program having unspecified objectives on an often vaguely delineated recipient group whose level or variety of pathology is virtually impossible to assess, either before or after their exposure to the program.”

24. What is the Intervention?

25. Domino Approach: What happens in the sequence to make the whole chain of events occur? What is the Theory of Action?

26. Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

27. Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

28. Choose the Outcome(s) to Measure Example: Smoking Cessation n Ask n Advise n Assess n Assist n Arrange n Stages of Change n Cessation Attempts n Cessation During Pregnancy n Improved Low Birth Weight

29. Choose the Outcome(s) to Measure n Relates to the Theory of Action n Sufficient Evidence n Measurable n Accurate and Reliable n Feasible in a Public Health Setting n Affordable

30. Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

31. Choose the Standard or Comparison Group 5 As

32. Choose the Standard or Comparison Group n Standard n Internal Comparison –Randomized –Systematic n External Comparison

33. Choose a Comparison Group Internal Starting Event PretestPosttest Inter vention Randomize Select Individual Groups

34. Choose a Comparison Group External Starting Event PretestPosttest Inter vention

35. Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

36. CityMatCH Urban MCH Leadership Confluence Measuring Outcomes: Potential Approaches in Evaluating Public Health Case Study: “Show Me the Results!”

37. Midstate Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling