Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH MCH EPI Program Team Leader Division of Reproductive Health, CDC Carolyn B. Slack, MS, RN Director, MCH Division Columbus Health Department

Framework for Program Evaluation in Public Health n Adopted: MMWR 48(RR11) Sept 99 n Summarizes the essential elements n Provides framework n Clarifies the steps n Reviews standards n Addresses misconceptions

Shifts Essential Shifts in Public Health Planning and Policy EFFORT RESULTS RESULTS ACCOUNTABILITY ACCOUNTABILITY BUDGETS

Premises of Reinventing Government n What gets measured gets done. n If you don't measure results, you can't tell success from failure. n If you can't see success, you can't reward it. n If you can't reward success, you're probably rewarding failure. n If you can't see success, you can't learn from it. n If you can't recognize failure, you can't correct it. n If you can demonstrate success, you can win public support. Osborne and Gaebler (1992: chapter 5, "Results- Oriented Government")

Being Effective in Public Health

Program Evaluation Steps Engage Stakeholders Describe the Program Focus the Evaluation Design Gather Credible Evidence Ensure Use and Share Lessons Learned Justify Conclusions Standards: Utility Feasibility Propriety Accuracy

Step 1: Engage Stakeholders Fostering input, participation, and power- sharing among those persons who have an investment n Those involved in program operations n Those served or affected by the program n Primary users of the evaluation

Step 2: Describing the Program n Need n Expected effects n Activities n Resources n Stage of development n Context n Logic model Scrutinizing the features of the program being evaluated, including its purpose and place in a larger context.

Columbus, OH Columbus Health Department Perinatal Program

Program Map Self ReferralOutreach VisitPregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit No Show Outreach Waiting List Referral

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Counseling

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Counseling

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

Step 3: Focusing the Evaluation Design Purpose: n Gain insight n Change practice n Assess impact n Inquiry affects participants Components: n Users n Uses n Questions n Methods n Agreements Planning in advance where the evaluation is headed and what steps will be taken; iterative process.

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

Step 4: Gathering Credible Evidence Compiling information that stakeholders perceive as trustworthy and relevant for answering their questions. n Indicators n Sources of evidence n Quality of information n Quantity of evidence n Logistics

Step 5: Justifying Conclusions Making claims regarding the program that are warranted on the basis of the data. n Stakeholder standards n Analysis and synthesis of findings n Interpretation of the facts n Judgments n Recommendations

Step 6: Ensuring Use and Sharing Lessons Learned Ensuring that stakeholders are aware, findings are used in decisions, and participants have a beneficial experience. n Designed for use n Preparation for findings n Feedback from all parties n Follow-up of users n Dissemination process n Additional Uses

Program Evaluation Steps Engage Stakeholders Describe the Program Focus the Evaluation Design Gather Credible Evidence Ensure Use and Share Lessons Learned Justify Conclusions Standards: Utility Feasibility Propriety Accuracy

Measuring Outcomes: Potential Approaches in Evaluating Public Health

n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

What is the Intervention? Bloom, The evaluation of primary prevention programs in Greenfield and Miller, Comprehensive Mental Health: The Challenge of Evaluation, University of Wisconsin Press. “We are generally asked to evaluate an outcome of an undefined program having unspecified objectives on an often vaguely delineated recipient group whose level or variety of pathology is virtually impossible to assess, either before or after their exposure to the program.”

What is the Intervention?

Domino Approach: What happens in the sequence to make the whole chain of events occur? What is the Theory of Action?

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

Choose the Outcome(s) to Measure Example: Smoking Cessation n Ask n Advise n Assess n Assist n Arrange n Stages of Change n Cessation Attempts n Cessation During Pregnancy n Improved Low Birth Weight

Choose the Outcome(s) to Measure n Relates to the Theory of Action n Sufficient Evidence n Measurable n Accurate and Reliable n Feasible in a Public Health Setting n Affordable

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

Choose the Standard or Comparison Group 5 As

Choose the Standard or Comparison Group n Standard n Internal Comparison –Randomized –Systematic n External Comparison

Choose a Comparison Group Internal Starting Event PretestPosttest Inter vention Randomize Select Individual Groups

Choose a Comparison Group External Starting Event PretestPosttest Inter vention

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

CityMatCH Urban MCH Leadership Confluence Measuring Outcomes: Potential Approaches in Evaluating Public Health Case Study: “Show Me the Results!”

Midstate Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling