Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH MCH EPI Program Team Leader Division of Reproductive.

Slides:



Advertisements
Similar presentations
Engaging Your Community in Practical Program Evaluation Stephanie Welch, MS-MPH, RD, LDN Knox County Health Department Phone (865)
Advertisements

Introduction to Monitoring and Evaluation
From Research to Advocacy
Donald T. Simeon Caribbean Health Research Council
Measuring Value: Using Program Evaluation to Understand What’s Working -- Or Isn’t Juliana M. Blome, Ph.D. , MPH Office of Program Analysis and Evaluation.
P re-Referral and Referral Activities Produced by NICHCY, 2014 Module 3.
What You Will Learn From These Sessions
Building the Digital Infrastructure for Vermont’s Learning Health System ONC HIT Policy Committee Testimony September 14, 2011 Hunt Blair, Deputy Commissioner.
Designing an Effective Evaluation Strategy
Program Evaluation and Measurement Janet Myers. Objectives for today… To define and explain concepts and terms used in program evaluation. To understand.
Ray C. Rist The World Bank Washington, D.C.
Decision-Making and Strategic Information Workshop on M&E of PHN Programs July 24-August 11, 2006 Addis Ababa.
Logic Models. A logic model is your program ROAD MAP. Where are you trying to go? How are you trying to get there? What will tell you that you’ve arrived?
Reality Check For Extension Programs Deborah J. Young Associate Director University of Arizona Cooperative Extension.
Evaluation. Practical Evaluation Michael Quinn Patton.
Purpose of the Standards
Evaluating Physical Activity Intervention Programs Thomas Schmid, PhD Physical Activity and Health Branch CDC Atlanta, Georgia, USA.
CDC Evaluation Process Harlen Hays, MPH Office of Health Promotion, KDHE.
School Development Planning Initiative “An initiative for schools by schools” Self-Evaluation of Learning and Teaching Self-Evaluation of Learning and.
How to Develop the Right Research Questions for Program Evaluation
“Walking Through the Steps and Standards” Presented by: Tom Chapel Focus On…
Health Resources and Services Administration Maternal And Child Health Bureau Healthy Start What’s Happening Maribeth Badura, M.S.N. Dept. of Health and.
COMMUNITY HEALTH NURSING
Program Collaboration and Service Integration: An NCHHSTP Green paper Kevin Fenton, M.D., Ph.D., F.F.P.H. Director National Center for HIV/AIDS, Viral.
The County Health Rankings & Roadmaps Take Action Cycle.
Sociology 3322a. “…the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards.
Too expensive Too complicated Too time consuming.
Program Evaluation EDL 832 Jeffrey Oescher, Instructor 6 June 2013.
Project design & Planning The Logical Framework Approach An Over View Icelandic International Development Agency (ICEIDA) Iceland United Nations University.
HEALTH HOMES ARKANSAS DEPARTMENT OF HUMAN SERVICES LONG-TERM CARE POLICY SUMMIT SEPTEMBER 5, 2012.
Substance-Exposed Pregnancy Prevention: An Environmental Scan of Local Health Department Practices Molly Schlife, MPH.
2013 NEO Program Monitoring & Evaluation Framework.
1 CHAPTER.
Neal D. Kohatsu, MD, MPH Medical Director May 17, 2012 Evaluation and Quality Improvement.
1 © 2010 Regents of the University of Minnesota. All rights reserved. 11 Logic Models A ROADMAP TO SUCCESS.
Telemedicine Technology Use Plan Brenda Janot Telemedicine Coordinator
Addressing Maternal Depression Healthy Start Interconception Care Learning Collaborative Kimberly Deavers, MPH U.S. Department of Health & Human Services.
Copyright 2012 Delmar, a part of Cengage Learning. All Rights Reserved. Chapter 9 Improving Quality in Health Care Organizations.
HMP Evaluation Overview and Tools UNE-CCPH June 22, 2011 Presenters: Kira Rodriguez Michelle Mitchell Patrick Madden Pamela Bruno-MacDonald.
The Major Steps of a Public Health Evaluation 1. Engage Stakeholders 2. Describe the program 3. Focus on the evaluation design 4. Gather credible evidence.
Monitoring and Evaluation
Incorporating Preconception Health into MCH Services
11 The CPCRN, DCPC, NCI, and the Community Guide: Areas for Collaboration and Supportive Work Shawna L. Mercer, MSc, PhD Director The Guide to Community.
Copyright © 2014 by The University of Kansas A Framework for Program Evaluation.
Evaluation Approaches, Frameworks & Designs HSC 489 – Chapter 14.
Evaluation Revisiting what it is... Who are Stakeholders and why do they matter? Dissemination – when, where and how?
HRSA Health Disparities Collaboratives 2006: Perinatal & Patient Safety Pilot Ada Determan, M.P.H Division of Clinical Quality Bureau of Primary Health.
Evaluation design and implementation Puja Myles
Slide 1 Oregon Smoke Free Mothers and Babies Project Lesa Dixon-Gray, MSW, MPH Office of Family Health (503)
Prepared by: Forging a Comprehensive Initiative to Improve Birth Outcomes and Reduce Infant Mortality in [State] Adapted from AMCHP Birth Outcomes Compendium.
Using Logic Models to Create Effective Programs
Outcome-based Planning and Evaluation Gloria Latimer, Ed.S, Director of Community Programs Jason Vahling, M.P.H., Community Program Specialist.
Erik Augustson, PhD, National Cancer Institute Susan Zbikowski, PhD, Alere Wellbeing Evaluation.
Unit 9: Evaluating a Public Health Surveillance System #1-9-1.
Demonstrating the Value of Stakeholder Input for Evaluation Design, Implementation and Dissemination Yvonne Abel, MS Abt Associates Inc. AEA Annual Conference,
Yvonne Abel, Abt Associates Inc. November 2010, San Antonio, TX Enhancing the Quality of Evaluation Through Collaboration Among Funders, Programs, and.
Building Capacity for EMR Adoption and Data Utilization Among Safety Net Organizations Presented by Chatrian Reynolds, MPH, Evaluator, LPHI Shelina Foderingham,
Copyright © 2010, 2006, 2002 by Mosby, Inc., an affiliate of Elsevier Inc. Chapter 10 Evidence-Based Practice Sharon E. Lock.
Evaluating with Perspective: Mobilizing Community Participation in Program Evaluation …its not too late!
A Systematic Approach for Evaluating Health Related Programs: Adaptation for Community- based Participatory Research and Clinical Application to Reduce.
Overview of Intervention Mapping
Evaluating the Quality and Impact of Community Benefit Programs
Presented by Anne C. Adams, MSW (919) and Bea Sweet
Monitoring and Evaluation Systems for NARS Organisations in Papua New Guinea Day 3. Session 9. Periodic data collection methods.
Strategies to Reduce Antibiotic Resistance and to Improve Infection Control Robin Oliver, M.D., CPE.
Fundamentals of Monitoring and Evaluation
Tim McAfee, M.D., M.P.H. Director, CDC Office on Smoking and Health
MaryCatherine Jones, MPH, Public Health Consultant, CVH Team
A Framework for Program Evaluation
Presentation transcript:

Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH MCH EPI Program Team Leader Division of Reproductive Health, CDC Carolyn B. Slack, MS, RN Director, MCH Division Columbus Health Department

Framework for Program Evaluation in Public Health n Adopted: MMWR 48(RR11) Sept 99 n Summarizes the essential elements n Provides framework n Clarifies the steps n Reviews standards n Addresses misconceptions

Shifts Essential Shifts in Public Health Planning and Policy EFFORT RESULTS RESULTS ACCOUNTABILITY ACCOUNTABILITY BUDGETS

Premises of Reinventing Government n What gets measured gets done. n If you don't measure results, you can't tell success from failure. n If you can't see success, you can't reward it. n If you can't reward success, you're probably rewarding failure. n If you can't see success, you can't learn from it. n If you can't recognize failure, you can't correct it. n If you can demonstrate success, you can win public support. Osborne and Gaebler (1992: chapter 5, "Results- Oriented Government")

Being Effective in Public Health

Program Evaluation Steps Engage Stakeholders Describe the Program Focus the Evaluation Design Gather Credible Evidence Ensure Use and Share Lessons Learned Justify Conclusions Standards: Utility Feasibility Propriety Accuracy

Step 1: Engage Stakeholders Fostering input, participation, and power- sharing among those persons who have an investment n Those involved in program operations n Those served or affected by the program n Primary users of the evaluation

Step 2: Describing the Program n Need n Expected effects n Activities n Resources n Stage of development n Context n Logic model Scrutinizing the features of the program being evaluated, including its purpose and place in a larger context.

Columbus, OH Columbus Health Department Perinatal Program

Program Map Self ReferralOutreach VisitPregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit No Show Outreach Waiting List Referral

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Counseling

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Counseling

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

Step 3: Focusing the Evaluation Design Purpose: n Gain insight n Change practice n Assess impact n Inquiry affects participants Components: n Users n Uses n Questions n Methods n Agreements Planning in advance where the evaluation is headed and what steps will be taken; iterative process.

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

Step 4: Gathering Credible Evidence Compiling information that stakeholders perceive as trustworthy and relevant for answering their questions. n Indicators n Sources of evidence n Quality of information n Quantity of evidence n Logistics

Step 5: Justifying Conclusions Making claims regarding the program that are warranted on the basis of the data. n Stakeholder standards n Analysis and synthesis of findings n Interpretation of the facts n Judgments n Recommendations

Step 6: Ensuring Use and Sharing Lessons Learned Ensuring that stakeholders are aware, findings are used in decisions, and participants have a beneficial experience. n Designed for use n Preparation for findings n Feedback from all parties n Follow-up of users n Dissemination process n Additional Uses

Program Evaluation Steps Engage Stakeholders Describe the Program Focus the Evaluation Design Gather Credible Evidence Ensure Use and Share Lessons Learned Justify Conclusions Standards: Utility Feasibility Propriety Accuracy

Measuring Outcomes: Potential Approaches in Evaluating Public Health

n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

What is the Intervention? Bloom, The evaluation of primary prevention programs in Greenfield and Miller, Comprehensive Mental Health: The Challenge of Evaluation, University of Wisconsin Press. “We are generally asked to evaluate an outcome of an undefined program having unspecified objectives on an often vaguely delineated recipient group whose level or variety of pathology is virtually impossible to assess, either before or after their exposure to the program.”

What is the Intervention?

Domino Approach: What happens in the sequence to make the whole chain of events occur? What is the Theory of Action?

Columbus Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

Choose the Outcome(s) to Measure Example: Smoking Cessation n Ask n Advise n Assess n Assist n Arrange n Stages of Change n Cessation Attempts n Cessation During Pregnancy n Improved Low Birth Weight

Choose the Outcome(s) to Measure n Relates to the Theory of Action n Sufficient Evidence n Measurable n Accurate and Reliable n Feasible in a Public Health Setting n Affordable

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

Choose the Standard or Comparison Group 5 As

Choose the Standard or Comparison Group n Standard n Internal Comparison –Randomized –Systematic n External Comparison

Choose a Comparison Group Internal Starting Event PretestPosttest Inter vention Randomize Select Individual Groups

Choose a Comparison Group External Starting Event PretestPosttest Inter vention

Measuring Outcomes: Potential Approaches in Evaluating Public Health n Understand the Theory of Action n Choose the Outcome(s) to Measure n Choose the Standard or Comparison Group

CityMatCH Urban MCH Leadership Confluence Measuring Outcomes: Potential Approaches in Evaluating Public Health Case Study: “Show Me the Results!”

Midstate Health Department Perinatal Program Program Map Self Referral Outreach Visit Pregnancy Test Risk Assessment Service Plan F/U Visit Delivery F/U Visit Nursing Service Plan Coordination Medical Treatment Social Counseling Nutrition WIC Substance Abuse Intervention Food Stamps Prenatal Care Entry Reduced Risk Reduced Stress Quit Rates Weight Gain Reduced LBW Counseling