Avoiding Evaluation “Detours” Using the CDC Evaluation Framework

Avoiding Evaluation “Detours” Using the CDC Evaluation Framework
Presented to: Hepatitis Coordinators 2003Conference January 30, 2003 Welcome Present the materials Explain timing, focus Logistics By: Thomas J. Chapel, MA, MBA Office of the Director Office of Program Planning and Evaluation Centers for Disease Control and Prevention

Objectives Typical minefields in evaluation
Self-explanatory We’ll start with a quick definition Read definition Note that it is “systematic” which means disciplined and objective Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation Define PROGRAM broadly as well Typical minefields in evaluation CDC Evaluation Framework as a means to avoid mines Key insights in each steps in the Framework Illustrate Framework with some hepatitis examples

Why We Evaluate “... The gods condemned Sisyphus to endlessly roll a rock up a hill, whence it would return each time to its starting place. They thought, with some reason, that there was no punishment more severe than eternally futile labor....” The Myth of Sisyphus Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 2

Defining Evaluation Evaluation is... Program is...
Self-explanatory We’ll start with a quick definition Read definition Note that it is “systematic” which means disciplined and objective Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation Define PROGRAM broadly as well Evaluation is... the systematic investigation of the merit, worth, or significance of an “object” – Michael Scriven Program is... any organized public health action/activity

Research vs. Program Evaluation
We’ll start with a quick definition Read definition “the systematic investigation of the merit, worth, or significance of an “object” Note that it is “systematic” which means disciplined and objective We won’t define “worth”or “merit” since, as we’ll see, that can vary depending on the situation Read definition of PROGRAM Note that PROGRAM is broadly defined. Virtually any type of organized public health activity can be evaluated. A continuum, not a dichotomy, but at far ends may differ in: Framework and steps Decision making Standards Key questions Design Data collection sources and measures Analysis timing and scope Role of values in making judgments Centrality of attribution as conclusion Audiences for dissemination of results

The Continuum Efficacy…does my effort work in ideal circumstances
Self-explanatory We’ll start with a quick definition Read definition Note that it is “systematic” which means disciplined and objective Note that we don’t define “worth”/“merit” since as we’ll see that can vary in any situation Define PROGRAM broadly as well Efficacy…does my effort work in ideal circumstances Effectiveness…does my effort work in real world settings, and work the same way across settings Implementation fidelity…is my (efficacious and effective) effort being implemented as intended.

Top Detours on the Road to Good Evaluation!
Today’s Focus Top Detours on the Road to Good Evaluation! Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 6

Not linking planning and evaluation…
Detour # 8 Not linking planning and evaluation… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 7

Evaluating only what you can measure…
Detour # 7 Evaluating only what you can measure… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 8

You Get What You Measure…
“…In Poland in the 1970s, furniture factories were rewarded based on pounds of product shipped. As a result, today Poles have the world’s heaviest furniture…” (New York Times, 3/4/99) Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 9

Thinking evaluatively only at the end…
Detour # 6 Thinking evaluatively only at the end… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 10

Good program evaluation shifts our focus from
When to Evaluate…. Good program evaluation shifts our focus from “Did it (my effort) work?” to “Is it (my effort) working?” Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 11

Not asking “ who (else) cares…
Detour # 5 Not asking “ who (else) cares… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 12

Neglecting intermediate outcomes…
Detour # 4 Neglecting intermediate outcomes… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 13

Forgetting Intermediate Outcomes
14

Neglecting process evaluation…
Detour # 3 Neglecting process evaluation… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 15

Confusing attribution and contribution…
Detour # 2 Confusing attribution and contribution… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 16

“Networked” Interventions

Using more “sticks” than “carrots”…
Detour # 1 Using more “sticks” than “carrots”… Self-explanatory Probably used to think about “program” as things like “interventions” or investigations, but important to see this range as “program” and worthy of evaluation. Indeed, for people in managerial roles, these definitions of “program” are probably more like what you are called on to improve and enhance through systematic use of feedback. – MMWR Framework for Program Evaluation in Public Health 18

Framework for Program Evaluation
And so we have a complete picture of the CDC Eval Framework. A series of six steps which cycle back. Four clusters of standards that act as a template for each step to guide each step and measure result. 19

Broadening Our Thinking About Evaluation
What to evaluate When to evaluate Who should be involved in evaluation How to evaluate 20

Standards for Effective Evaluation
Four groups encompassing 30 standards. Not HOW TO do an evaluation step, but how to choose best option from among many options at each step. At each step, best option is the one which will maximize: Utility (7): Choices which best serve information needs of intended users Feasibility (3): Choices which are most consistent with logistics and resources Propriety (8): Choices most consistent with law, ethics, and due regard for the welfare of those involved and affected Accuracy (12): Choices which best reveal and convey technically accurate information 21

Eval F’work Integrates Program and Technical Staff and Planning and Evaluation
And so we have a complete picture of the CDC Eval Framework. A series of six steps which cycle back. Four clusters of standards that act as a template for each step to guide each step and measure result. 22

Step 1: Why Involve Stakeholders?
Pre-Evaluation: Early identification of disagreements in… Definition of the problem Priority activities Priority outcomes What constitutes “proof” of success Post-Evaluation: Get their help with.. Credibility of findings Access to key players Follow-up Dissemination of results So, we can see that stakeholders influence what we measure and also how we measure. 23

Step 2: Program Description: How Logic Models Help Evaluation
Clarity for YOU and/or clarity between you and stakeholders on: What are activities What are intended effects What is the sequence/order of intended effects Which activities are to produce which effects Helps in making decisions on where to focus “outcome” evaluation (effects) and “process” evaluation (activities) Because the logic modeling approach integrates planning and evaluation from the start, once the logic model is done, about 90 percent of the decisions to be made about evaluation design are also made. Key decision left to be made is where in the logic model to focus the evaluation. We’ll discuss this in detail in step 3. But the key decisions are First, how far out to evaluate: This is similar to the issue of staking the claim, but because measurement is involved, get into more logistical considerations. Issues of accountability and reasonableness have been broached already in staking the claim. Measurement may limit the scope of the evaluation, BUT this should never be a long-term constraint. Rather, a decision that because we cannot measure this NOW, we cannot take our evaluation out as far as we need to. BUT, we do need to determine in time how to measure the important effects. Or else, why bother doing the program at all. 24

Step 3: Focusing the Design: What Question Is Being Asked?
What intervention was actually delivered? Were impacts and outcomes achieved? Was the intervention responsible for the impacts and outcomes? These are the three main evaluation questions. The first is the classic process evaluation question. The second is the classic impact and outcome evaluation question. The third is the “causal attribution” question and seeks to know not only if the outcomes were achieved, but can the intervention be assigned the credit for doing so? Important to note that causal attribution is not always the evaluation question of import in an evaluation. And, even if it is, there are situations where the techniques of causal attribution “get in the way” of evaluation and we are not able to make causal statements except with great difficulty. Still, it is fair to think of causal attribution as the “gold standard” for an evaluation where it can be obtained.

Some Factors Re: Choice of Evaluation “Effects” Focus
Lots of this will have been decided when you “staked the claim” but worth considering again. How far out the chain is determined by several considerations. These are not a formula that will produce a result, but rather separate conversations which may lead to contradictory conclusions. In that case, then a need to reconcile where to put the resources. Key questions include: Ripple effect or reasonableness: This one should have been answered in “staking the claim”. The question is how far out the chain of causation is it reasonable to expect the intervention to have an impact. Through the process of refinement in staking the claim, the program should have made decisions about synchronizing the level and mix of activities and the desired “claim.” Accountability: This is what the major stakeholders are expecting of you. As mentioned, they might be “wrong” and err in either a too ambitious or not ambitious enough direction. Still, in designing an evaluation, the information of most import to the stakeholder is a major consideration. Measurability: As mentioned, this should never be the reason to scale back an evaluation, except as an admission that NOW we are unable to measure this effect. Users and uses – Who wants the information and what are they interested in? Accountability to (other) stakeholders – For what effects are key stakeholders expecting to see results? Resources – Time, money, expertise Stage of development –How long has the program been in existence? “Ripple effect”- How far out would an intervention of this intensity reasonably be expected to have an effect?

Setting Factors Re: Choice of Evaluation “Process” Focus
What are the likely key challenges to “implementation fidelity? “Dropped baton” issues are key Partner failed to do their part Client/family/patient failed to fulfill their referral Other common challenges Inadequate dosage Bad access Failure to retain participants Wrong match of staff and participant These are some of the typical questions that process evaluation seeks to answer. The general question is what needs to be in place for the intervention to work, but the types of things that might be barriers may vary with the intervention as listed: Number is the dosage effect question. Access is a classic problem and refers to physical, financial, and psychological access to the intervention (i.e. cultural competency issues). Retention is related to the dose effect but also refers to situations where the intervention is a sequence of activities–i.e. outreach leads to screening leads to treatment. Because the effects don’t happen until the last activity is completed, then attrition between any two stages becomes a process concern. Staff is classic process issue in that all aspects of the intervention may be well designed, but if the wrong persons are delivering it or if the delivery is widely inconsistent, then results will be hurt. 27

Step 4: Gathering Evidence: Choosing Data Collection Methods
Typical factors might be: Time Cost Sensitivity of the issue “Hawthorne effect” Ethics Validity Reliability Different ways of expressing the four evaluation standards Usually trade-off of accuracy and feasibility PROMPT CLASS: What are the various factors that might enter into our decision to use one data collection method as opposed to another? TALKING POINTS: Time Cost Sensitivity of the issue “Hawthorne effect”: Tendency of people to change their behavior when observed Ethics Validity: Degree to which the data collection method reliably captures the concept being measured Reliability: Degree to which the data collection method consistently measures each time 28

Step 5: Justifying Conclusions: Some Standards for Valuing Evidence
I won’t consider this project to have been successful unless….. I can attribute any changes to the project The project reduces disparities The project leaves a “legacy” The project can be sustained long-term But, not just: “what is success”, but also “how can we be confident that we have achieved it?’ Per prior discussion, a very high standard is the FIRST ONE–performance via a control group. BUT, note the other standards that may be equally acceptable.. Note in particular: Changes over time. Even in the absence of a control group, can we assert success if we determine changes are occurring the longer the program is in existence? Finding other ways to explain away confounding factors. One pseudo-comparison group may be standards of expectation from other similar programs like this. I.e., If all programs dealing with teen smoking achieve a 10% reduction in Year 1-2 and we do 15%, can we say we are having some success? 29

Step 5: Justifying Conclusions: Claims About Intervention Effectiveness
Performance vs. a comparison/control group Time sequence Plausible mechanisms (or pathways toward change) Accounting for alternative explanations Similar effects observed in similar contexts But, not just: “what is success”, but also “how can we be confident that we have achieved it?’ Per prior discussion, a very high standard is the FIRST ONE–performance via a control group. BUT, note the other standards that may be equally acceptable.. Note in particular: Changes over time. Even in the absence of a control group, can we assert success if we determine changes are occurring the longer the program is in existence? Finding other ways to explain away confounding factors. One pseudo-comparison group may be standards of expectation from other similar programs like this. I.e., If all programs dealing with teen smoking achieve a 10% reduction in Year 1-2 and we do 15%, can we say we are having some success? 30

Step 6: Maximizing Use of Results
Step 1 creates “market” for the product from the start. Remaining issues are ones of format, tone, and medium, such as: Who is the audience? What will be of greatest importance to them? How will they use the information provided? How much time will they be willing to spend reading and assimilating the material? What type of vocabulary will express the information most clearly? Note the similarity of these questions to some of the initial stakeholder questions. When good stakeholder analysis has been done, then this step becomes almost moot. 31

Applying the CDC F’work
VHIP Global Model VHIP Component Models Note the similarity of these questions to some of the initial stakeholder questions. When good stakeholder analysis has been done, then this step becomes almost moot. 32

Describing the Program: VHIP
Short-Term ID High-risk Clients Offer Vaccine Track Dosage/Follow-up & Refusals Hepatitis C Testing & FUs INPUTS Coop Internal/External Programs Trained Staff Vaccine Access & Delivery System Data collection/ Dose tracking Activities Risk Assessment Counseling/Education Testing Vaccination Referrals Long-Term Reduce morbidity & mortality of viral hepatitis & Functional client database

Describing the Program: Outcomes
Inputs + Activities = Short Term & Long Term Effects ID HR Groups Offer Vaccine Data Tracking Hepatitis C Testing & FU Reduce Morbidity & Mortality of Viral Hepatitis

Focus the Evaluation Design: VHIP Key Objectives
Inputs Collaboration w/relevant programs (ex. STD/HIV, Substance abuse, & correctional facilities) Trained Staff Vaccine Access & Delivery System Data collection/Dose tracking Risk Assessment Identify high-risk clients Counseling & Education Offer Vaccination Testing Tests Results & FU Hep C Reduce Morbidity & Mortality of Viral Hepatitis Vaccination Track Dosage & Refusals Referrals Provide Hepatitis Referrals Technical Assistance Functional Client Database

VHIP Program Description + Program Focus = Logic Model
Planning program evaluation and intervention Risk assessment survey completed at intake Identify who accessing services offered Able to identify high-risk groups Trained Staff Offer hepatitis counseling & education materials Increase awareness about transmission & positively change behaviors Promote prevention (offer vaccine, pamphlets & FAQs) Collaboration of Internal & External Programs Vaccine Delivery System Reduce Morbidity & Mortality of Viral Hepatitis Track dosage & refusals among high-risk groups Reduce transmission in high-risk populations Testing Vaccine Data Collection & Dose Tracking Establish a referral system Manage chronic cases with medical specialists Provide outreach medical & social services Technical Assistance

Helpful Publications @ www.cdc.gov/eval
37

Community Tool Box http://ctb.lsi.ukans.edu
38

Avoiding Evaluation “Detours” Using the CDC Evaluation Framework

Similar presentations

Presentation on theme: "Avoiding Evaluation “Detours” Using the CDC Evaluation Framework"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Avoiding Evaluation “Detours” Using the CDC Evaluation Framework

Similar presentations

Presentation on theme: "Avoiding Evaluation “Detours” Using the CDC Evaluation Framework"— Presentation transcript:

Similar presentations

About project

Feedback