1.  2007 Johns Hopkins Bloomberg School of Public Health Introduction to Program Evaluation Frances Stillman, EdD Institute for Global Tobacco Control Johns Hopkins Bloomberg School of Public Health

2.  2007 Johns Hopkins Bloomberg School of Public Health 2 Objectives Components of program evaluation Types of evaluations A model for planning and conducting program evaluations

3.  2007 Johns Hopkins Bloomberg School of Public Health Section A Introduction to Program Evaluation

4.  2007 Johns Hopkins Bloomberg School of Public Health 4 Definition of Evaluation “Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object”* Important components of evaluation  A systematic process  Collecting and sifting through data  Making judgments about the validity of the information  Using information to make decisions *Source: Trochim, W. (1999).

5.  2007 Johns Hopkins Bloomberg School of Public Health 5 Research vs. Evaluation Evaluation is a methodological area that is closely related to, but distinguishable from, more traditional social research Evaluation utilizes many of the same methodologies used in traditional social research But because evaluation takes place within a political and organizational context, it requires:  Sensitivity to multiple stakeholders  Political dexterity  Management ability  Group skills  Other skills that social research, in general, does not rely on as much

6.  2007 Johns Hopkins Bloomberg School of Public Health 6 Research vs. Evaluation “Research seeks to prove, evaluation seeks to improve...” — M.Q. Patton

7.  2007 Johns Hopkins Bloomberg School of Public Health 7 Surveillance and Monitoring vs. Program Evaluation Surveillance: tracks disease or risk behaviors Monitoring: tracks changes in program outcomes over time Evaluation: seeks to understand specifically why these changes occur

8.  2007 Johns Hopkins Bloomberg School of Public Health 8 Why Evaluate Your Program? Monitor progress toward goal Demonstrate effectiveness of program or activity Determine if program components are producing the desired effect Justify need for funding and support Learn how to improve program Remember, the goal of evaluation is to improve a program  Evaluation is not useful unless the findings are used to make a difference!

9.  2007 Johns Hopkins Bloomberg School of Public Health 9 The Four Standards 1.Utility: who needs the information; what information do they need? 2.Feasibility: how much money, time, and effort can we put into this? 3.Propriety: what steps need to be taken for the evaluation to be ethical? 4.Accuracy: what design will lead to accurate information?

10.  2007 Johns Hopkins Bloomberg School of Public Health 10 When to Conduct Evaluation? The stage of program development influences the reason for program evaluation

11.  2007 Johns Hopkins Bloomberg School of Public Health 11 Types of Evaluation Formative  Strengthens or improves the program being evaluated Summative  Determines if program expectations were met and assesses program’s performance

12.  2007 Johns Hopkins Bloomberg School of Public Health 12 Examples of Formative Research Needs assessment Evaluability assessment Structured conceptualization Implementation evaluation Process evaluation

13.  2007 Johns Hopkins Bloomberg School of Public Health 13 Types of Questions: Formative Evaluation What is the definition and scope of the problem or issue?  Or, What is the question? Where is the problem; how big or serious is it? How should the program be delivered to address the problem? How well is the program delivered?

14.  2007 Johns Hopkins Bloomberg School of Public Health 14 Examples of Summative Research Outcome evaluations Impact evaluation Cost-effectiveness and cost-benefit analysis Secondary data analysis Meta-analysis

15.  2007 Johns Hopkins Bloomberg School of Public Health 15 Types of Questions: Summative Evaluation What type of evaluation is feasible? What was the effectiveness of the program? What is the net impact of the program?

16.  2007 Johns Hopkins Bloomberg School of Public Health 16 Methods to Collect Information Questionnaires, surveys, and checklists Interviews Documents Observation Case studies