College of Public Health and Human Sciences Public Health Policy Institute Making a Policy Evaluation Plan Presenter: Jangho Yoon, PhD Date: August 23, 2013

Evaluation

Types of evaluation Type Purpose Main questions Relevance A priori assessment of need for program Is the program appropriate for the defined problem? Adequacy Assessment of the extent to which the program may address a problem Is the program adequate in size and scope to make a difference? Progress Assess the degree to which program implementation complies with the plan. Are appropriate personnel, equipment, and financial resources available in the right quantity, in the right place, and at the right time to meet program need? Efficiency Determine whether program results could be obtained less expensively. Are program benefits sufficient for the cost incurred? Effectiveness Assess whether program results meet predetermined objectives. Did the program meet its stated objectives? Impact   Assess the long-term effects of the program. Did the program produce the observed effect? Sustainability Assess whether the effects of the program are likely to continue. What is the likelihood of the program to be self-sustaining?

Why? When? Improved decision making All phases of program development and implementation

Evaluation tool Program time line Evaluation type Evaluation method Policy gap analysis (need assessment) Relevance Adequacy Surveys Expert opinions Available data Implementation Progress Monitoring Assessing results Efficiency   Effectiveness Impact Sustainability Economic analysis Experiment Survey Expert opinion

Major realms and components

Measurement Program effects must be measured accurately. Numbers Rates The number of people served by the program, dollars budgeted and expended by the program, net value in cost analysis, etc. Rates A ratio of two measures Standardization: clients served per dollar spent. Useful for making comparisons across a number of similar programs of substantially different size. Prevalence and incidence Prevalence: the number of people at any time who actually show evidence of a disease. Incidence: the number of people who succumb to the condition or disease within a given time range. November 21, 2018

Measurement Perception Represents a broad category of measures that are generally defined along ordinal or interval scales. A common measure of attitudes, beliefs, or perceptions, uses a five-point scale: e.g., (1) strongly disagree; (2) disagree; (3) neutral; (4) agree; (5) strongly agree. Particular measures may be more or less appropriate in certain settings. Rates are more useful in discussing geographic areas or programs. Perception measures are often more useful when the individual is the unit of analysis. November 21, 2018

Pretest-posttest, single-group design Designing evaluation Pretest-posttest, single-group design Limitations Experimental design Pinnacle of evaluation research efforts. November 21, 2018

Pretest-posttest, single-group design Effect of program: (OP2 – OP1) . T1 T2   P OP1 X OP2 November 21, 2018

Design problems Suppose a health department implement a program to promote regular leisure-time physical activity. November 21, 2018

Additional events–Something else caused the outcome. Design problems Additional events–Something else caused the outcome. Time trends (or maturation)–It is due to general “upward” time trend in physical activity. Drop-outs–Participants dropped out of the program during the evaluation time period. Regression to the mean–Statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated. November 21, 2018

Regression to the mean November 21, 2018

Pretest-posttest design with treatment and control groups Experimental design Pretest-posttest design with treatment and control groups Effect of Program: (OP2– OP1) – (OC2– OC1) November 21, 2018

Multiple group pretest-posttest design Experimental design Multiple group pretest-posttest design November 21, 2018