1. College of Public Health and Human Sciences
Public Health Policy Institute
Making a Policy Evaluation Plan
Presenter: Jangho Yoon, PhD
Date: August 23, 2013

2. Evaluation

3. Types of evaluation Type Purpose Main questions Relevance
A priori assessment of need for program
Is the program appropriate for the defined problem?
Adequacy
Assessment of the extent to which the program may address a problem
Is the program adequate in size and scope to make a difference?
Progress
Assess the degree to which program implementation complies with the plan.
Are appropriate personnel, equipment, and financial resources available in the right quantity, in the right place, and at the right time to meet program need?
Efficiency
Determine whether program results could be obtained less expensively.
Are program benefits sufficient for the cost incurred?
Effectiveness
Assess whether program results meet predetermined objectives.
Did the program meet its stated objectives?
Impact
Assess the long-term effects of the program.
Did the program produce the observed effect?
Sustainability
Assess whether the effects of the program are likely to continue.
What is the likelihood of the program to be self-sustaining?

4. Why? When? Improved decision making
All phases of program development and implementation

5. Evaluation tool Program time line Evaluation type Evaluation method
Policy gap analysis (need assessment)
Relevance
Adequacy
Surveys
Expert opinions
Available data
Implementation
Progress
Monitoring
Assessing results
Efficiency
Effectiveness
Impact
Sustainability
Economic analysis
Experiment
Survey
Expert opinion

6. Major realms and components

7. Measurement Program effects must be measured accurately. Numbers Rates
The number of people served by the program, dollars budgeted and expended by the program, net value in cost analysis, etc.
Rates
A ratio of two measures
Standardization: clients served per dollar spent.
Useful for making comparisons across a number of similar programs of substantially different size.
Prevalence and incidence
Prevalence: the number of people at any time who actually show evidence of a disease.
Incidence: the number of people who succumb to the condition or disease within a given time range.
November 21, 2018

8. Measurement
Perception
Represents a broad category of measures that are generally defined along ordinal or interval scales.
A common measure of attitudes, beliefs, or perceptions, uses a five-point scale: e.g., (1) strongly disagree; (2) disagree; (3) neutral; (4) agree; (5) strongly agree.
Particular measures may be more or less appropriate in certain settings.
Rates are more useful in discussing geographic areas or programs.
Perception measures are often more useful when the individual is the unit of analysis.
November 21, 2018

9. Pretest-posttest, single-group design
Designing evaluation
Pretest-posttest, single-group design
Limitations
Experimental design
Pinnacle of evaluation research efforts.
November 21, 2018

10. Pretest-posttest, single-group design
Effect of program: (OP2 – OP1) .
T1 T2
P OP1 X OP2
November 21, 2018

11. Design problems
Suppose a health department implement a program to promote regular leisure-time physical activity.
November 21, 2018

12. Additional events–Something else caused the outcome.
Design problems
Additional events–Something else caused the outcome.
Time trends (or maturation)–It is due to general “upward” time trend in physical activity.
Drop-outs–Participants dropped out of the program during the evaluation time period.
Regression to the mean–Statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated.
November 21, 2018

13. Regression to the mean
November 21, 2018

14. Pretest-posttest design with treatment and control groups
Experimental design
Pretest-posttest design with treatment and control groups
Effect of Program: (OP2– OP1) – (OC2– OC1)
November 21, 2018

15. Multiple group pretest-posttest design
Experimental design
Multiple group pretest-posttest design
November 21, 2018