Outcomes Evaluation A good evaluation is …. –Useful to its audience –practical to implement –conducted ethically –technically accurate.

Evaluation Outcomes/impact evaluation –measurement of changes in attitude, knowledge, health status, behavior, nutrition status

6 Steps of Evaluation Establish an evaluation plan from beginning of program Obtain buy-in from administrators Allow enough staff time to make evaluation a priority

6 Steps of Evaluation Obtain permission & encourage participation from participants Be flexible & creative

6 Steps of Evaluation Use a strong research design & measures which generate data you need to support your program’s goal

Validity Internal validity –extent to which an observed effect can be attributed to a planned intervention

Validity External validity –extent to which an observed impact can be generalized to other settings & populations with similar characteristics

Threats to Validity History Measurement Selection

History External event (H) Internal programmatic or internal participant events (I) Treatment effects (X)

Measurement Methods used to collect data Instruments need to be –reliable –valid

Selection Define eligibility to participate in a program –criteria –is there a difference between those who stay in the program & those that drop

Selection Of those eligible, who accept & who refuse –3222/5000 = 64% –attended/eligible

Selection Drop out rate –1600/3222 = 50% –drop out/initially say yes

Selection Lost to follow-up –300/1622 = 18% –can’t find/completed program

Selection Identify contextual or structural variables to decrease selection bias

Regression Effects If score high on pretest, little room for improvement Will show program has poor impact Is pretest score a threat to validity?

Synergistic Effects All work together to lower internal validity

Evaluation Designs Design to increase internal validity

Evaluation Designs Design is selected based on –objectives of the program –purpose of the evaluation –availability of eval resources –type of health & behavior problem, setting & audience

Evaluation Designs Notation –R - random assignment –E - intervention group –C - true control group –C - comparison group –X - treatment

Evaluation Designs Notation –N - number of subjects –O - observation to collect data –T - time

One Group One group & one time –posttest only E OXO Non-experimental No random assignment

One Group No control/comparison group

One Group What are some of the main weaknesses of this design for increasing internal validity?

One Group When could this design be used & be appropriate?

Nonequivalent Comparison E OXO C OXO Comparison group –any group not formed by random assignment

Nonequivalent Comparison What threats to internal validity are lessened?

Baseline Data

Time Series E OOO X OOO Pattern of outcome variable Know stability of outcome measure Collect outcome variable unobtrusively

Time Series Multiple data points –increases the power of the design Equal intervals

Time Series Must still try to control of history, selection & measurement

Time Series With control or comparison group, much stronger Better control over history threat to validity

Time Series with Comparison Group E OOO X OOO C OOO X OOO

True Experimental R E OXO R C O O Establish at baseline 2 groups not sig different Best control of threats to validity

True Experimental Advantages? Disadvantages?

Post-then-pre Appropriate for assessing behavior change Participants have limited knowledge at beginning of program

Post-then-pre Example of a typical pre-test question –Do you include one food rich in vitamin C in your diet daily?

Post-then-pre Implement See handout, Table 2 After program give only a posttest

Post-then-pre Q1 - Asks about behavior because of program Q2 - What the behavior had been before the program (i.e.. The pretest question)

Post-then-pre U of NE handout data analysis problem 6 U of NE ETHT report

Success Stories Testimonials Qualitative info Audience testing

Case Study Story of an individual Can be biased Cannot be generalized

More Services Available Change in the environment

Evaluation Plans Read Moving to the Future example PERT & Gantt charts

Evaluation Plans Must be in place –objectives –specifications of the intervention & program methods

Evaluation Plans Must be in place –measurement & data collection procedures –description of methods

Evaluation Plans Main reason –when –from whom –how –by whom

Evaluation Plans Worksheet 1 & 2 Worthen & Sanders, 1987 Fill out form as a team

END Evaluation Questions?