HW 425 Unit 9 Seminar

Agenda  Considerations of Evaluation Design  Experimental vs. Non-Experimental Designs  Benefits and Drawbacks of Design Types  Discussion of Seminar Question

Approaches to Evaluation Design  Objective-oriented Focus on outcomes; measure stated objectives or use goal-free approach (Scriven, 1972)  Management-oriented Provide info for management decisions  Consumer-oriented Comparison model  Expertise-oriented Uses experts, accreditation processes  Adversary-oriented Solicit opposing viewpoints  Naturalistic-oriented Also participant-oriented

Question: What are the benefits to including multiple approaches in your evaluation approach?

Issues in Evaluation Design  Evaluations use research techniques, but research and evaluation are different  Research should be value-free or objective; evaluations are meant to determine value  “The evaluation design be driven by developing the best ways in which to answer evaluation questions, rather than by any particular design or approach.” (Hodges & Videto, 2006, p. 159)

Considerations for Evaluation Questions  Need to collect data to answer each question  Use multiple sources  Include quantitative and qualitative data types  Consider internal and external validity Internal validity: ability to say program caused effect External validity: applicability of results

Evaluation Designs  Experimental Design Participants randomly assigned to one or more groups Randomization allows evaluator to assume traits could influence outcome equally distributed between groups ProsCons Strongest evidenceParticipants agree to randomization Minimize threats to internal validity Large number of participants required Delayed participation for control group

Evaluation Designs continued  Quasi-experimental design Compare two or more groups Do not require random assignment ProsCons Reduces threats to validity No evidence for causation – evidence not as strong as experimental May be more realistic for health promotion Requires careful selection of group participants Does not require random assignment

Examples of Experimental & Quasi- Experimental Designs  Post-test Only Control Group  Pre-test Post-test Control  Time Series  Pre-test Post-test Follow-up  Cross over designs

Non-experimental Designs  Do not involve comparison groups  Are not able to address questions about program effects  Often included within program evaluation, early in program  Provide structure for gathering other information about staff and participant experiences, barriers to program implementation  Examples: Case Study, Pre-test Post-test no control

Seminar Discussion Question Why are non-experimental designs often appropriate for process evaluation activities?

Wrap-Up  Considerations of Evaluation Design  Experimental vs. Non-Experimental Designs  Benefits and Drawbacks of Design Types  Discussion of Seminar Question  Questions?? Thanks for a fun term, Everyone! Good luck on final projects. Remember, we’ll have open discussions in Unit 10 – hope to “see” you again