How to Plan a Local Evaluation and Lessons Learned Philip Rodgers, Ph.D. Evaluation Scientist American Foundation for Suicide Prevention 2013 Garrett Lee Smith Combined Annual Grantee Meeting June 11-13, 2013, Washington DC

Acknowledgements U.S. Department of Health & Human Services Substance Abuse and Mental Health Services Agency Howard Sudak, MD AFSP Katrina Bledsoe, PhD SPRC

Defining Evaluation

5 Why is evaluation important? Required by funding agencies. Improves performance. Demonstrates effectiveness. Advances knowledge. As not everything can be done, there must be a basis for deciding which things are worth doing. Enter evaluation. M. Q. Patton Source for Patton Quote: U.S. Department of Health and Human Services, P. H. S. (2001). National Strategy for Suicide Prevention: Goals and Objectives for Action. Rockville, MD: U.S. Department of Health and Human Services, Public Health Service.

6 What are types of evaluation? Participatory evaluation Formative evaluation Summative evaluation Responsive evaluation Goal-free evaluation Empowerment evaluation Advisory evaluation Accreditation evaluation Adversary evaluation Utilization evaluation Consumer evaluation Theory-driven evaluation We will address: Process, Outcome, and Impact Evaluations

7 What is process evaluation? Process “evaluation assesses the extent to which a program is operating as it was intended.” Source: General Accounting Office. (2005). Performance Measurement and Evaluation: Definitions and Relationships (Vol. GAO SP). Washington DC: United States Government Accounting Office.

8 What is outcome evaluation? Outcome “evaluation assesses the extent to which a program achieves its outcome-oriented objectives.” Source: General Accounting Office. (2005). Performance Measurement and Evaluation: Definitions and Relationships (Vol. GAO SP). Washington DC: United States Government Accounting Office.

9 What is impact evaluation? “Impact evaluation…assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program.” Source: General Accounting Office. (2005). Performance Measurement and Evaluation: Definitions and Relationships (Vol. GAO SP). Washington DC: United States Government Accounting Office.

I. Logic Models & Evaluation

11 InputsActivitiesOutputs Funds Trainers Materials Trainees Gatekeeper Training  People trained  Satisfaction with training  Fidelity of training Logic models can drive evaluations Generic Gatekeeper Training Logic Model Outcome Evaluation Process Evaluation Short-term a.  Knowledge b.  Attitudes Intermediate c.  Identification of those at risk d.  Referral Long-term f.  Treatment g.  Suicide deaths a.  Knowledge b.  Attitudes c.  Identification of those at risk d.  Referral f.  Treatment g.  Suicide deaths Impact Evaluation vs. Control Group

II. How to Develop Evaluation Questions

13 Where do evaluation questions come from? Generally, from the goals listed in a logic model. More specifically, defined by stakeholders through a collaborative process. Depending upon circumstances, stakeholders can be funders, participants, trainers, evaluators, and others, or a combination of these.

14 Divergent, convergent process… Stakeholders meet with relevant materials (grant application, logic model, etc.). After a review of materials, engage in divergent process—a free association of evaluation questions After the divergent process, stakeholders collectively narrow list of questions to manageable proportions through a convergent process.

III. How to Answer Evaluation Questions

16 What is measurement? Measurement is the means you use to collect data. It includes how you collect data and what data you collect (and how well you collect data).

17 How will you collect data? Questionnaires (in-person, mail, , phone) Psychological Tests Interviews Health Records Health Statistics ObservationsLogs

18 Where do you find measures? Create your own (pilot test!) Borrow from other evaluations (with permission!) Search the literature (see Additional Resources) Use standardized measures (may cost) –Brown (adult) and Goldston (adolescent) reviews Use existing data sources and records

IV. How to Develop an Evaluation Plan

20 What evaluation design do you need? What is your purpose? –Performance assessment? –Evidence of concept? –Evidence of effectiveness?

21 There are four basic evaluation designs Posttest OnlyXO Pre- and PosttestOXO Posttest Only w/controlXO O Pre- and Posttest w/controlOXO OO

22 Additional data collection points can be added to the basic desgins Posttest OnlyXOO Pre- and PosttestOXOO Posttest Only w/controlXOO OO Pre- and Posttest w/controlOXOO OOO

23 Best evidence comes when subjects are randomly assigned to groups x Exp. Group Con. Group Exp. Group Con. Group Pool of Subjects or Groups Random assignment increases the likelihood that subjects in both groups are equivalent in regards to factors that may be related to outcomes.

24 If random assignment is not possible… Compare groups that are similar. Use a pretest so that group differences—to some extent—are accounted for.

Evaluation Planning Forms 25

Philip Rodgers, PhD American Foundation for Suicide Prevention