2. Too expensive

3. Too complicated

4. Too time consuming

5. Not a priority

6. Just don’t know where to start

7.  Lack of research/statistics skills  Lack of time  Lack of resources  Other priorities  Lack of incentive  Fear  Don’t see value

8. The process of determining the merit, worth, or value of a program (Scriven, 1991)

9. Systematic inquiry that describes and explains, policies’ and programs’ operations, effects, justifications, and social implications (Mark, Henry, & Julnes, 2000)

10. The systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs (Rossi & Freeman, 1989)

11. In simpler terms….. Collection of information to determine the value of a program e VALU ation

12.  Auditing  Personnel assessment  Monitoring (although this can be part of an evaluation process)  Used to end or shut down programs

13. Evaluation is an extraneous activity that generates lots of boring data with useless conclusions

14. Evaluation is about proving the success or failure of a program

15. Evaluation is a unique and complex process that occurs at a certain time in a certain way, and almost always includes the use of outside experts.

16.  Demonstrate program effectiveness or impacts

17.  Better manage limited resources

18.  Demonstrate program effectiveness or impacts  Better manage limited resources  Document program accomplishments

19.  Demonstrate program effectiveness or impacts  Better manage limited resources  Document program accomplishments  Justify current program funding

20.  Demonstrate program effectiveness or impacts  Better manage limited resources  Document program accomplishments  Justify current program funding  Support need for increased funding

21.  Demonstrate program effectiveness or impacts  Better manage limited resources  Document program accomplishments  Justify current program funding  Support need for increased funding  Satisfy ethical responsibility to clients to demonstrate positive and negative effects of participation

22.  Demonstrate program effectiveness or impacts  Better manage limited resources  Document program accomplishments  Justify current program funding  Support need for increased funding  Satisfy ethical responsibility to clients to demonstrate positive and negative effects of participation  Document program development and activities to help ensure successful replication

23. To improve program performance which leads to better value for your resources

24.  No evidence that your program is working or how it works

25.  Lack of justification for new or increased funding

26.  No evidence that your program is working or how it works  Lack of justification for new or increased funding  No marketing power for potential clients

27.  No evidence that your program is working or how it works  Lack of justification for new or increased funding  No marketing power for potential clients  Lack of credibility

28.  No evidence that your program is working or how it works  Lack of justification for new or increased funding  No marketing power for potential clients  Lack of credibility  Lack of political and/or social support

29.  No evidence that your program is working or how it works  Lack of justification for new or increased funding  No marketing power for potential clients  Lack of credibility  Lack of political and/or social support  No way to know how to improve

30. DevelopmentImplementationEvaluation RevisionSustainability

31.  Types of Evaluation › Outcome (summative) › Process (formative)

32.  Types of Evaluation › Outcome (summative) › Process (formative)  Outcomes

33.  Types of Evaluation › Outcome (summative) › Process (formative)  Outcomes  Indicators

34.  Types of Evaluation › Outcome (summative) › Process (formative)  Outcomes  Indicators  Measures

35.  Types of Evaluation › Outcome (summative) › Process (formative)  Outcomes  Indicators  Measures  Benchmarks

36.  Types of Evaluation › Outcome (summative) › Process (formative)  Outcomes  Indicators  Measures  Benchmarks  Quantitative vs. qualitative

37. Engage stakeholders Clearly define program Written evaluation plan Collect credible/useful data Analyze data Share/use results

38.  Those involved in program design, delivery, and/or funding  Those served by the program  Users of the evaluation results

39.  Resources, activities, outcomes  Context in which program operates  Logic model › Explicit connections between “how” and “what” › Helps with program improvement › Good for sharing program idea with others › Living, breathing model

40. IF THEN

41. IF THEN I take an aspirin

42. IF THEN I take an aspirin My headache will go away

43. IF = Inputs & Activities THEN = Outcomes

45.  Outcomes  Indicators  Tools  Timelines  Person(s) responsible (optional)

46. PROGRAM OUTCOME INDICATOR(S) DATA COLLECTION TOOL DATA COLLECTION SCHEDULE Training participants know how to recognize a seizure Percent of training participants who correctly identify 10 out of 13 possible symptoms of a seizure Participant pre, post and follow- up surveys Pre survey given prior to training; post survey given immediately after training; follow up survey given 30 days after training

47.  Valid and reliable tools › Valid=measures what it is intended to measure › Reliable=consistent results over time  Qualitative  Quantitative  Will answer your evaluation questions and inform decision-making

48.  Quantitative › Surveys › Tests › Skill assessments  Qualitative › Focus groups › Interviews › Journals › Observations

49.  Many methods  Answer evaluation questions  Engage stakeholders in interpretations  Justify conclusions and recommendations  Get help if needed!

50.  Reporting format  Getting results into the right hands  Framing the results  Collaborative vs. confrontational approach  Keeping users “in the loop”  Debriefs and follow-up

51.  Purpose

52.  Audience

53.  Purpose  Audience  Resources

54.  Purpose  Audience  Resources  Data

55.  Purpose  Audience  Resources  Data  Timeline

56.  Purpose  Audience  Resources  Data  Timeline  Planning is key

57.  Purpose  Audience  Resources  Data  Timeline  Planning is key  Expertise

58.  Staff to perform work

59. › Available

60.  Staff to perform work › Expertise › Available

61.  Staff to perform work › Expertise › Available  Credibility

62.  Staff to perform work › Expertise › Available  Credibility  Technological support

63.  Staff to perform work › Expertise › Available  Credibility  Technological support › Collect data

64.  Staff to perform work › Expertise › Available  Credibility  Technological support › Collect data › Analyze data

65.  Staff to perform work › Expertise › Available  Credibility  Technological support › Collect data › Analyze data  Time frame

66.  Training program for caretakers of seniors with epilepsy/seizures  ADC staff and primary care providers  Training provided by affiliates  Delivery varies but content is consistent

67.  Meeting with EF staff to learn about the program  Collaboration with affiliate staff to design logic model  Decisions regarding which outcomes to measure  Decisions regarding how to best collect data  Designed data collection tools  Pilot testing and revision

69.  What impact did the training program have on knowledge of seizures in seniors? › Pre and post knowledge assessment › Post-training survey  What impact did the training program have on participants’ confidence and comfort in working with seniors ? › Post-training survey

71. Our benchmark is a rating of 7.0 or higher

72. Kathleen Dowell, Ph.D., President EvalSolutions 6408 Whistling Wind Way Mt. Airy, MD 21771 410-707-0763 kathy@eval-solutions.com www.eval-solutions.com