2. http://sis.nlm.nih.gov/outreach/training_outreach.html

4. Purpose

5. Objectives

6. Today’s Topics

8. Program Evaluation is a combination of: – Good Reporting – Good Detective Work – Good Historical Documentation and Storytelling Program Evaluation helps organizations see how and in what ways they are really helping the community, OR how and if an organization is achieving its goals and objectives.

10. Describing and documenting the course of a program For Example: – Who are the Members or Participants? – What activities were conducted? – How many people were in attendance? – When and Where are events and meetings being held? – What staff was there? – Who was involved in the planning? – Who was involved in the implementation?

15. Determine if goals were met Show effectiveness and impact of a program Provide evidence of the process and outcomes

23. Awareness Marketing/awareness of information Community partnerships/community engagement Access Retrieval skills Access to computers/mobile/Internet Proof of access/number of visits/types of resources

24. Awareness In what ways is information advertised to potential consumers/clients? How are community partnerships engaged to help provide information access to residents? Access How often are clients retrieving information? How are clients accessing information via computers, Internet and mobile? How many times & which types of resources clients are accessing?

25. Awareness Marketing plan objectives, outcomes Number of MOAs/MOUs Number & quality of community partnership/community advisory board meetings Observations of community partner engagement Access Social Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google Analytics

26. Social Media/ Health communications planning with evaluation component https://www.healthcommworks.org/Account/LogOn?signInArea=SocialM ediaWorks US Department of Education Evaluating Online Learning www.ed.gov/admins/lead/academic/evalonline/evalonline.pdf Evaluation strategy for mobile device learning http://www.salt.org/salt.asp?pn=procabs&ss=l&key=153320&pid=2406

30. Training Development of trainings/types of: access/utilization/ general computer skills/database training Training target populations: staff/clients/ patrons/agencies/special populations Number of trainings/trained users Training satisfaction Training outcomes: knowledge/skills

31. Utilization Types/amount of resources/information utilized: Internet/social media/email/mobile/brochures/factsheets/face-to-face Utilization outcomes: Knowledge/skills/empowerment/efficacy/service improvement/community outcomes

32. Training What types of trainings are being developed? Who is the target training population? How many people are being trained? (Attendance/completion) Are consumers satisfied with trainings? What are the training outcomes/skills developed? Utilization What type of information consumers are utilizing? How is that information being utilized? What are the outcomes associated with consumer utilization of information resources?

33. Training Training Development Template Sign-In sheets Pre/Post assessments Satisfaction surveys Utilization Social Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google Analytics Questionnaires, Surveys, Qualitative analysis of responses

34. Training development template Sample pre/post skills assessments Writing measurable learning outcomes http://www.gavilan.edu/research/spd/Writing-Measurable-Learning- Outcomes.pdf Evaluating Training Programs-Kirkpatrick’s 4 levels http://www.wa.gov/esd/training/toolbox/tg_kirkpatrick.htm

39. Resource needs Target population needs assessment/types of resources needed Resource development Resource materials created: type such as cultural specific, bibliography Resource materials utilized/sustained/maintained/updated Purchases of hardware/software to develop resources Utilization of consultants/contractual relationships to develop resources Access Purchases of hardware/software to access and share information Utilization of consultants/contractual relationships to access and share information

40. Resource needs Has a resource needs assessment been conducted for the target population? If so, what were the outcomes of that needs assessment? Resource development What types of resource materials are being developed? How are the resource materials being utilized/sustained/maintained/ updated? Was hardware/soft purchased to develop resource material? If so, what kind and how was it used? Were consultants/contractual relationships used to develop resource material? If so, who and what work did they complete?

41. Access Was hardware/soft purchased to help agencies provide access/share materials? If so, what kind and how was it used? Were consultants/contractual relationships used to help agencies provide access/share materials? If so, who and what work did they complete?

42. Resource needs Outcomes of Community/resource needs assessment Resource development Resource development template Resource material tracking (How utilized/sustained/maintained/updated) Comparison checklist for equipment/software purchases Consultant/contractual reports Access Equipment/software inventory Utilization log Consultant/contractual reports

43. TeamLab-University of Southern California Designing materials for particular audiences/understanding culture http://teamlab.usc.edu/learn/roadmap/audience.html http://teamlab.usc.edu/learn/roadmap/audience.html CenterLink-The Community of LBT Centers Conducting Needs assessment http://www.lgbtcenters.org/how-to-do-a-community-needs-assessment.aspx

44. Harvard Family Research Project Evaluation Exchange, Understanding Cost-Benefit and Cost-Effectiveness Evaluation http://www.hfrp.org/evaluation/the-evaluation-exchange/issue- archive/methodology-15/at-what-price-benefit-cost-analysis-and-cost- effectiveness-analysis-in-program-evaluationhttp://www.hfrp.org/evaluation/the-evaluation-exchange/issue- archive/methodology-15/at-what-price-benefit-cost-analysis-and-cost- effectiveness-analysis-in-program-evaluation TechSoup.org Website for Non-Profits and technology. Planning and how to articles http://www.techsoup.org/support/articles-and-how-tos/articles-by- subject?subjectid=14

45. Quarterly reports Final reports ACIOP Resources Blog http://nlmacioawardeeblog.wordpress.com http://nlmacioawardeeblog.wordpress.com Individual agency evaluation conference calls Evaluation technical assistance

46. Visit ACIOP Blog http://nlmacioawardeeblog.wordpress.comhttp://nlmacioawardeeblog.wordpress.com Individual agency calls to discuss: – Agency Evaluation capacity – Project evaluation plans – Technical assistance Mid Year Webinar – Milestones – Evaluation needs of agencies – Final reports

48. Arletha D. Livingston, Ph.D., MPH, CHES Email: transformingcomm@gmail.comtransformingcomm@gmail.com Phone: 404.368.9252