1. Step 4: Gather Evidence Luann D’Ambrosio, MEd

2. Review: What to measure What amount of $ and time were invested What did the program actually consist of To what extent did knowledge and skills increase To what extent did practices change 1)Reduce 2)Acrue

3. Identifying Indicators How will you know it when you see it? What will be the evidence? What are the specific indicators that will be measured? Often expressed as #, % Can have qualitative indicators as well as quantitative indicators

4. Typical activity indicators to track Amount of products, services delivered #/type of customers/clients served Timeliness of service provision Accessibility and convenience of service Location; hours of operation; staff availability Accuracy, adequacy, relevance of assistance Courteousness Customer satisfaction For example: # of clients served # of consultations # of workshops held # of attendees # of referrals Quality of service

7. CDC Evaluation Framework

8. Measuring the Effects of Programs

9. Do You Need? Practice guidelines? Systematic reviews? Journal articles? Grey literature? Information for patients? A fact or data set?

10. Searching the Literature The Community Guide (CDC) http://www.thecommunityguide.org/ http://www.thecommunityguide.org/ Health Policy Guide (Center for Health Improvement) http://www.healthpolicyguide.org/ http://www.healthpolicyguide.org/ Cochrane review http://www.cochrane.org/reviews/ http://www.cochrane.org/reviews/

11. Pub Med http://pubmed.gov > 16 million citations to biomedical journal articles, 1949 to the present 5,000+ journals indexed Covers all aspects of biosciences and healthcare 75% of citations have abstracts Updated daily

12. Healthy People 2020 Expertly pre-formulated PubMed searches for each of the HP 2020 objectives http://phpartners.org/hp2020/index.html

13. Google Responsibly Evaluate what you find Keep track of where you’ve been Take advantage of advanced search features

14. Search Features Links, e.g., pages that link to the page File format Occurrences Date Domain

15. Program Evaluation

16. Data Collection Pre-Planning Project NameStateMethodDateLead OR Training InstituteORPre Post 3-month post Feb, 2005Luann Sandy Jennifer AK Training InstituteAKPre Post 3-month post April, 2005Luann Sandy Jennifer MT Training InstituteMTPre Post 3-month post May, 2005Luann Sandy Jennifer Disaster PreparednessIDPostFeb, 2005Luann Jennifer Leadership InstituteWAPre Post 3-month post 6-month post May, 2005Luann Sandy UW InstituteWAPre Post 3-month post July, 2005Luann Sandy iLincAllPostongoingConnie Margaret

18. Data Collection Flow Chart

19. Collecting Information Getting started… Clarify roles Develop timeline Identify sources of data Decide on collection methods Develop data management and analysis plan

20. Definitions Quantitative: what is or may be estimated by quantity; of or pertaining to the description or measuring of quantity. Qualitative: pertaining to, or concerned with quality; descriptive information that may capture experience, behavior, opinion, value, feeling, knowledge, sensory response or observable phenomena.

21. Practical Definitions Quantitative data: numbers, tallies (bean count) Qualitative data: Descriptions, people Opinions, reactions, behaviors, experiences Feelings (touchy feely)

22. Evaluation Qualitative Narrative based Analysis of text, rather than numbers Hypothesis generating Rather than hypothesis testing Explanatory Theory building, not theory testing Asking questions Answering questions Analyzing, reflecting on, and judging evidence Making changes

23. Qualitative Methods Key informant interviews Focus groups Participant observation

24. Use of Qualitative Methods Pilot research on interventions Programs involving behavioral issues Program questions starting with “why” Evaluations of public health programs Development of theory

25. Benefits Data will likely have much greater depth and coherence Improved program confidence Will enhance subsequent quantitative studies by not making assumptions of subject’s viewpoints Surprises

26. Participant Observation Commonly used by anthropologists Require long-term commitment to data collection Data consists of: Behavioral observations Narrative from subjects Notes or tape

27. Focus Groups Format Moderator and observer Discussion group of 5–12 people Open-ended questions are usually narrowly framed Focus is on ‘concept’ or ‘product’ Allow for subject-subject interchange

28. Considerations Homogeneity Assemble by likely demographic predictors of viewpoint (e.g., gender, cultural views, age, geography) Constitution of Focus Groups

29. Key Informant Interviews Great for use in rapid assessment for gathering information from an affected community.

30. Key Informant Questions

31. Quantitative Methods Counting systems Surveys (e.g., questionnaires)

32. Survey A data collection tool used to gather information about individuals.

33. Questionnaire A questionnaire is a series of questions asked to individuals to obtain statistically useful information about a given topic.

34. Methods Personal interviews Phone interviews Distribution: mail, internet

35. Purposes Identify user likes / dislikes Gather baseline data on KABS’s of target population Gather baseline data on rates Measure satisfaction Identify problems

36. Other Types Nominal group process 1)Small group responds to question and ranks the answer in order of importance Tests or assessments 1)Help determine knowledge and needs of target groups

37. Analyzing Your Data Keep it simple Word Excel

38. Conclusion Qualitative and quantitative methods helps to draw conclusions about effectiveness of program materials, plans, activities and target population.