1 Making Sense of Evaluation Data Mary Michaud, MPP University of Wisconsin— Cooperative Extension Fall 2002

2 Why are these skills important? Fundamental skills for public health service Make sense of your own data Interpret other data Make your case

3 Let’s get started Introductions and agenda Making sense of evaluation data: –Top ten takeaway lessons from the workshop What about this report do you find informative?this report What would you change?

4 Myths One report is enough. People read written reports. Complex analysis and big words impress people. Oral reports have the same effect as written reports. Describing limitations weakens report. Everything should be reported. The audience knows why they are getting the report.

5 Building an evaluation plan 1.Identify the purpose of evaluation 2.Clarify who will use the results 3.Clearly describe what is being evaluated (use a logic model) 4.Specify questions to ask 5.Identify sources of information 6.Select methods to collect information 7.Analyze and interpret information 8.Report and use results

6 Making sense of the data Start with a plan before you collect data –Purpose –Who will use the information –Resources –Sources of information and data collection methods Collect data Clean data Code data Tabulate your data Describe and interpret data

7

8

9

10 Why collect quantitative data? To make comparisons… –Between groups Smokers vs. Non-smokers Opinions of people who heard radio ad vs. people who didn’t Men vs. Women –Over time Change in public support for smoke-free ordinance

11 Why collect qualitative data? Explore meaning, motivation, emotion Understand experiences Understand language people use to describe their experiences Examples: interviews, focus groups, journals, document review

12

13 How many? How often? Frequency, or count Useful when real numbers adequately tell the story Ten worksites in Williams County have more than 300 employees. Between 2001 and 2003, six of those worksites implemented policies to ban smoking. As a result, 2,400 workers in Williams County now work in smoke-free environments. This report documents the role the Williams County Tobacco Free Coalition played in promoting this change.

14 *Surveys sampled worksites with more than five employees. **Sources: University of Wisconsin Monitoring and Evaluation Program. Results of 2001 Wisconsin Worksite Smoking Policy Survey. March Williams County Tobacco Free Coalition. Results of 2001 Worksite Smoking Policy Survey. October What proportion?

15 What’s the norm? (or central tendency)

16 How much do the data vary? Range Standard deviation (SD) –The larger the SD, the greater the variability in data –With a smaller sample size, outliers receive more “weight” –In a normal distribution, 65% of data lie within one SD and 95% lie within two SD Key to interpreting other studies

17 Mean 1 SD 2 SD Number of 6 th graders Height y x n = 32 n = 320

18 Sampling What is a random sample? Why sampling works Claims you cannot make Save time, save money

19 Is my survey “valid?” Validity of results depends on: Sampling –Quality of sampling frame –Sampling method Questionnaire design Questionnaire administration –Telephone –Mail …And other things!

20 Validity and reliability Validity: Are you measuring what you think you are measuring? –There are multiple types of validity Reliability: If something was measured again using the same instrument, would it produce the same (or damn near the same) results? –There are multiple places reliability can break down Why are these important?

21 Interviewer 1. I would like to ask you a few questions about smoking. Interviewer 2. I would like to ask you a few questions about smoking. [I’m a smoker, so you don’t have to worry about telling me if you smoke. It will really help us if you are honest about this.]

22 Do you favor or oppose a city ordinance that would make all Williamsburg restaurants smoke-free? Yes71.3% No28.7%

23 Supports restaurant ordinance Opposes restaurant ordinance Undecided/ declined to comment Current smokers (n=55) 8 (15% of smokers) 33 (60% of smokers) 14 (25% of smokers) Non-smokers (n=200) 170 (86% of non- smokers) 16 (8% of non- smokers) 12 (6% of non- smokers) Total (N=255) 178 (70% of all respondents) 49 (19% of all respondents) 26 (11% of all respondents)

24 “Farming it out” Pros Expertise Time Scope Cons Expertise? Expense Supervision required Find out: Exactly what services they provide Types of past accounts Willingness to share the work with you Willingness to do “pro bono” or reduced-fee work If they will provide you technical assistance

25 “Farming it out” Remember: Do not show your gold! Request a proposal Get your raw data after it is collected

26 Analysis tips Analyzing “by hand” Excel Other programs: –Epi info (CDC data management and analysis program: info –SPSS (statistical software)SPSS –Microsoft Access (database)Access

27

28 Analyzing qualitative data “Content analysis” steps: 1.Transcribe data (if audio taped) 2.Read transcripts 3.Highlight quotes and note why important 4.Code quotes according to margin notes 5.Sort quotes into coded groups (themes) 6.Interpret patterns in quotes 7.Describe these patterns

29 Qualitative data analysis Words Context Internal consistency Frequency of comments Extensiveness of comments Intensity of comments Specificity of responses What was not said

30

31 Example data set

32 Discussing limitations Written reports: Be explicit about your limitations Oral reports: Be prepared to discuss limitations Be honest about limitations Know the claims you cannot make –Do not claim causation without a true experimental design –Do not generalize to the population without random sample and quality administration (e.g., <60% response rate on a survey)

33 Reporting results Format depends on purpose and audience Written, oral Summative, formative What is the audience used to hearing or seeing? Common graphics –PhotographsPhotographs

34 Using graphics Title Clear units of measure Date(s) data collected Simple, straightforward design without “clutter” Font size 10 point or larger Explicit data source(s) Sample size, if applicable for the audience

35 *Surveys sampled worksites with more than five employees. **Sources: University of Wisconsin Monitoring and Evaluation Program. Results of 2001 Wisconsin Worksite Smoking Policy Survey. March Williams County Tobacco Free Coalition. Results of 2001 Worksite Smoking Policy Survey. October 2001.

36 Reporting results to the media All media: Avoid using too many statistics. Focus on the key points. For quotes, speak more globally about the issue. Always give the source and timeliness of your stats. It’s the “news peg.” Steve Busalacchi Director, News & Information Wisconsin Medical Society

37 Reporting results to the media Radio and TV: Do not offer exact statistics—ear cannot track. “73.6% of respondents” vs. “Nearly three quarters of those surveyed” Don’t go into great detail. Have backup info ready. Steve Busalacchi Director, News & Information Wisconsin Medical Society

38 Analyze! Worksite data set What is the average number of employees? Do worksites with smoking policies tend to be larger or smaller? Is there a need for health insurance coverage for cessation at these worksites? How many worksites have had cessation programs on site? What is the most important reason worksites have instituted smoking policies?

39 Myths One report is enough. People read written reports. Complex analysis and big words impress people. Oral reports have the same effect as written reports. Describing limitations weakens report. Everything should be reported. The audience knows why they are getting the report.

40 Online resources Evaluation assistance State and local data Economic impact of smoking policies

41

42

43

44

45

46 Making sense of your data What challenges have you faced? Top ten lessons: Review