1. WRITING GREAT TOBACCO CONTROL FINAL EVALUATION REPORTS WEBINAR TRAINING APRIL 4 & 5, 2013 Welcome!

2. AGENDA Thursday, April 4 th 10:00-10:10 Welcome! 10:10-10:45 The Project Description Section:: Overview, scoring the report, discussion 10:45-11:30 The Methods Section Overview, scoring the report, discussion Friday, April 5 th 10:00-10:45Writing Up Results & Conclusions, Overview 10:45-11:30Scoring and Discussion 2

3. WHY FINAL REPORTING? Organizational History Sharing with colleagues idi.org 3

4. STATEWIDE & NATIONAL RELEVANCE TCEC publishes summary reports and articles 4

5. Reporting Guidelines 5

6. FER RATING FORM (maximum 32 points) Report SectionMaximum Points Title Page2 Abstract6 Project Description6 Evaluation Methods8 Evaluation Results4 Conclusion & Recommendations 6 6

7. PROVIDING SOME CONTEXT: PROJECT DESCRIPTION SECTION

8. PROJECT DESCRIPTION SCORING (maximum 6 points) Project Description Maximum Points Background2 Objective2 Intervention2 8

9. PROJECT DESCRIPTION ELEMENTS  Provide context for the objective  Explain rationale for choosing the objective  State the objective and indicator  Describe what you did and why  Identify activity timeframes  Explain any changes to plan 9

10. IT’S YOUR TURN TO SCORE 10-15 minutes Read and score the project description of this report Write on the report any questions you have 5-10 minutes Discuss scores as a group. 10

11. EVALUATION METHODS

12. DESIGN TYPE PROCESS AND OUTCOME  Experimental (control group, random assignment)  Quasi-experimental (at least three waves, comparison group)  Non-experimental (no comparison, only two waves)  Process and/or Outcome 12

13. DESIGN LIMITATIONS 13

14. SAMPLES 14

15. DATA COLLECTION INSTRUMENTS AND PROCEDURES 15

16. DATA ANALYSIS 16

17. EVALUATION METHODS SCORING (MAXIMUM 8 POINTS) Evaluation Methods Evaluation design2 Samples from which data were collected 2 Data collection instruments and procedures 2 Data analysis2 17

18. IT’S YOUR TURN TO SCORE 10-15 minutes Read and score the Evaluation Methods section in pairs Write on the report any questions you have 5-10 minutes Discuss scores as a group. 18

19. SCORING METHODS SECTION 19

20. REMEMBER TO JOIN US TOMORROW! PART 2 begins at 10:00 a.m. Same login info: 1. Call in for audio at 1.866.740.1260 and enter access code 2974659, AND 2. Log in for the visuals at www.readytalk.com and enter the access code 2974659www.readytalk.com 20

21.  Our team — Diana Cassady, Jeanette Treiber, Robin Kipke, Catherine Dizon, Diana Dmitrevsky  Our website — http://tobaccoeval.ucdavis.eduhttp://tobaccoeval.ucdavis.edu  Our email — tobaccoeval@ucdavis.edutobaccoeval@ucdavis.edu  Our main line — 530.752.9951 We are the statewide technical assistance center on evaluation for all Prop. 99-funded projects in California 21

22. WRITING GREAT TOBACCO CONTROL FINAL EVALUATION REPORTS WEBINAR TRAINING APRIL 4 & 5, 2013 Part 2 Welcome!

23. AGENDA Thursday, April 4 th 10:00-10:10 Welcome! 10:10-10:45 The Project Description Section Overview, scoring the report, discussion 10:45-11:30 The Methods Section Overview, scoring the report, discussion Friday, April 5 th 10:00-10:45Writing Up Results & Conclusions, Overview 10:45-11:30Scoring and Discussion 23

24. WRITING UP RESULTS

25. BRAINSTORM  What is the purpose of the results section of a report?  What function(s) should it serve? 25

26. PURPOSE OF THE RESULTS SECTION  Present the findings of data analysis  Explain what the data reveal  Point out what is important  Describe how the data informed your project  Justify report conclusions 26

27. FER SCORING CRITERIA Evaluation ResultsMaximum Points Findings are clearly & logically presented. Tables and figures are used when needed and are clearly labeled. 2 Findings are objective (include both positive and negative when relevant) 2 Section total4 27

28. WHAT TO INCLUDE  Report on every activity  But don’t report every result 28

29. QUANTITATIVE DATA  Provide totals and percentages  Note statistical significance (where appropriate) 29 202 (79%) Alta Vista residents were in favor of the policy

30. EXCEL-GENERATED TABLE 30 A paired T-test comparing 2010 and 2013 shows a statistically significant difference in the number of cigarette butts between 2010 and 2013 (p<0.05)

31. Table 1. Number of Cigarette Butts in Sacramento Area Parks, by Year Location2010201120122013 Arden68728137 Sunrise28234331445 Fair Oaks12118019658 McKinley312369231111 Goethe216248239222 Total Citywide*99912121061*473 31 *Statistically significant difference p<0.05 VISUALLY IMPROVED TABLE

32. LINE GRAPH 32

33. BAR CHART 33

34. QUALITATIVE DATA  Common themes  Contrasts  Indicative quotes  Avoid percentages 34

35. SUMMARIZING CONTENT ANALYSIS 35 Figure 3. Incentives and Barriers to Smoke Free Casino Policy

36. BALANCING ACT  Focus on what’s most important  Report both + and - results  Make results transparent 36

37. ATTENTION GETTERS  Supplement text with visual elements  Draw focus to key findings  Make data digestible  Match format to data type 37

38. CONCLUSIONS & RECOMMENDATIONS

39. FER SCORING CRITERIA Conclusions & RecommendationsMaximum Points Discuss outcome/process evaluation2 Conclusions are supported by data2 Includes recommendations for future work2 Section total6 39

40. INTERPRETING THE FINDINGS  Confirm what you expected to find, and did  Dispel what you thought was true, but isn’t  Bring to light what you didn’t know, but should 40

41. LEAVING A ROADMAP  What strategies worked well or poorly?  What would you do differently next time?  What steps should follow? 41

42. IT’S YOUR TURN TO SCORE 10-15 minutes: Read and score the results and conclusions/ recommendations section Note any questions you have 10-15 minutes: Reconvene to share scores 42

43. AS YOU REVIEW THE RESULTS… 1. What questions does it leave unanswered? 2. Do the results address what the project wanted to know? 3. Did the section provide enough information for the reader to assess the findings? 4. Are the most important findings brought out in the narrative and/or the tables? 5. Do the findings appear to be objective? 6. Does the section adequately interpret the results by explaining how the data informed the project? 43

44. AS YOU REVIEW CONCLUSIONS 1. Do the conclusions evaluate the process and outcome findings in relation to the achievement of the objective? 2. Are the conclusions supported by the data? 3. Do the recommendations provide a clear roadmap for next time? 44

45. REPORT RATING  Total score: ___/32 possible points  Rating: ____ High (24-32) ____ Medium (16-23) ____ Low (0-15)  Overall Assessment:  Recommend report for intervention activities?  Recommend report for evaluation? 45

46. WORKSHOP RECAP  Why final evaluation reports matter  Essential components of a report  Provide enough detail about your path  Make rationale and linkages transparent  How reports are scored  Use ratings & feedback to improve 46

47. PARTICIPANT SELF-ASSESSMENT Help us learn how we’re doing. Complete the self-assessment form and rate your understanding of writing final evaluation reports before and after this training. Thank you! 47

48.  Our team — Diana Cassady, Jeanette Treiber, Robin Kipke, Catherine Dizon, Diana Dmitrevsky  Our website — http://tobaccoeval.ucdavis.eduhttp://tobaccoeval.ucdavis.edu  Our email — tobaccoeval@ucdavis.edutobaccoeval@ucdavis.edu  Our main line — 530.752.9951 We are the statewide technical assistance center on evaluation for all Prop. 99-funded projects in California 48