WRITING GREAT TOBACCO CONTROL FINAL EVALUATION REPORTS WEBINAR TRAINING APRIL 4 & 5, 2013 Welcome!

AGENDA Thursday, April 4 th 10:00-10:10 Welcome! 10:10-10:45 The Project Description Section:: Overview, scoring the report, discussion 10:45-11:30 The Methods Section Overview, scoring the report, discussion Friday, April 5 th 10:00-10:45Writing Up Results & Conclusions, Overview 10:45-11:30Scoring and Discussion 2

WHY FINAL REPORTING? Organizational History Sharing with colleagues idi.org 3

STATEWIDE & NATIONAL RELEVANCE TCEC publishes summary reports and articles 4

Reporting Guidelines 5

FER RATING FORM (maximum 32 points) Report SectionMaximum Points Title Page2 Abstract6 Project Description6 Evaluation Methods8 Evaluation Results4 Conclusion & Recommendations 6 6

PROVIDING SOME CONTEXT: PROJECT DESCRIPTION SECTION

PROJECT DESCRIPTION SCORING (maximum 6 points) Project Description Maximum Points Background2 Objective2 Intervention2 8

PROJECT DESCRIPTION ELEMENTS  Provide context for the objective  Explain rationale for choosing the objective  State the objective and indicator  Describe what you did and why  Identify activity timeframes  Explain any changes to plan 9

IT’S YOUR TURN TO SCORE minutes Read and score the project description of this report Write on the report any questions you have 5-10 minutes Discuss scores as a group. 10

EVALUATION METHODS

DESIGN TYPE PROCESS AND OUTCOME  Experimental (control group, random assignment)  Quasi-experimental (at least three waves, comparison group)  Non-experimental (no comparison, only two waves)  Process and/or Outcome 12

DESIGN LIMITATIONS 13

SAMPLES 14

DATA COLLECTION INSTRUMENTS AND PROCEDURES 15

DATA ANALYSIS 16

EVALUATION METHODS SCORING (MAXIMUM 8 POINTS) Evaluation Methods Evaluation design2 Samples from which data were collected 2 Data collection instruments and procedures 2 Data analysis2 17

IT’S YOUR TURN TO SCORE minutes Read and score the Evaluation Methods section in pairs Write on the report any questions you have 5-10 minutes Discuss scores as a group. 18

SCORING METHODS SECTION 19

REMEMBER TO JOIN US TOMORROW! PART 2 begins at 10:00 a.m. Same login info: 1. Call in for audio at and enter access code , AND 2. Log in for the visuals at and enter the access code www.readytalk.com 20

 Our team — Diana Cassady, Jeanette Treiber, Robin Kipke, Catherine Dizon, Diana Dmitrevsky  Our website —  Our —  Our main line — We are the statewide technical assistance center on evaluation for all Prop. 99-funded projects in California 21

WRITING GREAT TOBACCO CONTROL FINAL EVALUATION REPORTS WEBINAR TRAINING APRIL 4 & 5, 2013 Part 2 Welcome!

AGENDA Thursday, April 4 th 10:00-10:10 Welcome! 10:10-10:45 The Project Description Section Overview, scoring the report, discussion 10:45-11:30 The Methods Section Overview, scoring the report, discussion Friday, April 5 th 10:00-10:45Writing Up Results & Conclusions, Overview 10:45-11:30Scoring and Discussion 23

WRITING UP RESULTS

BRAINSTORM  What is the purpose of the results section of a report?  What function(s) should it serve? 25

PURPOSE OF THE RESULTS SECTION  Present the findings of data analysis  Explain what the data reveal  Point out what is important  Describe how the data informed your project  Justify report conclusions 26

FER SCORING CRITERIA Evaluation ResultsMaximum Points Findings are clearly & logically presented. Tables and figures are used when needed and are clearly labeled. 2 Findings are objective (include both positive and negative when relevant) 2 Section total4 27

WHAT TO INCLUDE  Report on every activity  But don’t report every result 28

QUANTITATIVE DATA  Provide totals and percentages  Note statistical significance (where appropriate) (79%) Alta Vista residents were in favor of the policy

EXCEL-GENERATED TABLE 30 A paired T-test comparing 2010 and 2013 shows a statistically significant difference in the number of cigarette butts between 2010 and 2013 (p<0.05)

Table 1. Number of Cigarette Butts in Sacramento Area Parks, by Year Location Arden Sunrise Fair Oaks McKinley Goethe Total Citywide* * *Statistically significant difference p<0.05 VISUALLY IMPROVED TABLE

LINE GRAPH 32

BAR CHART 33

QUALITATIVE DATA  Common themes  Contrasts  Indicative quotes  Avoid percentages 34

SUMMARIZING CONTENT ANALYSIS 35 Figure 3. Incentives and Barriers to Smoke Free Casino Policy

BALANCING ACT  Focus on what’s most important  Report both + and - results  Make results transparent 36

ATTENTION GETTERS  Supplement text with visual elements  Draw focus to key findings  Make data digestible  Match format to data type 37

CONCLUSIONS & RECOMMENDATIONS

FER SCORING CRITERIA Conclusions & RecommendationsMaximum Points Discuss outcome/process evaluation2 Conclusions are supported by data2 Includes recommendations for future work2 Section total6 39

INTERPRETING THE FINDINGS  Confirm what you expected to find, and did  Dispel what you thought was true, but isn’t  Bring to light what you didn’t know, but should 40

LEAVING A ROADMAP  What strategies worked well or poorly?  What would you do differently next time?  What steps should follow? 41

IT’S YOUR TURN TO SCORE minutes: Read and score the results and conclusions/ recommendations section Note any questions you have minutes: Reconvene to share scores 42

AS YOU REVIEW THE RESULTS… 1. What questions does it leave unanswered? 2. Do the results address what the project wanted to know? 3. Did the section provide enough information for the reader to assess the findings? 4. Are the most important findings brought out in the narrative and/or the tables? 5. Do the findings appear to be objective? 6. Does the section adequately interpret the results by explaining how the data informed the project? 43

AS YOU REVIEW CONCLUSIONS 1. Do the conclusions evaluate the process and outcome findings in relation to the achievement of the objective? 2. Are the conclusions supported by the data? 3. Do the recommendations provide a clear roadmap for next time? 44

REPORT RATING  Total score: ___/32 possible points  Rating: ____ High (24-32) ____ Medium (16-23) ____ Low (0-15)  Overall Assessment:  Recommend report for intervention activities?  Recommend report for evaluation? 45

WORKSHOP RECAP  Why final evaluation reports matter  Essential components of a report  Provide enough detail about your path  Make rationale and linkages transparent  How reports are scored  Use ratings & feedback to improve 46

PARTICIPANT SELF-ASSESSMENT Help us learn how we’re doing. Complete the self-assessment form and rate your understanding of writing final evaluation reports before and after this training. Thank you! 47

 Our team — Diana Cassady, Jeanette Treiber, Robin Kipke, Catherine Dizon, Diana Dmitrevsky  Our website —  Our —  Our main line — We are the statewide technical assistance center on evaluation for all Prop. 99-funded projects in California 48