TIFFANY COMER COOK, M.S. LAURA L. FELDMAN, ED.S. WYOMING SURVEY & ANALYSIS CENTER EVALUATION 2010: EVALUATION QUALITY SAN ANTONIO, TX NOVEMBER 13, 2010 Using Logic Models to Build Evaluation Capacity at the Community Level

The System Tobacco Prevention and Control (TP&C) Local program managers in 23 counties Local coalitions WYSAC (evaluators), WyPTAC (technical assistance), MHSASD (state agency) Large state with very few people and bad weather

The History Needs assessments Strategic plans/calendars Grant applications Previous logic models were not used to guide programming

The Goal Pilot an individualized approach to improve program implementation and achievement of outcomes Build evaluation capacity by…  Creating logic models to guide implementation of coalitions’ strategic plans  Creating evidence plans to show program success and improve program sustainability Use lessons learned for supplementary project

The Process: Choosing Participants Five volunteers Introductory phone conference

The Process: Logic Models Webinar about logic models Draft logic models One-on-one webinars on logic models Final logic models

The Process: Collecting Evidence Webinar on evidence and data collection Draft evidence plans One-on-one webinars on evidence plans Final evidence plans

The Process: Lessons Learned and Next Steps Face-to-face group meeting to obtain feedback Written feedback Guidance document for supplemental project

Core Content Logic models  Roadmap  A tool to assist  Components Doing vs. achieving (outputs vs. outcomes) The miracle effect Collecting data to reduce assumptions Being SMART and identifying achievable outcomes

Doing vs. Achieving (Outputs vs. Outcomes) You do activities. You achieve outcomes. Is what you’re doing impacting what you hope to achieve?

The Miracle Effect Media campaign on the health benefits of smokefree environments Then a miracle happens Comprehensive smokefree ordinance in Smalltown, WY

From Assuming to Knowing What are you assuming will come from an activity?  Example: Assumptions: The proper target audience (BUSINESS OWNERS) reads the brochures, comprehends them, learns something, and then changes not only their attitudes, but their behaviors, and THEN implements policies Distribute brochures at a health fair Increased awareness of the dangers of SHS More businesses with smokefree policies

From Assuming to Knowing We want assumptions to diminish Collecting evidence reduces assumptions  Example:  Who received your brochures?  Did they read and comprehend them?  Did they change beliefs and attitudes?  Did they change behavior?  Is this the group whose behavior you want to change?  Will brochures effectively change the behavior of your target group?  In other words, what results did you achieve because you participated in that health fair?

Identifying Outcomes What will change? (e.g., rates of youth smoking) For whom? (e.g., middle school students) By how much? (e.g., by one percentage point) By when? (e.g., by June 2011) How will you know you’ve achieved it? (e.g., as measured by PNA 2010)

Logic Model Eliminate Exposure to Secondhand Smoke WDH Funding Coalition Youth Coalition Input Activities Completed activities to disseminate information about secondhand smoke and smoke-free policies Completed activities to create and enforce smoke- free polices Number of pamphlets distributed about the dangers of second hand smoke. City council passes smoke- free ordinance. Outputs Change in knowledge, attitudes and behaviors regarding Secondhand smoke. Increased awareness of the need to eliminate exposure to second-hand smoke Increased commitment to protect citizens from secondhand smoke exposure Creation of tobacco free policies Outcomes Short-TermIntermediateLong-Term Compliance with smoke-free policies. Decrease in exposure to Secondhand smoke. Decrease in tobacco consumption Decrease in tobacco related costs, death and disability. Decrease in tobacco related health disparities. Fewer Citizens are exposed to second- hand smoke in public places. Right to Breathe Clean Air Media Campaign Monthly Newsletter Re-Messaging Media Campaign Recruit and Build Coalition Information dissemination and coalition recruiting at Bridger Valley Health Fair and Mini-Relay for Life Identify and work with policy makers (supportive and unsupportive of Smoke- free public policy

Logic Model Full-Functioning Coalition in Mt. View WDH Funding Input Activities Had conversation with 75% of former coalition members Booked venue for Special Event for April 15, 2010 Contacted everyone on mailing list; spoke to 30 people at events Outputs Reaffirmation of commitment from former coalition members Committed new coalition members Outcomes Short-TermIntermediateLong-Term Full- functioning coalition in Mt. View Essential members are committed to coalition (attorney, school rep, etc.) Contact main coalition members individually Contact those at the public hearing & invite to join coalition Find community meeting/events where Program Manager can present Hold special event (meal) to get interest in coalition Ask others who may be interested/network Coalition member recruits other members. Program Manager One coalition member Contacted all attendees by phone ACS WyPTAC Find out next step from city council 15 Coalition members attend Special Event Coalition Members willing to educate Mountain View Town Council Members

The Result “It taught me that I can regularly complete my own [small-scale] evaluation projects to ensure the work I am doing is having an impact in the community.” “I gained more insight on how effective certain activities were and that other activities are just busy work.” “With your help, I was able to design a better plan to implement my program.”

Lessons Learned The process removed the fear of logic models & evaluation  “It’s easier than I thought.” Individual, one-on-one attention was key to success Programs understood the importance of evaluation and connecting activities to outcomes Programs learned to prioritize activities Technology was great, but no substitute for face-to- face meetings

Lessons Learned All coalitions need to integrate process into strategic plans and grant applications  Know outcomes before choosing activities Need to increase communication among partners at all levels of involvement Participants had anxiety over being judged/graded Increase peer interaction Focus on documenting outputs before outcomes Need more time to report results and to assess level of evaluation capacity

Next Steps Implement the process with Cohort 2 (five county programs and one regional coalition) Site visits Full fiscal year Present the project in a way that’s useful and reassuring Provide slivers of information; piecemeal

Anticipated Challenges Moving Forward Tailoring the process to individual skills, coalitions, communities, and programs Covering more content Successfully addressing anxiety about the process Addressing the perception of course corrections Receptivity and honesty Reporting well vs. working effectively Wyoming’s unique challenges

Questions ???

Contact Information Tiffany Comer Cook, M.S. (307) Laura L. Feldman, Ed.S. (307)