Evaluating Public Health Technical Assistance and Training Presented by: W. Douglas Evans, Ph.D. National Conference on Tobacco or Health San Francisco, CA November 2002 Presented by: W. Douglas Evans, Ph.D. National Conference on Tobacco or Health San Francisco, CA November 2002

Why evaluate TAT? TAT is essential for community-based components of comprehensive programs Knowing what works in TAT: Matching solutions with problems and programs TAT is costly, evaluation helps to target scarce resources most effectively Replicating effective TAT as state’s roll out comprehensive programs TAT is essential for community-based components of comprehensive programs Knowing what works in TAT: Matching solutions with problems and programs TAT is costly, evaluation helps to target scarce resources most effectively Replicating effective TAT as state’s roll out comprehensive programs

Theory Community-based programs need to build capacity to produce systemic change TAT has potential both for:  Short-term boost in program effectiveness &  Systemic changes in programs Community-level tobacco control indicators  Wakefield & Chaloupka (1998) How do we define community?  Definitions differ depending on context (Minkler & Wallerstein 2000) Community-based programs need to build capacity to produce systemic change TAT has potential both for:  Short-term boost in program effectiveness &  Systemic changes in programs Community-level tobacco control indicators  Wakefield & Chaloupka (1998) How do we define community?  Definitions differ depending on context (Minkler & Wallerstein 2000)

Relevant TAT Evaluations National tobacco control programs  ASSIST, Smokeless States State comprehensive programs  Florida SWAT, Indiana community coalitions SAMHSA Model Programs Decision Support HIV/AIDS community service providers  HRSA TAT Data Systems National tobacco control programs  ASSIST, Smokeless States State comprehensive programs  Florida SWAT, Indiana community coalitions SAMHSA Model Programs Decision Support HIV/AIDS community service providers  HRSA TAT Data Systems

Challenges in Evaluating TAT TAT assumes many forms and can be delivered in many ways  E.g., onsite, at-a-distance, products, etc. As a result, multiple process measures & data collection systems may be required TAT has multiple outcomes  E.g., Organizational improvement, capacity building, staff skill building Outcomes are influenced by many variables  E.g., Growth, Funding, Staff turnover, etc. TAT assumes many forms and can be delivered in many ways  E.g., onsite, at-a-distance, products, etc. As a result, multiple process measures & data collection systems may be required TAT has multiple outcomes  E.g., Organizational improvement, capacity building, staff skill building Outcomes are influenced by many variables  E.g., Growth, Funding, Staff turnover, etc.

Methods Ongoing TAT process measurement  Web-based program reporting systems Satisfaction surveys after TAT delivery Follow-up surveys to identify uses of TAT, systemic changes Long-term TAT tracking  Data systems with planned TAT recipient follow-up Ongoing TAT process measurement  Web-based program reporting systems Satisfaction surveys after TAT delivery Follow-up surveys to identify uses of TAT, systemic changes Long-term TAT tracking  Data systems with planned TAT recipient follow-up

Sample Measures Demographics  When, where, Who received, Type delivered Reasons for requesting/using TAT  Staff skills, knowledge, service delivery TAT Satisfaction  With delivery, products, other  Would recommend, met needs, compared to other TAT, met learning objectives Demographics  When, where, Who received, Type delivered Reasons for requesting/using TAT  Staff skills, knowledge, service delivery TAT Satisfaction  With delivery, products, other  Would recommend, met needs, compared to other TAT, met learning objectives

Sample Measures (2) Systems change  Services delivered, numbers served, use IT, responsiveness, efficiency, Products  Type of product, opinions of product Type of TAT received  Onsite, remote delivery, product Systems change  Services delivered, numbers served, use IT, responsiveness, efficiency, Products  Type of product, opinions of product Type of TAT received  Onsite, remote delivery, product

Example: Indiana Community Coalitions Funding local coalitions in 92 counties, and 29 special population coalitions Providing capacity building & skills-based training per CDC guidelines Tracking events, progress, TAT delivery, systemic changes related to TAT Web-based program tracking system (PTS) Funding local coalitions in 92 counties, and 29 special population coalitions Providing capacity building & skills-based training per CDC guidelines Tracking events, progress, TAT delivery, systemic changes related to TAT Web-based program tracking system (PTS)

Web-based PTS

Web-based PTS (2)

Web-based PTS (3)

Community Constructs Various ideas of the “unit of analysis” Community as a set of mediating structures:  Groups to which people belong Community as social institutions  Level of coordination or collaboration  Partnership literature is relevant Community as geographic or political entity  Characterized by power structures Various ideas of the “unit of analysis” Community as a set of mediating structures:  Groups to which people belong Community as social institutions  Level of coordination or collaboration  Partnership literature is relevant Community as geographic or political entity  Characterized by power structures

Community Constructs (2) Communities as mediating structures:  Psychological sense (membership, influence, fulfillment of needs, and emotional connection) Communities as social institutions  Provision of services, degree of collaboration, coordination among institutions Communities as power structures  Social capital constructs, geographic definitions (neighborhoods, municipalities, counties, etc.) Communities as mediating structures:  Psychological sense (membership, influence, fulfillment of needs, and emotional connection) Communities as social institutions  Provision of services, degree of collaboration, coordination among institutions Communities as power structures  Social capital constructs, geographic definitions (neighborhoods, municipalities, counties, etc.)

Indiana Community Indicators

Indiana measures Top down: How do funders define community? Bottom up: How do local coalitions define community? For what purposes? Examples:  School-based  Neighborhood  Jurisdiction of tobacco control ordinances  Catchment areas of tobacco control programs Top down: How do funders define community? Bottom up: How do local coalitions define community? For what purposes? Examples:  School-based  Neighborhood  Jurisdiction of tobacco control ordinances  Catchment areas of tobacco control programs

Conclusions TAT is critical to community-based components of tobacco control programs Evaluating TAT as independent and dependent variable(s) complex Proven examples for TAT evaluation Choice of TAT measurement must consider community, how defined for local program TAT is critical to community-based components of tobacco control programs Evaluating TAT as independent and dependent variable(s) complex Proven examples for TAT evaluation Choice of TAT measurement must consider community, how defined for local program