Development of a system for measuring sustainment of prevention programs and initiatives Lawrence A. Palinkas, Suzanne Spear, Sapna Mendon, Juan Villamar,

Slides:



Advertisements
Similar presentations
RTI International is a trade name of Research Triangle Institute Technical Assistance to North Carolinas Health & Wellness Trust Funds TUPC.
Advertisements

Expanding & Sustaining Systems of Care: New Challenges and Opportunities Presentation Beaver County (PA) System of Care: Optimizing Resources, Education.
Reclaiming Futures as part of the OJJDP required Three-Year State Plan A Presentation to State SAGs Tom Begich.
Sustainability Planning Pat Simmons Missouri Department of Health and Senior Services.
Family Resource Center Association January 2015 Quarterly Meeting.
1 Minority SA/HIV Initiative MAI Training SPF Step 3 – Planning Presented By: Tracy Johnson, CSAP’s Central CAPT Janer Hernandez, CSAP’s Northeast CAPT.
Presented By: Tracy Johnson, Central CAPT
PHAB's Approach to Internal and External Evaluation Jessica Kronstadt | Director of Research and Evaluation | November 18, 2014 APHA 2014 Annual Meeting.
Community Planning Training 1-1. Community Plan Implementation Training 1- Community Planning Training 1-3.
How to Develop the Right Research Questions for Program Evaluation
1-2 Training of Process FacilitatorsTraining of Coordinators 5-1.
Webinar: Leadership Teams October 2013: Idaho RTI.
Participatory Evaluation Mary Phillips, BME Former Circles of Care Program Coordinator, Oakland and an Evaluator, Los Angeles, CA.
Program Collaboration and Service Integration: An NCHHSTP Green paper Kevin Fenton, M.D., Ph.D., F.F.P.H. Director National Center for HIV/AIDS, Viral.
School-Wide Positive Behavior Support District Planning Louisiana Positive Behavior Support Project.
Planning for Sustainability National Child Traumatic Stress Network All Network Meeting February 6, 2007.
KENTUCKY YOUTH FIRST Grant Period August July
Strategic Prevention Framework Overview Paula Feathers, MA.
Colorado Prevention Partners SPF-SIG Overview Colorado Prevention Partners.
Overview June,  Sub-recipients grant applications will go to ADAMHS/ADAS Boards only.  ADAMHS/ADAS Boards will be expected to identify a primary.
Organizational Conditions for Effective School Mental Health
Crosswalk of Public Health Accreditation and the Public Health Code of Ethics Highlighted items relate to the Water Supply case studied discussed in the.
Community Partnerships to Protect Children: Challenges and Opportunities Deborah Daro.
Partnership Analysis & Enhancement Tool Kit Cindy S. Soloe Research Triangle Institute (RTI) April Y. Vance Centers for Disease Control and Prevention.
State of California Department of Alcohol and Drug Programs State Incentive Grant Project Overview Michael Cunningham Deputy Director, Program Services.
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
Sustainability Planning A Framework for Strategic Financing.
1 Center Mission Statements SAMHSA ? CSAT Improving the Health of the Nation by Bringing Effective Alcohol and Drug Treatment to Every Community CMHS Caring.
Presentation By L. M. Baird And Scottish Health Council Research & Public Involvement Knowledge Exchange Event 12 th March 2015.
RE-AIM Framework. RE-AIM: A Framework for Health Promotion Planning, Implementation and Evaluation Are we reaching the intended audience? Is the program.
Implementation Science: Finding Common Ground and Perspectives Laura Reichenbach, Evidence Project, Population Council International Conference on Family.
DBHDS Vision: A life of possibilities for all Virginians NEW INITIATIVES IN BEHAVIORAL HEALTH WELLNESS Office of Behavioral Health Wellness Virginia Department.
Approaches to Linking Process and Outcome Data in a Cross-Site Evaluation Laura Elwyn, Ph.D. Kristin Stainbrook, Ph.D. American Evaluation Association.
Wisconsin Personnel Development System Grant Click on the speaker to listen to each slide. You may wish to follow along in your WPDM Guide.
Monitoring & Evaluation Capacity Strengthening Workshop WORKSHOP INTRODUCTION AND OVERVIEW.
Caring Across Communities Collaborative Empower ment Across Communiti es Portland, Maine Lead Agency: Multilingual and Multicultural Center Portland Public.
Community Resilience Jill J Artzberger, MPH 2011 Texas Emergency Management Conference Thursday, April 28, 2011.
First Things First Grantee Overview.
National Coalition Academy Summary
Framework for Getting Results at Scale
Building and Sustaining a Multicultural Partnership
Capacity Building.
Maryland Healthy Transition Initiative
Identifying Necessary and Sufficient Conditions for Sustainment of Evidence-Based Substance Abuse and Mental Health Programs Sapna J. Mendon, Lawrence.
Florida’s MTSS Project: Self-Assessment of MTSS (SAM)
Cultural Competence and Consumer Involvement: Practice and Theory
Annual Plan Earlier this week, the SNA Board reviewed the progress we have made to date on the new Strategic Plan that was introduced last year.
Sustainability of Prevention Programs and Initiatives: A Community Building Framework Suzanne Spear, Lawrence A. Palinkas, Sapna Mendon, Juan Villamar,
Overview – Guide to Developing Safety Improvement Plan
RtI Innovations: Evaluation Anna Harms & Jose Castillo
AspireMN Member Meeting
Overview – Guide to Developing Safety Improvement Plan
Strategic Prevention Framework – Capacity Building
Partnering with 12 community sectors:
BSAS Quarterly Coordinator Meeting Friday, October 28, 2016
Identification of Infrastructure Gaps to
Cheryl Holton, Program Director
As we reflect on policies and practices for expanding and improving early identification and early intervention for youth, I would like to tie together.
Continuity Guidance Circular Webinar
Strategic Prevention Framework - Introduction
SAMHSA’S FY 2018 BUDGET As Proposed in the President’s Budget.
Key Stakeholders are aware of the Coalitions activities
Service Array Assessment and Planning Purposes
Deconstructing Standard 2a Dr. Julie Reffel Valdosta State University
The Trauma-Informed Community Network of Greater Richmond: Building a Resilient Community Lisa Wright, MSW, LCSW and Melissa McGinn, MSW, LCSW Greater.
Bob Flewelling Amy Livingston
Bob Flewelling Amy Livingston
Comprehensive M&E Systems
MODULE 11: Creating a TSMO Program Plan
Presentation transcript:

Development of a system for measuring sustainment of prevention programs and initiatives Lawrence A. Palinkas, Suzanne Spear, Sapna Mendon, Juan Villamar, and C. Hendricks Brown

Grants: NIDA R34DA037516 & NIDA P30DA027828 Acknowledgements Grants: NIDA R34DA037516 & NIDA P30DA027828

Acknowledgements Center Dir. Frances Harding, CSAP Center for Substance Abuse Prevention (CSAP) Division of Community Programs Division of State Programs Center for Mental Health Services (CMHS) Division of Prevention, Traumatic Stress, and Special Programs

Background Sustaining prevention efforts directed at drug and alcohol abuse, HIV, and mental disorders as well as suicide is currently one of the greatest, yet least understood challenges in implementation science. A large gap in knowledge exists regarding what is meant by the term “sustainment” and what factors predict or even measure sustainability of effective prevention programs and support systems, in communities, tribal nations, states, and the federal environment.

Background (Continued) Essential to build a partnership between a federal implementation agency -- here SAMHSA -- and a research organization -- here the NIDA funded Center for Prevention Implementation Methodology (Ce-PIM), to develop a sustainment research tool that can be validated and then implemented within the agency legally mandated to lead federal efforts to support state, territory, and local behavioral prevention strategies.

Project Aims Identify core components and their interrelationships across time for sustainability of prevention programs and their support infrastructures. Design a measurement system for monitoring and providing feedback regarding sustainment within SAMHSA. Pilot test the predictability of the Sustainment Measurement System (SMS) and the feasibility and acceptability of this system to evaluate and improve sustainment likelihood.

SAMHSA Ce-PIM Partnership Four SAMHSA Prevention Programs: Strategic Prevention Framework State Incentive Grant (SPF-SIG) Sober Truth on Preventing Underage Drinking (STOP Act) Garrett Lee Smith State Youth Suicide Prevention (GLS) Implementing Prevention Practices in Schools (PPS)

SAMHSA Sustainment Study Phase 1 Telephone interviews and site visits with 10 grantees Interviews with 45 key staff. Collection of grant proposals, progress reports, other documents. Web-based survey of coalition social networks.

Methods Data Collection Open-ended questions about experience with implementation and sustainment and identification of barriers and facilitators to sustainment. Free list exercise to elicit participant conceptions of what is meant by the term sustainment, what elements of their program they wish to see sustained, and what it will take to sustain those program elements. Checklist of domain elements from the Consolidated Framework for Implementation Research (CFIR: Damschroder et al., 2009).

Methods Data analysis Semi-structured interview Grounded theory approach (Glaser & Straus, 1967) of coding, consensus, co-occurrence and comparison (Willms et al., 2992) Coding: Three levels using Dedoose qualitative software Consensus: Achieved an average of 91.25% agreement across three levels of codes Comparison: Used for classifying individual codes into larger groups Co-occurrence: Examining association between themes and participant role and pace of adoption

Methods Data analysis Free lists Frequency counts of what is meant by sustainment/sustainability, program elements to be sustained, and requirements for sustainment. Constant comparison to identify meaningful clusters of items representing similar constructs. Rank order by total nominations of top ten items.

Methods Data analysis CFIR checklist Percent of informants citing domain as high or very high importance. Mean domain score (0 = not important, 1 = yes/no, 2 = important, 3 = very important).

Table 1. Themes and subthemes identified from semi-structured interview questions

Table 1. Themes and subthemes identified from semi-structured interview questions

Table 2. Percent of free list nominations of definition of sustainment, recommendations for what should be sustained, and requirements to sustainment

Table 2. Percent of free list nominations of definition of sustainment, recommendations for what should be sustained, and requirements to sustainment

Table 2. Percent of free list nominations of definition of sustainment, recommendations for what should be sustained, and requirements to sustainment

Table 3 CFIR Domain Percent of Importance by SAMHSA Program: 76-100%

Table 3 CFIR Domain Percent of Importance by SAMHSA Program: 76-100%

Table 3 CFIR Domain Percent of Importance by SAMHSA Program: 76-100%

Conclusions Four elements for measurement identified by all three data sets Ongoing coalitions, collaborations, and networks Infrastructure and capacity to support sustainment Ongoing evaluation of performance and outcomes Availability of funding and resources

Conclusions Five elements identified by two of three data sets Community need Community buy-in and support Leadership Presence of a champion Evidence of positive outcomes

Conclusions Some differences in element priorities observed across the 4 SAMHSA programs Norms. Values and guiding principles of organizations Influence of other states, tribes, communities Perception of current situation as needing change Knowledge and beliefs about program Coalitions, collaboration and networking Each SAMHSA program supports different activities that grantees wish to see sustained

The Sustainment Measurement System SUSTAINMENT INDICATORS (n = 4) The project continues to operate as described in the original application for funding. 1 2 3 4 5 6 7 The project continues to deliver prevention services to its intended population This project periodically measures the fidelity of the prevention services that are delivered. The project continues to deliver prevention services that are evidence-based.

The Sustainment Measurement System FUNDING AND FINANCIAL SUPPORT (n = 6) The project is supported by federal, state or local government funding. 1 2 3 4 5 6 7 The project is funded through non-profit, private, and/or non-governmental sources. The project has a combination of stable (i.e., earmarked) and flexible (i.e., discretionary) funding. The project has sustained funding. Diverse community organizations are financially invested in the success of the project. The project is financially solvent.

The Sustainment Measurement System RESPONSIVENESS TO COMMUNITY NEEDS AND VALUES (n = 7) The project delivered meets the needs of the intended target populations. 1 2 3 4 5 6 7 The project addresses the behavioral health needs of the communities/populations being served. The project can be adapted to meet the needs of the communities or populations being served. The project is consistent with the norms, values and guiding principles of participating organizations. The project fits well with the values of the organization(s) responsible for sustaining it and the communities where it is being sustained. Participating organizations have a shared perception of the importance of the project. The current social or health issue addressed by the project is perceived as intolerable or unacceptable to the community.

The Sustainment Measurement System COALITIONS, PARTNERSHIPS AND NETWORKS (n = 8) The grantee organization is networked with other organizations committed to sustaining the programs funded by the SAMHSA grant. 1 2 3 4 5 6 7 The community members are passionately committed to sustaining the project. The community is actively engaged in the development of project goals. The project is supported by a coalition/partnership/network of community organizations. Coalition/partnership/network members actively seek to expand the network of community organizations, leaders, and sources of support for this project. The coalition/partnership/network is committed to the continued operation of this project. There is a high level of networking and communication within the organizations responsible for sustaining the project. The community has access to knowledge and information about the project.

The Sustainment Measurement System INFRASTRUCTURE AND CAPACITY TO SUPPORT SUSTAINMENT (n = 9) There are available resources dedicated for implementing and sustaining the project 1 2 3 4 5 6 7 The project exhibits sound fiscal management The project is well integrated into the operations of the organization and its partners Plans for implementing and sustaining the project are developed in advance The project is carried out or accomplished according to plan The project has adequate staff to sustain the program’s goals and activities The project offers sufficient training to agency staff and community members Staff possess adequate knowledge and supportive beliefs about the project Staff feel themselves to be capable of implementing the project

The Sustainment Measurement System LEADERSHIP (n = 5) Leaders in organization or coalition/partnership/network are actively engaged in the process of implementing and sustaining the program. 1 2 3 4 5 6 7 Community leaders are actively involved in the project. The program has staff who are formally appointed to coordinate the process of implementing and sustaining the project. The project is also supported by a champion who is actively engaged in the process of implementing and sustaining the project. “We have a process in place to implement the program in the event our champion leaves”.

The Sustainment Measurement System MONITORING AND EVALUATION (n = 2) There is ongoing evaluation of progress made towards sustainment. 1 2 3 4 5 6 7 There is sufficient and timely feedback about project delivery to improve implementation and quality. OUTCOMES (n = 1) The program provides strong evidence of positive outcomes.

Next steps Administer 50 item scale (42 domain items + 8 additional project status items) to 140+ SAMHSA grantees. Develop guidelines for offering feedback to grantees based on data collected from SMS. Test effectiveness of guidelines on sustainment as an outcome.