1. Development of a system for measuring sustainment of prevention programs and initiatives
Lawrence A. Palinkas, Suzanne Spear, Sapna Mendon, Juan Villamar, and C. Hendricks Brown

2. Grants: NIDA R34DA037516 & NIDA P30DA027828
Acknowledgements
Grants: NIDA R34DA & NIDA P30DA027828

3. Acknowledgements Center Dir. Frances Harding, CSAP
Center for Substance Abuse Prevention (CSAP)
Division of Community Programs
Division of State Programs
Center for Mental Health Services (CMHS)
Division of Prevention, Traumatic Stress, and Special Programs

4. Background
Sustaining prevention efforts directed at drug and alcohol abuse, HIV, and mental disorders as well as suicide is currently one of the greatest, yet least understood challenges in implementation science.
A large gap in knowledge exists regarding what is meant by the term “sustainment” and what factors predict or even measure sustainability of effective prevention programs and support systems, in communities, tribal nations, states, and the federal environment.

5. Background (Continued)
Essential to build a partnership between a federal implementation agency -- here SAMHSA -- and a research organization -- here the NIDA funded Center for Prevention Implementation Methodology (Ce-PIM), to develop a sustainment research tool that can be validated and then implemented within the agency legally mandated to lead federal efforts to support state, territory, and local behavioral prevention strategies.

6. Project Aims
Identify core components and their interrelationships across time for sustainability of prevention programs and their support infrastructures.
Design a measurement system for monitoring and providing feedback regarding sustainment within SAMHSA.
Pilot test the predictability of the Sustainment Measurement System (SMS) and the feasibility and acceptability of this system to evaluate and improve sustainment likelihood.

7. SAMHSA Ce-PIM Partnership
Four SAMHSA Prevention Programs:
Strategic Prevention Framework State Incentive Grant (SPF-SIG)
Sober Truth on Preventing Underage Drinking (STOP Act)
Garrett Lee Smith State Youth Suicide Prevention (GLS)
Implementing Prevention Practices in Schools (PPS)

8. SAMHSA Sustainment Study
Phase 1
Telephone interviews and site visits with 10 grantees
Interviews with 45 key staff.
Collection of grant proposals, progress reports, other documents.
Web-based survey of coalition social networks.

9. Methods Data Collection
Open-ended questions about experience with implementation and sustainment and identification of barriers and facilitators to sustainment.
Free list exercise to elicit participant conceptions of what is meant by the term sustainment, what elements of their program they wish to see sustained, and what it will take to sustain those program elements.
Checklist of domain elements from the Consolidated Framework for Implementation Research (CFIR: Damschroder et al., 2009).

10. Methods Data analysis Semi-structured interview
Grounded theory approach (Glaser & Straus, 1967) of coding, consensus, co-occurrence and comparison (Willms et al., 2992)
Coding: Three levels using Dedoose qualitative software
Consensus: Achieved an average of 91.25% agreement across three levels of codes
Comparison: Used for classifying individual codes into larger groups
Co-occurrence: Examining association between themes and participant role and pace of adoption

11. Methods Data analysis Free lists
Frequency counts of what is meant by sustainment/sustainability, program elements to be sustained, and requirements for sustainment.
Constant comparison to identify meaningful clusters of items representing similar constructs.
Rank order by total nominations of top ten items.

12. Methods Data analysis CFIR checklist
Percent of informants citing domain as high or very high importance.
Mean domain score (0 = not important, 1 = yes/no, 2 = important, 3 = very important).

13. Table 1. Themes and subthemes identified from semi-structured interview questions

14. Table 1. Themes and subthemes identified from semi-structured interview questions

15. Table 2. Percent of free list nominations of definition of sustainment, recommendations for what should be sustained, and requirements to sustainment

16. Table 2. Percent of free list nominations of definition of sustainment, recommendations for what should be sustained, and requirements to sustainment

17. Table 2. Percent of free list nominations of definition of sustainment, recommendations for what should be sustained, and requirements to sustainment

18. Table 3 CFIR Domain Percent of Importance by SAMHSA Program: 76-100%

19. Table 3 CFIR Domain Percent of Importance by SAMHSA Program: 76-100%

20. Table 3 CFIR Domain Percent of Importance by SAMHSA Program: 76-100%

21. Conclusions
Four elements for measurement identified by all three data sets
Ongoing coalitions, collaborations, and networks
Infrastructure and capacity to support sustainment
Ongoing evaluation of performance and outcomes
Availability of funding and resources

22. Conclusions Five elements identified by two of three data sets
Community need
Community buy-in and support
Leadership
Presence of a champion
Evidence of positive outcomes

23. Conclusions
Some differences in element priorities observed across the 4 SAMHSA programs
Norms. Values and guiding principles of organizations
Influence of other states, tribes, communities
Perception of current situation as needing change
Knowledge and beliefs about program
Coalitions, collaboration and networking
Each SAMHSA program supports different activities that grantees wish to see sustained

24. The Sustainment Measurement System
SUSTAINMENT INDICATORS (n = 4)
The project continues to operate as described in the original application for funding.
The project continues to deliver prevention services to its intended population
This project periodically measures the fidelity of the prevention services that are delivered.
The project continues to deliver prevention services that are evidence-based.

25. The Sustainment Measurement System
FUNDING AND FINANCIAL SUPPORT (n = 6)
The project is supported by federal, state or local government funding.
The project is funded through non-profit, private, and/or non-governmental sources.
The project has a combination of stable (i.e., earmarked) and flexible (i.e., discretionary) funding.
The project has sustained funding.
Diverse community organizations are financially invested in the success of the project.
The project is financially solvent.

26. The Sustainment Measurement System
RESPONSIVENESS TO COMMUNITY NEEDS AND VALUES (n = 7)
The project delivered meets the needs of the intended target populations.
The project addresses the behavioral health needs of the communities/populations being served.
The project can be adapted to meet the needs of the communities or populations being served.
The project is consistent with the norms, values and guiding principles of participating organizations.
The project fits well with the values of the organization(s) responsible for sustaining it and the communities where it is being sustained.
Participating organizations have a shared perception of the importance of the project.
The current social or health issue addressed by the project is perceived as intolerable or unacceptable to the community.

27. The Sustainment Measurement System
COALITIONS, PARTNERSHIPS AND NETWORKS (n = 8)
The grantee organization is networked with other organizations committed to sustaining the programs funded by the SAMHSA grant.
The community members are passionately committed to sustaining the project.
The community is actively engaged in the development of project goals.
The project is supported by a coalition/partnership/network of community organizations.
Coalition/partnership/network members actively seek to expand the network of community organizations, leaders, and sources of support for this project.
The coalition/partnership/network is committed to the continued operation of this project.
There is a high level of networking and communication within the organizations responsible for sustaining the project.
The community has access to knowledge and information about the project.

28. The Sustainment Measurement System
INFRASTRUCTURE AND CAPACITY TO SUPPORT SUSTAINMENT (n = 9)
There are available resources dedicated for implementing and sustaining the project
The project exhibits sound fiscal management
The project is well integrated into the operations of the organization and its partners
Plans for implementing and sustaining the project are developed in advance
The project is carried out or accomplished according to plan
The project has adequate staff to sustain the program’s goals and activities
The project offers sufficient training to agency staff and community members
Staff possess adequate knowledge and supportive beliefs about the project
Staff feel themselves to be capable of implementing the project

29. The Sustainment Measurement System
LEADERSHIP (n = 5)
Leaders in organization or coalition/partnership/network are actively engaged in the process of implementing and sustaining the program.
Community leaders are actively involved in the project.
The program has staff who are formally appointed to coordinate the process of implementing and sustaining the project.
The project is also supported by a champion who is actively engaged in the process of implementing and sustaining the project.
“We have a process in place to implement the program in the event our champion leaves”.

30. The Sustainment Measurement System
MONITORING AND EVALUATION (n = 2)
There is ongoing evaluation of progress made towards sustainment.
There is sufficient and timely feedback about project delivery to improve implementation and quality.
OUTCOMES (n = 1)
The program provides strong evidence of positive outcomes.

31. Next steps
Administer 50 item scale (42 domain items + 8 additional project status items) to 140+ SAMHSA grantees.
Develop guidelines for offering feedback to grantees based on data collected from SMS.
Test effectiveness of guidelines on sustainment as an outcome.