1. Innovations in Prevention & Implementation Science: Implications for Improvement in Schools C Hendricks Brown, Northwestern University Feinberg School of Medicine Sheppard Kellam, Johns Hopkins University Jeanne Poduska, American Institutes for Research Juan Villamar, Northwestern University Feinberg School of Medicine Gracelyn Cruden, Northwestern University Feinberg School of Medicine 1

2. Acknowledgments: for Ce-PIM National Institute on Drug Abuse: CENTER FOR PREVENTION IMPLEMENTATION METHODOLOGY FOR DRUG ABUSE AND SEXUAL RISK BEHAVIOR (PI Brown P30 DA027828) Substance Abuse and Mental Health Services Administration SAMHSA Ce-PIM Partnership 2

3. Acknowledgements: Work to Date Funding Agencies: IES NIDA NIMH NICHD 3

4. Process in this Workshop: Active Engagement 4

5. 1.A Challenging Tour of Implementation Science and its Home in Service Delivery Systems 2.Public Education and Public Health: Toward a System of Systems 3.Systems Thinking and Partnerships in Daily Life 4.Illustrative Innovations of Ce-PIM 5.Summary: Building an Integrated Structure to Support Children and Youth Reaching their Full Potential 5 Workshop Outline

6. 1. A Challenging Tour of Implementation Science and its Home in Service Delivery Systems The Perspective at Ce-PIM A. Implementation as an action strategy working on systems change B. Why Study Implementation? C. Approaches to Moving Effective Interventions into Practice D. Three Interacting and Evolving Components of Implementation E. A Systems Approach involving 3 interacting components Small group exercise 6

7. Ce-PIM Mission: Building scientific and methodologic rigor into implementation research and practice. AIMS: 1.Develop Systems Methods (Systems Science, Engineering, Computational) Methods for Implementation Science 2.Infuse systems methods into the field of Implementation Research 3.integrating methods within the practice of prevention implementation at the federal, state, county, and local levels. 7

8. A. What is Implementation Research? According to NIH (2008): The use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within and across specific systems Action Oriented Within Settings or Systems AND collects data Chambers DA. Advancing the science of implementation: A workshop summary. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1-2):3-10. 8

9. B. Why Study Implementation? 1. We know a lot about what works 10K reviewed studies in What Works Clearinghouse 2. We are short on implementation action strategies to put what works into practice: WWC Practice Guides: Reducing Behavior Problems in the Elementary School Classroom 2008 3. It takes too long for research to affect practice 9

10. Closing the Gap between What We “Know” Works and What We Do. “17 Year Gap” in Health Care 10

11. 17 Years in Implementing an Statistical Methodology Generalized Estimating Equations (GEE: Brown et al., APMH 2012) 11

12. Is the gap between research and practice similar in education to that existing in health? Types of Gaps? As long as? As important to shorten? Which way? As resistant to change? 12

13. C. Approaches to Moving Effective Programs into Practice 13

14. Making a Program Work Does a Program Work? Could a Program Work? Traditional Research Pipeline for Implementation 14 IOM 2009 Landsverk, Brown et al. 2012 Aarons et al., 2011 Implementation Exploration Adoption / Preparation Implementation Sustainment Effectiveness Studies Efficacy Studies Preintervention Traditional Translation Pipeline Real World Relevance Local knowledge Generalized knowledge Intervention

15. A Focus on the Intervention itself is Not Sufficient The use of effective interventions without implementation strategies is like serum without a syringe; the cure is available, but the delivery system is not Fixsen, Blase, Duda, Naoom, Van Dyke,2010. 15

16. D. Three Interacting and Evolving Components of Implementation (Chambers et al., 2013 Imp Sci) Intervention: Program, Practice, Policy, Principles Practice Setting: Delivery Support System Ecological System: Population and Community/Cultural Context 16

17. Key Parameters on Implementation Science A. Transitions of Research Questions: Effectiveness – Can a Program Work in real life systems? Implementation – Making a Program Work B. 4 Stages: Exploratory, Adoption, Implement w/ Fidelity, Sustainability C. Focus on Interactions of Program, Delivery System, and Ecology D. Science: Produce Generalizable Knowledge rather than Local Knowledge Measurement, Modeling, and Testing Using Rigorous Methods E. Partnerships 17

18. Definition of System A set of connected parts forming a complex whole 18

19. Group Exercise: Produce a System Map of the Educational System Three Steps: 1.Who needs to be at the table? What are the components needed represented? 2.How do they Interact with one another? 3.Where would you intervene to affect intervention usage? Education Policy, i.e. Common Core Standards Identify one person from your small group to bring this back to the group 19

20. Follow-up Question on Systems: Would your list of players and location intervention differ based on implementing New policy Replacement of an existing curriculum Removal of a expulsion policy 20

21. Summary of Part 1 Multi-goal, Interacting, Dynamic and Multilevel System System View Looks at Interactions between components. Behavior of a system is not linear: complex. A useful way to understand a system is to intervene and study its behavior. 21

22. 2. Public Education and Public Health: Toward a System of Systems Sheppard G. Kellam, MD Center for Prevention Implementation Methodology Johns Hopkins Bloomberg School of Public Health Society for Research on Educational Effectiveness Washington, DC, September 4-5, 2014

23. The Baltimore Education and Prevention Partnership  The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials.  Trials were directed at helping children master obeying rules of behaving, attending, academic learning, socializing appropriately in 1 st grade classroom.  Interventions were tested separately in 1 st generation (our focus today), then together in later trials. 23

24. Two Dimensional View of Mental Health Social Adaptational Status—level of success or failure in main social fields at each stage of life Psychological/physical Well-Being—includes affect, cognitive development, i.e., internal condition of the individual 24

25. Kellam/Langevin, 2003 DEVELOPMENTAL EPIDEMIOLOGY: directed at early proximal targets MORE IMMEDIATE RISK: directed at more recent proximal targets COMMUNITY PREVENTION: directed at Community & School proximal targets COMMUNITY / SOCIETAL: directed at Policies & Laws as proximal targets Prevention Research and Service Strategies 25

26. Universal Selective Indicated Rx Med, MH, Soc Welfare Levels of Prevention and Treatment 26

27. Current Problems in System Coordination Prevention and treatment not integrated Service Agencies are Silos—minimal collaboration No shared information system monitoring developmental progress Mystery as to who gets “referred” for services Little follow-up of “no-shows” 27

28. Early Risk in Prevention Research  Over the last four decades much has been learned about early risk factors and paths leading to drug abuse, and other behavioral, mental health, and school problems.  Most if not all are strongly related to school failure, also a major risk factor for later drug abuse, alcohol, tobacco, depression, anti-social and other problem outcomes.  Aggressive, disruptive behavior as early as 1 st grade has been repeatedly found a risk factor for later drug and alcohol abuse and disorders, delinquency, violence, tobacco use, high risk sex, school failure and other high risk behaviors. 28

29. When a child misbehaves… 29

30. The Baltimore Education and Prevention Partnership The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials. They were directed at helping children master key social task demands in 1 st grade classroom. Interventions were tested separately, then together. The 1 st generation will be our main focus here, where the Good Behavior Game (GBG) was tested by itself and the children were followed up at ages 19-21 and beyond. 30

31. High Risk Children in Well vs. Poorly-Managed Classrooms (control classrooms only) If the top 25% of children on aggressive behavior were in disrupted classrooms, their risk of severe aggressive behavior by middle school was up to 59 times the average child’s. If similar children were in well-managed classrooms, the risk was up to 2.7 times the average child’s. 31

32. Kellam et al., Addiction Science & Clinical Practice; 6:73-84, 2011 32

33. The Role of School Information Systems in our New System of Systems Following individual children over time, schools and communities is required for assessing their progress, needs and for integrating services across agencies Information systems tracking each child exist in most school districts and can be extended and used to integrate services and monitor progress Confidentiality is by law guaranteed by school authorities and can be expanded to include data as needed for services 33

34. Partnerships for Building and Overseeing the New System of Systems Analyze political constituencies Engage and work through trust with each Learn their mission and vision Identify mutual self-interests and Encourage their coming together as oversight committee 34

35. Whole Group Activity: Partnerships What do we mean by “who” needs to be included? The nature of the process of engaging and developing the base of support/collaboration. 35

36. 3. Systems Thinking and Partnerships in Daily Life Jeanne Poduska, Sc.D. American Institutes for Research Society for Research on Educational Effectiveness Washington, DC, September 4-5, 2014 36

37. Systems Concepts Introduction of programs and action strategies requires adaptation to systems, e.g. staffing, fiscal allocations We operate in complex systems: multi-level, multi-actor Change occurs locally (even if simultaneously) Systems thinking focuses on interactions 37

38. Working Definition of Scaling-Out “Scaling out is the deliberate use of strategies to implement and sustain evidence-based interventions through or across settings to promote the greatest public health impact”. Ce-PIM Workgroup on Scaling Out, unpublished, 2014 38

39. Scaling Out Perspective on Implementation Research 39 Scale Out Across Diverse Contexts Local knowledge Generalized knowledge Scale Up Explore Adopt/Prepare Implement Sustain Baltimore Houston NE urban NE rural

40. 40 AIR’S mission is to conduct and apply the best behavioral social science research and evaluation towards improving peoples’ lives, with a special emphasis on the disadvantage.

41. 41 Chambers et al, 2013 Implementation Science

42. Prevention Science into…. Johns Hopkins: 1983-present RCT 1: GBG | ML RCT 2: GBG + curricula | family AIR: 2003-2007 RCT 3: GBG + reading + family Develop model of training and support for GBG Baltimore ’83-’07 Case Study: Good Behavior Game at AIR Implementation Science RCT 1: -GBG: 2 models of PD -Implement/Sustain -”Adaptation” Service contract to develop local GBG capacity: Face-to- face training and support Houston ‘08- present Implementation Science Service contract: Face-to-face training and support Under review: GBG for Middle School Nebraska urban ’11- present Implementation Science Development grant: Develop and pilot web-based tools to deliver training to teachers Under review: Develop and pilot web-based tools for coaching and on-going support to teachers Nebraska rural ’13- present JHSPH American Institutes for Research Domitrovich et al., 2008; Poduska et al., 2009; Poduska et al., 2012; Poduska & Kurki, 2014

43. Scaling Out Perspective on Implementation Research 43 Scale Out Across Diverse Contexts Local knowledge Generalized knowledge Scale Up Explore Adopt/Prepare Implement Sustain Baltimore Houston NE urban NE rural

44. Who needs to be at table at AIR? Leadership: CEO/President Board of Directors Dir, Ed Program Legal Contracts Human Resources Information Technology (IT) Financial Institutional Review Board (IRB) Publication Services Communications Web Services ???? ?? Colleagues: Researchers Practitioners Data Use Professional Development Adolescence Ed Tech Early Childhood Middle School Social Emotional Learning Mental Health Services Quantitative Analysts Qualitative Analysts

45. Additional Systems Concepts Systems have developmental trajectories Systems are adaptive, fluid Adaptation occurs locally A note about silos: people may know one another, talk with one another and not be working towards a common goal 45

46. 4. Innovations: Activities of Ce-PIM Juan Villamar, MS.Ed Gracelyn Cruden, M.A. C Hendricks Brown, Ph.D. Center for Prevention Implementation Methodology Department of Psychiatry and Behavioral Sciences Northwestern University 46

47. 47 5. Improve efficiency and effectiveness of implementation delivery Implementation Agency or Agencies Target Developer/ Purveyor Broker, Intermediary 1. Person, Place, and Time, Reach, Engagement 2. Delivery medium 3. Simplify and support intervention delivery 4. Training, fidelity monitoring and feedback 6. Intervention R&D 7. Monitoring Large-Scale Implementation Funder / Oversight 8. Modeling and instituting implementation and oversight policies General Model to Support Implementation of Evidence Based Intervention Brown et al, 2013 JAIDS

48. Using a mobile application to improve fidelity of the GBG – Rapid Development of a Prototype 48

49. Visualization Tool – Classroom Disruptiveness Comparison Model 49 2 classrooms, 24 students Chance that students will act disruptively and their disruptiveness will spread.

50. Major Gap – Sustainable Partnerships Partnerships between content experts and technology developers “What do you recommend we use?” VS “Tell me what you want to see in a report” Bring to the table those players that will interact with the report FIRST Partnerships between Research and Practice Iterative testing with feedback loops Technology System, Educational System, Public Health System Integrated System? 50

51. Understanding the System: Policy Problem: Dissemination of Evidence-Based Interventions Who can pay, sustainably  USPSTF How to influence this powerful entity? Understand how they came to power, to whom they report, and who is required to listen Review their decision making process Concrete challenge: # of prevention programs in parenting that are linked to primary care nearly 0 (despite IOM 2009 report) Is primary care even the best environment? What about schools, who have a larger denominator?

52. Strategic Engagement Identify areas for involvement Short term – engage with AHRQ and USPSTF; identify brokers Long term – Scaling out research designs (primary care) Research design for increased knowledge utilization (ex: harmonized measures) Publications in strategic journals Is there a similar influential agent in Education? How would you begin to engage in the short and long term? 52

53. Innovation: Opening integration between primary care and research AHRQ USPSTF NIH Schools -Nurses -School Wide Programs -Information Systems Primary Care Clinics CMMI grants ODP 53

54. Implementation Science can be conducted with high rigor through: Measurement Modeling Testing 54

55. CAL-OH Head-to-Head Randomized Implementation Trial Single evidence-based intervention: Multidimensional Treatment Foster Care (MTFC) 2 alternative strategies of implementing this same program Standard implementation (Stnd) Community Development Team (CDT) Randomize 51 Counties to implementation strategy* Evaluate implementation success using Stages of Implementation Completion (Saldana IS 2014) Faster implementation more (# families served), better (fidelity) Chamberlain P, et al. (2008). Engaging and Recruiting Counties in an Experiment on Implementing Evidence–Based Practice in California. Administration and Policy in Mental Health and Mental Health Services Research, 35(4): 250-260. 55

56. Existing Implementation Supports for County, Agency, Group Home MTFC Intervention MTFC Implementation Supports for County, Agency, Clinicians, Parent Two-Arm Trials Effectiveness vs. Implementation 56 MTFC Intervention CDT Implementation Supports for County Control Condition Standard Implementation Supports for County Youth

57. Randomize 51 Counties in CA and OH to Implementation Strategy and Time (Cohort) Randomized Roll-Out Design* 57 40 CA Counties 26 Wait LIsted CDT Stnd Wait Listed 13 Wait LIsted COHORT 1 COHORT 2 COHORT 3 COHORT 4 *Brown, et al. 2009 Ann Rev PH 11 OH Counties

58. Summary of Findings (Brown et al., under review) Mixed Results No evidence that CDT affected rate of adoption CDT changed speed of implementation Evidence that CDT increased numbers of families served CDT counties completed implementation more thoroughly 58 Number Served Quantiles for CDT versus IND (EQQ Plot)

59. 5. Building an Integrated Structure to Support Children and Youth Reach their FULL Potential Opportunities, Strengths, Next Steps and Closing 59

60. 1.Implementation Science involves a systematic study of actions within and across systems 2.Requires partnerships that need to built on mutual self interest 3.Systematic way to build such partnerships that coalesce the interests of diverse systems, including public education and public health 4.Implementation lives inside the organizations and communities that are responsible for delivering or overseeing services. 5.There are rigorous methodologic approaches to Implementation Science that address different stages of implementation with well define measures, modeling approaches and research designs. 60

61. Contact Information: C Hendricks Brown, hendricks.brown@northwestern.edu hendricks.brown@northwestern.edu Sheppard G Kellam, skellam@jhu.eduskellam@jhu.edu Jeanne Poduska, jpoduska@air.orgjpoduska@air.org Juan Villamar, juan.villamar@northwestern.edujuan.villamar@northwestern.edu Gracelyn Cruden, gracelyn.cruden@northwestern.edugracelyn.cruden@northwestern.edu Slides and References are Available: http://cepim.northwestern.edu/products/presentatio ns-and-workshops/ 61