1. Conceptual framework and a research agenda in the transition from efficacy to effectiveness Irwin Sandler ASU Prevention Research Center P30 MH068685-01

2. Overview What we learn from efficacy – New Beginnings Program Effectiveness conceptualized Expanded conceptualization of effectiveness Research agenda on transition from efficacy to effectiveness: NBP Research studies to bridge from efficacy to effectiveness –Marketing study –Court factors study

3. Efficacy: New Beginnings Program Efficacy trial tests whether a program does more good than harm when delivered under optimal conditions (Flay, 1986) New Beginnings Program Efficacy Trial (NBP)  Selection process Extensive recruitment methods Participants agreed to accept randomization and assessment Residential mothers meeting 9 eligibility criteria  Implementation process 11 session manualized (scripted) program Delivered at ASU Prevention Center Implemented by 13 selected group leaders Training (30 hours, plus 1.5 hours prior to each session) Supervision (videotaped sessions, 1 hour supervision per wk)

4. Selection and Implementation Selection 36% of eligible families enrolled in trial Higher SES; 88% non Hispanic white Low drop out rate from intervention (11%) Implementation Very high (90% +) level of program implementation Very low variability in implementation

5. Randomized Efficacy Trial: Reduce Multiple Problem Outcomes Six-Years Later Effects on outcomes over 6 years: –36% reduction in 1-year prevalence of diagnosed mental disorder on C-DISC –Reduce total psychiatric symptoms –Reduce Marijuana, Alcohol and Drug Use for higher risk group –Improve Grade Point Average –Reduce number of sexual partners Effects are mediated by improved parenting Effects strongest for high risk group Demonstrates viability of concept but not delivery in natural settings

6. Effectiveness Trial (Flay, 1986) “…concerned with whether program does more harm than good when delivered via a real world program”. Effects of trial may be due to: –Selection (acceptance): rate of acceptance; who accepts (program x person interaction) –Implementation - fidelity, adaptation Both acceptance and implementation are multi- level concepts that apply to the program and the organization that delivers the program

7. Transition from Efficacy to Effectiveness Research agenda for making the transition involves developing systems to study real world conditions in which it will be delivered and design program to optimize successful implementation Prevention programs often don’t have pre-existing delivery systems – but host institutions with which they share an interest and through which they are delivered Identify Domestic Relations Court as the Organizational Context –Access to the population –History of innovative services –Overlapping (though not identical) interests with prevention

8. Research Agenda Do the courts want this collaboration? Current practices and factors influencing adoption Identify factors influencing selection and implementation of NBP using quality management concepts and methods – research with consumer Court factors Cultural factors Provider agency factors Design program and system for robust delivery through courts Optimizing court, culture, provider factors that influence implementation Optimize participant selection processes Pilot test selection and redesigned program and receive feedback on viability Effectiveness trial to test whether the program have the desired effects Design of multi-court effectiveness trial

9. Market Study: National Survey of Courts (Cookston, Braver, Sandler & Genalo, 2002) Question: What is the current practice and readiness of county courts to adopt “evidence- based” parenting program in the court? Background: o1556 Counties provide divorce parenting services in 1998 o11 states require divorcing families to participate in parenting programs oPrograms are generally short (4 hours) and evaluations of efficacy are limited

10. Method for Survey of Courts Stratified random sample drawn of 154 courts with parenting program Interview with key informant who is most knowledgeable about the program Questions addressed: Description of current programs offered Planned changes in programs Support for parenting program like NBP Barriers to implementation

11. Support and Opposition for Lengthier Parent Education Programs Influence GroupSupport*Oppose^ Judges77% (N = 115)11% ( N = 16) Members of the bar73% (N = 111)14% (N = 21) County supervisors71% (N = 102)17% (N = 25) State legislators63% (N = 89)22% (N = 31) Child advocacy groups 96% (N = 148)3% (N = 4) Local child and family agencies 97% (N = 147)1% (N = 1) State Supreme Court87% (N = 111)7% (N = 9) * - percent of respondents who indicated the influence group would support lengthier parent education programs ^ - percent of respondents who indicated the influence group would not support lengthier parent education programs and would, in fact, oppose such programs

12. Barriers To The Implementation Of Lengthier Divorcing Parent Education Program

13. Court Factors Affecting Adoption and Implementation National Advisory Group of Court Leaders – judges, court administrators, mediators, court service providers, executive director of national professional organization (AFCC) Identified by core working group of program advocates – program developers, director of mediation services

14. National Advisory Board Role Provide advice and guidance to “translate the NBP ‘experimental prototype’ into a service that fits as a service for divorcing families through the Family Court.” Three stage process a)Understand issues in translation into a real- world service b)Advise on specific aims of effectiveness trial and assessment of court readiness c) Advise on protocol for court recruitment

15. Advisory Board Input on Court Level Selection and Implementation Issues (2/03) Respond to program selection scenario “You have just received an RFP from your state government for a parenting program to be delivered through your court. Two proposals for parenting education have been submitted and you must choose between them. What factors are likely to make you select one program over another?”

16. Factors that Influence Program Selection Members independently listed 65 factors Group discussion of factors – reduced to 24 factors Factors grouped into six categories by research team Categories rank ordered by the members

17. Rank Order of Program Selection Factors 1.Compatibility and resource investment (compatible with court priorities, cost of implementation) 2.Program efficacy (broad support from stakeholders and meet needs of parents) 3.Perceived program credibility (e.g., Reputation from other courts) 4.Program structure and content (e.g., number of sessions; program content) 5.Program delivery quality (e.g., mode of presentation; accompanying material) 6.Program accessibility (avg. rank = 4.7) (e.g., customer access; multicultura/multilingual)

18. Factors that Influence Quality of Implementation Respond to quality of implementation scenario. “When you receive your state funding award one of the stipulations for continuation of future funding was assuring high quality of implementation of this parenting program. What factors are most likely to influence quality of implementation in your court?”

19. Rank Order of Quality of Implementation Factors Staffing (on-site coordinator; administrator – audits, details of administration) Stakeholder/consumer feedback (ongoing; evaluation of impact on consumer and the court) Training (Training and motivating providers) Expanded reach of program (expand range of providers and participants) Program visibility and acceptance (Advisory committee; publicity) Program facilities (appropriate space)

20. Summary Research for transition to effectiveness trial involves study of court, provider, cultural, selection factors and design system to optimize selection and implementation Braver will present studies to evaluate alternative systems to recruit participants MacKinnon will discuss issues in the effectiveness trial – focusing on role of mediational analysis

21. Service Providers New Beginnings Program Domestic Relations Court Families System for Implementation of New Beginnings Program Through Domestic Relations Courts Families Training, technical assistance and monitoring Selection Methods Stakeholder input Families