1. Understanding the Move Toward Evidence-Based Programs: Considerations for Early Childhood Leaders Michelle Hughes, MA, MSW Project Director, Benchmarks September 28, 2011

2. Today’s Presentation Why evidence-based programs? – Why all the fuss about EBPs? – Changing Landscape of Practice – What does evidence-based actually mean? Getting to outcomes: More than choosing an evidence-based program off a list!! – NIRN Framework for Assessing EBPs – Implementation, Implementation, Implementation (Webinar #2) Resources and Questions

3. Evidence-based programs…. What’s all the fuss?

4. Some Contributing Factors Increased focus on accountability in multiple sectors. Increased knowledge about affecting change. – More information about “what works” and what doesn’t – Increased focus on measuring community-level impact (are we moving the “community needle”) Stewardship of public/philanthropic dollars: limited resources need to be used strategically.

5. The Changing Landscape Focus on EBPs seen across all fields, in public and private sectors, at all levels of government This is a learning process – changing systems and practice is a long-term (and challenging) endeavor.

6. At the National Level Increased focus/guidance/policy change with regard to evidence in federal agencies (SAMSHA, CBCAP, etc.) Nonprofits and Academic Institutions: Coalition for Evidence-Based Policy, Child Trends, Center for Early Literacy Learning, National Implementation Network “What Works” Sites: Promising Practices Network, California Clearinghouse (child maltreatment); Blueprints for Violence Prevention; National Registry of Effective Evidence-Based Programs and Practices (substance abuse, mental health)…

7. And in North Carolina… Most public agencies and many private philanthropic agencies are increasingly focused on supporting EBPs – NC Division of Social Services, CBCAP and IV-B, part 2 funding for prevention and family support; – NC Division of Public Health, Evidence-Based Home Visiting Initiative – The Duke Endowment, Kate B. Reynolds Charitable Trust – Alliance for Evidence-Based Family Strengthening Programs launched Nurse-Family Partnership, Incredible Years, and Strengthening Families Program

8. In Smart Start… NCPC has been encouraging local partnerships to move toward the use of ebp’s for several years. In 2011 NC General Assembly passed legislation. “State funding for local partnerships shall also be used for evidence-based or evidence-informed programs for children from birth to five years of age that: – Increase child’s literacy – Increase the parents’ ability to raise healthy, successful children – Improve children’s health – Assist four and five star rated facilities in improving and maintaining quality”

9. Smart Start Definitions NCPC Board of Directors adopted definitions: Evidence-based programs or practices are those that have repeatedly and consistently demonstrated desirable outcomes through application of scientific research methods (replicated experimental, experimental, or quasi experimental) Evidence-informed practice is one that is guided by child development theory, practitioner wisdom, qualitative studies, and findings from basic research and has written guidelines, a strong logic model, and a history of demonstrating positive results. They may be rated “Promising” or “Emerging” by at least one source that rates evidence-based programs.

10. Why the focus on “rigorous” scientific research methods??? I don’t get it…if I’m getting good pre-test, post-test results, doesn’t that prove what I’m doing is working? It is a great first step, but we know from research that some programs that show good preliminary results may not be as effective as we think when we do more rigorous evaluation.

11. How Do You Know If Something Works? A Quick Review of Evaluation Methods…bear with me for a few minutes!! A program or practice “works” if it produces the intended positive results (outcomes). We can determine if something works through evaluation methods…but some methods (like experimental studies) are better at determining “cause-effect” than others. Let’s talk about why….

12. Levels of Evidence Continuum of evaluation methods: Non- experimental, quasi-experimental, experimental. Non-experimental: anecdotes, qualitative interviews, case studies, surveys, pre-post assessments without comparison/control groups. These are IMPORTANT methods that contribute to our knowledge base about program effectiveness but they are insufficient for determining if a program is effective. Why???

13. Here’s why…. A non-experimental design cannot determine if the positive effects seen in a program are the result of the program, or some other cause. – Might occur anyways without the intervention; – Might occur because of some other characteristics about the group participating in the intervention; – Might occur because of chance. How can we determine cause and effect with confidence? – We need a group to compare with (i.e., a “control” group). – Helps you determine if outcomes are caused by other factors besides the intervention

14. Experimental Research Methods Quasi-experimental: Use a non-randomized comparison group in which intervention and comparison group are closely matched (e.g., demographics, SES, etc.). Differences in outcomes between groups can be attributed to the intervention with some degree of confidence, depending on strength of study design. Experimental: Considered the “gold standard” in determining cause and effect. Randomly assign a target population to an experimental or a control group. Differences in outcomes can be attributed to the intervention with high degree of confidence if strong study design.

15. FRIENDS Diagram of Evidence Cited from FRIENDS Website, http://friendsnrc.org/cbcap-priority-areas/evidence- base-practice-in-cbcap Used with child abuse prevention programs funded through Federal Community- Based Child Abuse Prevention Program NC Division of Social Services uses for prevention funding

16. Some Examples of Why RCT Important Medicine: Hormone replacement therapy for post- menopausal women (increases risk of stroke and heart disease for many women) Dietary fiber to prevent colon cancer (shown ineffective) Stents to open clogged arteries (shown no better than drugs for most heart patients). Cited from Jon Baron, NC Family Impact Seminar, 2009.

17. More Examples… Social Welfare: Job Training Partnership Act – Pre-post scores of the men who went through the job training program indicated an increase in earnings (a positive finding), but when program participants were compared to a control group who did not go through the training, program participants demonstrated smaller increase in earnings than control group. : Prevention : DARE – ENORMOUS support for DARE. At one point, DARE taught in 80% of the school districts in the United States, in 54 other countries around the world, and was taught to 36,000,000 students each year (Hanson, D J., cited from www.alcoholfacts.org) – Pre-post assessments indicate increased student knowledge of risks, and improved attitude changes toward using alcohol & drug use, however, RCTs revealed program has no effect in reducing alcohol/drug use. Cited from Jon Baron, NC Family Impact Seminar, 2009.

18. Assessing Research on Programs Assessing the research on different programs can be overwhelming. Who has time to do literature reviews? Effect size; sample size; attrition; validated measures….OH MY!!

19. Some Steps to Take…. Use existing resources and registries (resources at end of presentation) who have already done this work Use NCPC state office staff and other resources (Sarah Currier/Emmy Marshall at Prevent Child Abuse NC; Catherine Joyner at DPH; Center for Child and Family Health; Duke Center for Child and Family Policy) Educate yourself to become a smart consumer of research

20. Getting to Outcomes… So, once I find a program that is evidence- based or evidence-informed I am ready to go right?? Let’s take a step back and use a framework developed by the National Implementation Research Network to talk about assessing EBPs for your community. Not exactly…but you are headed in the right direction!

21. EBP: 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4. HighMediumLow Need Fit Resources Availability Evidence Readiness for Replication Capacity to Implement Total Score: Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State and Local Priorities Organizational structures Community Values Resource Availability Resource Availability IT Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Evidence-Based Programs and Practices Intervention Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland

22. Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State and Local Priorities Organizational structures Community Values Resource Availability Resource Availability IT Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Evidence-Based Programs and Practices Intervention Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence

23. Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State and Local Priorities Organizational structures Community Values Resource Availability Resource Availability IT Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Evidence-Based Programs and Practices Intervention Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland Need Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need

24. Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State and Local Priorities Organizational structures Community Values Resource Availability Resource Availability IT Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Evidence-Based Programs and Practices Intervention Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland Evidence What is the need in my community? What are we trying to change? What program might address those needs and help us affect those changes (outcomes)? What is the evidence that this program will indeed meet those needs and result in those desired outcomes? Need

25. Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State and Local Priorities Organizational structures Community Values Resource Availability Resource Availability IT Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Evidence-Based Programs and Practices Intervention Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland Is that program a good “fit” for my agency and community? Does it fit with: Current initiatives? State and local priorities? Organizational mission and values? Organizational structure? Community values? Evidence Need Fit

26. Now Let’s Stop for a Moment…. Implementation, Implementation, Implementation….and…. IMPLEMENTATION!!!

27. Getting Positive Outcomes Means.. Recognize that “evidence” is necessary but insufficient for achieving positive outcomes for children and families. We must focus on supporting community-based agencies and practitioners in….. Effective Implementation

28. Implementation MATTERS! Successfully replicating evidence-based programs requires…. Proven practice + fidelity/quality implementation = Better Outcomes Fidelity: adherence to core elements which contribute to program effectiveness A poorly implemented program will yield poor outcomes (and results in a poor investment)

29. Need in Agency, Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit Fit with current - Initiatives State and Local Priorities Organizational structures Community Values Resource Availability Resource Availability IT Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Assessing Evidence-Based Programs and Practices Intervention Readiness for Replication Qualified purveyor Expert or TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Intervention Readiness for Replication Capacity to Implement Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency Capacity to Implement © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland Evidence Need Fit Is the program we’ve chosen ready for replication? What does the program we’ve chosen require for successful implementation? What capacity do we have to implement?

30. Stay tuned…. Next Webinar on October 12 th at 10:00 am. Getting Results with Evidence-Based Programs: The Importance of Effective Implementation. – Focused the science of implementation – what do we know about effective implementation and how to support it? – What are the key “drivers” for successfully implementing a program or practice? – How can these key “drivers” not only help us implement a specific EBP – but build our agencies’ capacity to offer an array of services to support children and families in our communities?

31. In Summary… The trend toward evidence-based programs is here to stay. You are not alone in trying to figure this out! Moving toward EBPs is a process of capacity building for everyone! (staff, agency leaders, Boards of Directors, your community partners, NCPC state office) – Becoming a good consumer of research – Using state and national resources – Building organizational capacity to implement EBPs (which strengthens the entire agency!)

32. Summary continued… Getting good outcomes is NOT about choosing a program off a list. It IS about building the capacity of your organization and your partnerships with the community to implement effectively. There are a lot of resources to help you with this work.

33. Resources FRIENDS National Resource Center for Community-Based Child Abuse Prevention. http://www.friendsnrc.org/ Provides information, online training, toolkits on evidence-based programs, evaluation, logic models and continuous quality improvement. Resources would be useful for any agency focused on family support programs and practices. Promising Practices Network www.promisingpractices.net Provides evidence-based information about what works to improve the lives of children, youth, and families. Features summaries of programs and practices proven to improve outcomes for children. National Implementation Research Network (NIRN) http://www.fpg.unc.edu/~nirn/ Provides extensive research and information on the science of implementation, including several frameworks for understanding effective implementation and examples from the field. Several resources listed cited from 2009 Duke University Family Impact Seminar

34. Resources Social Programs that Work www.evidencebasedprograms.org/ Summarizes the results of well-designed randomized controlled trials in social policy that have particularly important policy implications. Organized into the following substantive areas: early childhood, K-12 education, youth development, crime/violence prevention, substance abuse prevention and treatment, mental health, employment and welfare, and international development. National Registry of Evidenced-based Programs and Practices, SAMHSA nrepp.samhsa.gov Offers a searchable database of interventions for the prevention and treatment of mental and substance use disorders. Meant to help people, agencies, and organizations implement effective programs and practices in their communities. Blueprints for Violence Prevention, Center for the Study and Prevention of Violence www.colorado.edu/cspv/blueprints Identifies outstanding violence and drug prevention programs that meet a high scientific standard of effectiveness. Serves as a resource for governments, foundations, businesses, and other organizations trying to make informed judgments about their investments in violence and drug prevention programs.

35. RESOURCES North Carolina Family Impact Seminar, Duke University http://familyimpactseminars.org/index.asp?p=1&page=seminar&se minarid=181&siteid=34 This family impact seminar held in 2009 at Duke University provides several powerpoints presentations on evidence-based policy and programs nationally and in North Carolina, and a publication with good overview of evidence-based policy and program development. Washington State Institute for Public Policy, Olympia, WA www.wsipp.wa.gov/topic.asp?cat=19&subcat=0&dteSlct=0 Summarizes evidence-based programs and cost-benefit analyses for interventions in the following areas: developmental disabilities, child welfare/child maltreatment prevention, children’s mental health, criminal justice, juvenile justice, and K-12 education.

36. NCPC Resources Program Staff – Ann Ward, 919-821-9556, award@ncsmartstart.orgaward@ncsmartstart.org – Cynthia Turner, 919-821-9565, cturner@ncsmartstart.orgcturner@ncsmartstart.org – Gale Wilson, 919-821-9563, gwilson@ncsmartstart.orggwilson@ncsmartstart.org – Lois Slade, 919-821-9577, lslade@ncsmartstart.orglslade@ncsmartstart.org Evaluation Staff – Leigh Poole, 919-821-9580, lpoole@ncsmartstart.orglpoole@ncsmartstart.org – Meshell Reynolds, Counts & reporting, 919-821-9567, mreynolds@ncsmartstart.org mreynolds@ncsmartstart.org Smart Net Resources – Program & Evaluation tab> folders for Evidence-Based/Informed programs, & for Family Support, Health, & Early Care & Education

37. Presenter Contact Information Michelle Hughes, MA, MSW Project Director, Benchmarks mhughes@benchmarksnc.org (919) 357-7361 www.benchmarksnc.org