Bringing Evidence to Bear: What Facilitates Adoption of Research-based Evidence in Practice and Policy-making in Schools Large Group Panel OSEP Project Director’s Conference 2014 Washington, DC

Session Goals Discuss organizational and other factors that facilitate take-up of new knowledge and facilitate implementation of innovative programs and policies Discuss program features that lends itself to early adoption and uptake of programs in the school environment Adopt a systems perspective in intervention development for improved adoption

Panel Presenters Dr. Vivian Tseng, Vice President, William T. Grant Foundation, New York, NY Dr. David Test, Professor, Department of Special Education and Child Development, Co-Director, National Secondary Transition Technical Assistance Center, UNC Charlotte, NC Dr. Arun Karpur, Research Director, New York State PROMISE, Research Faculty, Cornell University, New York, NY Moderator: Dr. David Guardino, Research to Practice Division, U.S. Department of Education, Office of Special Education Programs

Format of LGP Presentation by individual presenters Q&A session moderated by Dr. Guardino Discussion with the attendees and the speakers

Promoting the Use of Research Evidence Vivian Tseng, Ph.D.

Supporting Research to inform policy and practice to Improve the Lives of Young People

Evidence-based Programs Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities (2006)

Research to Practice Improve Quality of Research Improve Communication, Dissemination, Marketing Demand Use of Research Evidence Incentivize Evidence-based Programs

Research and Practice Improve Research and Its Use Develop Shared Commitments Build Relationships and Trust

Organizations & Systems

Intermediaries

What’s an NSTTAC? National Secondary Transition Technical Assistance Center The Transition to College and Career TA&D Center for secondary transition funded by USDOE, OSEP (1/1/06-12/31/14) Co-Directors: David W. Test (UNC Charlotte) Paula Kohler (Western Michigan University) Project Coordinator: Cather Fowler (UNC Charlotte)

NSTTAC’s Purpose Disseminate information to State Education Agencies, Local Education Agencies, schools, and other stakeholders to improve the: implementation and scaling up of evidence-based practices to develop appropriate measurable postsecondary goals and implement transition services implementation of policies, procedures, and practices that facilitate and increase the participation of students with disabilities in programs and initiatives that are designed to ensure college- and career-readiness achievement of compliance with IDEA’s transition requirements Support efforts to ensure that all students with disabilities are prepared for college (or other postsecondary education and training) and the workforce.

Model for Extending Transition Research Effective Transition Practices Increase Capacity to Implement Effective Transition Practices Facilitate Implementation of Effective Transition Practices Data-Based Decision Making Professional Development Policy Analysis and Change Technical Assistance

Broad Definitions (Helsel, Hitchcock, Miller, Malinow, & Murray, 2006; Twyman, 2008) Are based on rigorous research designs Have demonstrated a record of success for improving student outcomes Have undergone systematic review process using quality indicators to evaluate level of evidence Evidence- Based Practices Are based on rigorous research designs Have demonstrated a record of success for improving student outcomes Research- Based Practices Are based on research Have demonstrated limited success Have used a ‘weak’ research design Promising Practices Are not based on research Have no data to support effectiveness Based on anecdotal evidence and/or professional judgment Unestablished Practices

What We Have Done Reviewed experimental research to identify evidence-based practices in secondary transition Identified 63 evidence-based practices Taxonomy Category Evidence-Based Practices Research to Practice Lesson Plan Starters Student Focused Planning 69 Student Development5698 Family Involvement10 Program Structure99

Student Development Teaching Grocery Shopping Skills Using Computer Assisted Instruction Using Community Based Instruction Using Response Prompting Using a System of Least to Most Prompts Teaching Home Maintenance Skills Using Response Prompting Using Video Modeling Teaching Laundry Tasks Using Response Prompting Teaching Leisure Skills Using Response Prompting Using Constant Time Delay Teaching Safety Skills Using Community Based Instruction Using Progressive Time Delay Using a System of Least to Most Prompts

Based on High-Quality Research, the Field of Secondary Transition now has Evidence-based: Practices Micro Level  Specific interventions Predictors! Macro Level Systems Programs “Generic” practices

In-School Predictors by Post-School Outcome Area Predictors/Outcomes EducationEmploymentIndependent Living Career Awareness XX Occupational Courses XX Paid Employment/Work Experience XXX Vocational Education XX Work Study X Community Experiences X Exit Exam Requirements/High School Diploma Status X Inclusion in General Education XXX Program of Study X

Continuation of In-School Predictors by Post-School Outcome Area Predictors/Outcomes EducationEmploymentIndependent Living Self-Advocacy/Self-Determination XX Self-Care/Independent Living XXX Social Skills XX Interagency Collaboration XX Parent Expectations XX Parental Involvement X Student Support XXX Transition Program XX

Levels of TA Universal/general – Website – Annual Capacity Building Institute Targeted/specialized – State specific topics/issues Intensive/sustained – State & LEA specific topic/issues

Data-Driven Decision Making Tools (Some Samples) Predictor Implementation Self-Assessment Data Use Toolkit (NPSO – Indicator 14) State Toolkit for Examining Post-School Success (STEPSS; Indicators 1, 2,13, & 14) NDPC-SD Data Tools lite (Indicators 1 & 2) NSTTAC Planning Tool (multiple data sources, including from other tools) Short Data Probe (Indicator focused) District Initiative Inventory (qualitative, broad focus)

Assessing Evidence-Based Programs and Practices Need Need in school, district, state Academic & socially significant Issues Parent & community perceptions of need Data indicating need Fit Fit with current Initiatives School, district, state priorities Organizational structures Community values Resource Availability Resources and supports for; Curricula & Classroom Technology supports (IT dept.) Staffing Training Data Systems Coaching & Supervision Administration & system Evidence Outcomes – Is it worth it? Fidelity data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or effectiveness Intervention Readiness for Replication Readiness Qualified purveyor Expert or TA available Mature sites to observe Several replications How well is it operationalized? Are Imp Drivers operationalized? Capacity to Implement Capacity Staff meet minimum qualifications Able to sustain Imp Drivers Financially Structurally Buy-in process operationalized Practitioners Families Agency EBP: 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4. HighMediumLow Need Fit Resources Availability Evidence Readiness for Replication Capacity to Implement Total Score: © National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland

Organizational & Program Factors Commitment to Adopting EBPs Readiness Flexibility Communication Time

Utilizing Continuous Quality Improvement for Promoting Adoption and Uptake of Knowledge NYS PROMISE Approach Arun Karpur, MD, MPH Research Director NYS PROMISE Employment and Disability Institute, Cornell University

NYS Promise NYS PROMISE will utilize an indigenous model that naturally equips and engages schools, local disability service providers, independent living centers, one stop centers, literacy zones, regional parent training centers, work incentive planners, regional transition specialists, and other community transition stakeholders to achieve higher postsecondary employment, education and economic outcomes for SSI youth by...

NYS PROMISE Intervention Increased Access to State and National TA on Transition Best Practices Services under outcomes-based payment model Improved access to key high quality services leading to intended outcomes PROMISE-sponsored case management & service coordination Improved collaboration between LEAs and local service providers CQI Feedback Loops : Project MIS, Fidelity Assessment, and TA PROMISE-sponsored parent training, information & family coaching Improved self- determination, expectations, engagement, attitudes toward work and financial literacy Higher post-secondary employment, education and economic outcomes for youth on SSI Improved quality of transition to adulthood programs in LEAs for youth in treatment group

Continuous Quality Improvement

Quality Planning Tools & Processes Outreach and Recruitment Guide PROMISE Recruitment Tracking System Intervention Guide for Practitioners and Evaluators Training for using project MIS Training for outcomes-based payment Organizational capacity assessment for implementation Brokering relationships between project partners Formation of regional teams of stakeholders NYS PROMISE State Steering Committee and Coordinating Council

Quality Control Tools & Processes Project MIS tool [NYESS]– tracking program participation and services received Online training and technical assistance tracking (OTAT) system Program fidelity assessment – qualitative and quantitative indicators Youth and Parent Voices – survey and qualitative study Stakeholder Voices – collaboration survey, bi- annual learning communities, and steering committee meetings

Quality Improvement Tools and Processes Quarterly automated NYESS reports for RDS, Providers and Parent Centers TA translating data into action plans at local, regional and state-level Bi-annual comprehensive project reports at Learning Community identifying (a) what was done well, (b)what worked and (c) what needs to be done Sharing data with steering committee at state-level Engaging stakeholders in concept mapping activity to synthesize information Annual progress report

Access to National and State TA Access to national and state TA on transition through partnerships with NYS PDSC, NSTTAC training events, and other research and TA entities Annual DCDT conference attendance Webinars on lessons learned from NYS PROMISE implementation Reports from the National Evaluator on NYS PROMISE