Researchers Without Borders Webinar 4 A Framework and Suite of Instruments for Examining Fidelity of Implementation Jeanne Century Center for Elementary.

Slides:

Advertisements

Similar presentations

Progress Towards Reading Success: The Reading First Evaluation Prepared by: Amy Kemp, Ph.D. Research Associate and Patricia A. Muller, Ph.D. Associate.

Advertisements

Program Evaluation in Gifted Education Rebecca Mann EDPS 540.

Continuous Improvement in the Classroom -Professional Learning Communities.

Agenda - January 28, 2009 Professional Learning Community – Jefferson HS Learning by Doing What does the data tell us? ITED results SIP Goals Data Questions.

Professional Learning Communities Connecting the Initiatives

The Importance of Treatment Fidelity in Delivering Parent Programmes Presentation for the Incredible Years Wales Conference Llandrindod Wells, 7th February.

Center for Curriculum Materials in Science AAAS, Michigan State University, Northwestern University, University of Michigan KSI 2006 Strand 2: Teacher.

Structuring Retreats to Share Findings and Discuss Recommendations Paul Cobb and the MIST Team.

© 2014, Florida Department of Education. All Rights Reserved. Defining STEM Elements of STEM Schools Dr. JP Keener.

Evaluation is a professional and ethical responsibility and is a core part of PHN professional practice Commitment to evaluation helps build the PHN intelligence.

Managing and Leading Elementary Schools: Attending to the Formal and Informal Organization James P. Spillane Bijou Hunt Kaleen Healy Northwestern University.

Developing and Supporting Highly Effective Teachers in Every Classroom Leaders of Learning Implementation Norman Public Schools Date.

Iowa Core Alignment of Instructional Content to the Iowa Core Sue Updegraff Keystone AEA.

National Center on Response to Intervention Developed by the National Center on Response to Intervention and RMC Research RTI Integrity Framework: A Tool.

Gifted Program Review Spring Process  In February 2013 a team of 41 individuals met to develop questions: parent, teachers, psychologists and administrators.

Title I Needs Assessment/ Program Evaluation Title I Technical Assistance & Networking Session October 5, 2010.

Identifying critical components To answer “What is the status of implementation of each STEM school’s model and to what extent does the model reflect the.

Coming June 30,  Purpose of PBIS Assessment  Implications of the move from PBIS Surveys  Overview of available Tools and Surveys  Criteria for.

Evaluation 101 Everything You Need to Know to Get Started Evaluating Informal Science Education Media Everything You Need to Know to Get Started Evaluating.

Developing an Effective Evaluation to Check for Understanding Susan E. Schultz, Ph.D. Evaluation Consultant PARK Teachers.

DEVELOPING ALGEBRA-READY STUDENTS FOR MIDDLE SCHOOL: EXPLORING THE IMPACT OF EARLY ALGEBRA PRINCIPAL INVESTIGATORS:Maria L. Blanton, University of Massachusetts.

1. 2 Why is the Core important? To set high expectations –for all students –for educators To attend to the learning needs of students To break through.

Designing Survey Instrument to Evaluate Implementation of Complex Health Interventions: Lessons Learned Eunice Chong Adrienne Alayli-Goebbels Lori Webel-Edgar.

Kirby School District 140 Everyday Mathematics (EM) and Science Program Research and Evaluation of Mathematics and Science Implementation.

Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington,

Improving Implementation Research Methods for Behavioral and Social Science Working Meeting Measuring Enactment of Innovations and the Factors that Affect.

DR-K12 PI Meeting Building the Knowledge Base of Teacher Learning in STEM Education A Framework and Suite of Adaptable Instruments for Examining Fidelity.

Digging Deeper LKSD Principal Inservice Feb. Our approach for our time together…

1 Introduction to Content Alignment. 2 Objectives 1.Summarize the relationship between the Iowa Core Curriculum and the Iowa Core Content Standards and.

1 Leadership Module 3: Introduction to Content Alignment.

Inspire the learner; ignite the potential! Welcome! Forest Lake Area School District Staff District In-Service Tuesday, August 26, 2003.

Setting the Stage: Workshop Framing and Crosscutting Issues Simon Hearn, ODI Evaluation Methods for Large-Scale, Complex, Multi- National Global Health.

Scientific Validation Of A Set Of Instruments Measuring Fidelity Of Implementation (FOI) Of Reform-based Science And Mathematics Instructional Materials.

NCATE Standard 3: Field Experiences & Clinical Practice Monica Y. Minor, NCATE Jeri A. Carroll, BOE Chair Professor, Wichita State University.

1. Housekeeping Items June 8 th and 9 th put on calendar for 2 nd round of Iowa Core ***Shenandoah participants*** Module 6 training on March 24 th will.

Laying the Foundation for Scaling Up During Development.

IMPLEMENTATION QUALITY RESEARCH OF PREVENTION PROGRAMS IN CROATIA MIRANDA NOVAK University of Zagreb, Faculty of Education and Rehabilitation Sciences.

Practical Definition of Terms Output Targets What do we want to achieve? Deliverables What do we get? Output Targets – To improve the public speaking skills.

 Welcome, introductions  Conceptualizing the evaluation problem  stakeholder interests  Divergent and convergent processes  Developing evaluation.

1. Administrators will gain a deeper understanding of the connection between arts, engagement, student success, and college and career readiness. 2. Administrators.

Ohio Department of Education March 2011 Ohio Educator Evaluation Systems.

Methods and Tools for Measuring Fidelity Greg Roberts, PhD. Vaughn Gross Center & National Center for Instruction The University of Texas at Austin.

MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings.

The Ohio STEM Learning Network: A Study of Factors Affecting Implementation Spread and Sustainability.

Implementing & Evaluating a School RTI Framework Shawna Moran, EII, ODE Odyssey - August 2013.

AIM: K–8 Science Iris Weiss Eric Banilower Horizon Research, Inc.

Part 2: Assisting Students Struggling with Reading: Multi-Tier System of Supports H325A

Quantitative Data Collection Techniques In contrast to qualitative data quantitative data are numerical. Counted, calculated tallied and rated. Also ratings.

A Multi-Level Framework to Understand Factors Influencing Program Implementation in Schools Celene E. Domitrovich, Ph.D. Penn State Prevention Research.

Combining Clinical Supervision with Teacher Evaluation

National Center on Response to Intervention RTI Essential Component: Schoolwide, Multi-Level Prevention System Katie Klingler Tackett National Center on.

The Cause…or the “What” of What Works? David S. Cordray Vanderbilt University IES Research Conference Washington, DC June 16, 2006.

Jeanne Century CEMSE, University of Chicago NRC Committee on Integrated STEM January 10, 2012 Defining iSTEM.

The Why and How of Using Case Studies to Evaluate Educational Programs Eric Barela Los Angeles Unified School District Sheryl Gowen Georgia State University.

Social Emotional Learning through an ABA Lens Annette Little, PhD, BCBA-D Lipscomb University.

COM 535, S08 Designing and Conducting Formative Evaluations April 7, 2008.

1 Innovative Teaching and Learning (ITL) Research Corinne Singleton SRI International.

ACS WASC/CDE Visiting Committee Final Presentation Panorama High School March

Documenting quality in complex residential child care systems: Challenges with assessing fidelity to a principle-based program model Lisa McCabe and Michael.

Instructional Practice Guide: Coaching Tool Making the Shifts in Classroom Instruction Ignite 2015 San Diego, CA February 20, 2015 Sandra

Preparing Facilitators to Use and Adapt Video-Based Professional Development Materials with Fidelity Nanette Seago WestEd, Redwood City, CA Karen Koellner.

Classroom teaching observation tools

Partnership for Practice

NC State Improvement Project

Melanie Taylor Horizon Research, Inc.

Educator Evaluations DARTEP 2017

Study Questions To what extent do English language learners have opportunity to learn the subject content specified in state academic standards and.

K–8 Session 1: Exploring the Critical Areas

Fidelity of Implementation in Scaling-up Highly Rated Science Curriculum Units for Diverse Populations Carol O’Donnell and Sharon Lynch The George Washington.

Social Validity and Treatment Integrity

Presentation transcript:

Researchers Without Borders Webinar 4 A Framework and Suite of Instruments for Examining Fidelity of Implementation Jeanne Century Center for Elementary Mathematics and Science Education University of Chicago July 7, 2010 This work has been supported in part by the National Science Foundation Project ESI

Why Measure Fidelity of Implementation (FOI)? Determine if programs are effective Determine which parts of programs are effective Determine why and under what conditions parts of programs are effective

What is actually happening?

NSF Project Challenges Instruments that could be used across multiple programs A suite of instruments that could be used alone or in combination depending on resources and goals Provide detailed and specific information about FOI

Five Dimensions Approach Dane and Schneider (1998)  39/162 studies measured integrity  In the 39 studies, the authors defined “integrity” along one or more of five dimensions  They are: 1) adherence; 2) exposure; 3) quality of delivery; 4) participant responsiveness and 5) program differentiation Critical Components Approach Hall and Hord (1987), Bond et al. 2000; Huntley, 2005; Sabelli & Dede (2001) and others Some called them model dimensions, fidelity criteria, critical parts Structure and Process Approach Mowbray et al. (1987), Wang et al. (1984) Structure (composition of the intervention) Process (human interactions that take place during delivery) FOI Approaches…

Combining Frameworks Approach Ruiz Primo (2005) – matrix of 5 dimensions and structure/process Lynch and O’Donnell (2005); Lastica and O’Donnell (2007) – matrix of 5 dimensions and structure/process CEMSE Approach combine critical component approach and structure process approach show relationships to 5 dimensions approach use combination of approaches to create categories of critical components and a framework that can apply to multiple interventions

Questions?

FOI The extent to which an enacted program is consistent with the intended program model Critical Components The elements of a program model that are essential to its implementation FOI Operationalized The extent to which the critical components of a program are present when the program is enacted Definitions

Written Materials Developers Critical Component Identification theory… Users Challenges Theories aren’t always explicit Critical components very in “grain size” The “it” can vary, depending on how the intervention is framed Critical Component Identification evidence…

FOI Framework

FOI Framework and Critical Components

Questions?

Categories of Differentiation Mathematics And Science Science Only Mathematics Only Program Specific FOI Framework and Differentiation

C ategories of Differentiation (differentiation) Common Across Interventions Unique to Interventions CEMSE Framework Aligned with (Others’ Approaches)

Questions?

Instruments Teacher Questionnaire Teacher Instructional Observation Protocol Teacher Interview Protocol Teacher Instructional Log Teacher Attitude Questionnaire School Leader Questionnaire School Leader Interview Protocol School-wide Observation Protocol

Structural Procedural Teacher Questionnaire XXXXXXXXXXXXXXXX School Leader Questionnaire XX Teacher Interview Protocol XXXXXXX School Leader Interview Protocol XX Classroom Observation Protocol XXXXXXXXXXXX School Wide Observation Protocol XXXXXX Teacher LogXXXXXXXXXXXX First Teacher LogXXXXX Teacher Attitude Questionnaire X Sample Instrument Construct Matrix

Categories of Differentiation Common Program A Program B Program C Impact of a single program Compare two or more programs Compare programs to “business as usual” Compare two versions of a program Uses of the Framework and Measures Developers U.S. Satellite, Inc. – additional elements Evaluators School–based change model – Lindblom High School Programmatic model – E2SP Researchers Trailblazers – look at a critical component in more depth

Programs Science and Technology for Children FOSS SEPUP IES Everyday Mathematics Science Companion

Jeanne Century Center for Elementary Mathematics and Science Education University of Chicago FOI Supplement Work Researchers Without Borders Resources FAQs; Research; Tools and Instruments Learning Webinars; Collaboration Community Development This work has been supported in part by the National Science Foundation Project ESI