Improving Implementation Research Methods for Behavioral and Social Science Working Meeting Measuring Enactment of Innovations and the Factors that Affect.

Slides:



Advertisements
Similar presentations
Definitions Innovation Reform Improvement Change.
Advertisements

Scaling-Up Early Childhood Intervention Literacy Learning Practices Maurice McInerney, Ph.D. American Institutes for Research Presentation prepared for.
Measuring Progress in Accountable Care Webinar March 14, 2013 Speakers: Eugene A. Kroch, Ph.D., vice president and chief scientist at the Premier Research.
Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.
What is Diffusion? The process of communicating innovation through certain channels over time through members of a social system.
Strategies for Improving Global Environmental Governance: Challenging the Status Quo Strategies for Improving Global Environmental Governance: Challenging.
Building the Foundations for Better Health Health Services Organization.
Questions from a patient or carer perspective
Community Level Models; Participatory Research and Challenges
Strategy Implementation
Identifying critical components To answer “What is the status of implementation of each STEM school’s model and to what extent does the model reflect the.
Pacific Southwest (Region 9) Model Training and Technology Transfer Activities California and Beyond Thomas E. Freese, PhD Pacific Southwest Addiction.
From Evidence to Action: Addressing Challenges to Knowledge Translation in RHAs The Need to Know Team Meeting May 30, 2005.
Striving for Quality Using continuous improvement strategies to increase program quality, implementation fidelity and durability Steve Goodman Director.
Designing Survey Instrument to Evaluate Implementation of Complex Health Interventions: Lessons Learned Eunice Chong Adrienne Alayli-Goebbels Lori Webel-Edgar.
1 Robert S. Webb and Roger S. Pulwarty NOAA Climate Service.
Kirby School District 140 Everyday Mathematics (EM) and Science Program Research and Evaluation of Mathematics and Science Implementation.
Organizational Change
Diffusion of Innovations Gerontology 820 Ashley Waldoch October 18, 2010.
Canadian Cancer Society Manitoba Division: Knowledge Exchange Network (KEN) & CancerCare Manitoba Manitoba Integrated Chronic Disease Primary Prevention.
Transboundary Conservation Governance: Key Principles & Concepts Governance of Transboundary Conservation Areas WPC, Sydney, 17 November 2014 Matthew McKinney.
Implementation Science Vision 21: Linking Systems of Care June 2015 Lyman Legters.
Program Fidelity Influencing Training Program Functioning and Effectiveness Cheryl J. Woods, CSW.
DR-K12 PI Meeting Building the Knowledge Base of Teacher Learning in STEM Education A Framework and Suite of Adaptable Instruments for Examining Fidelity.
PowerPoint Presentation by Charlie Cook
Scientific Validation Of A Set Of Instruments Measuring Fidelity Of Implementation (FOI) Of Reform-based Science And Mathematics Instructional Materials.
Boston Geneva San Francisco Seattle Beginning the Evaluation Journey with FSG KCIC Boot Camp March 24, 2010 Prepared for:
Integrating Knowledge Translation and Exchange into a grant Maureen Dobbins, RN, PhD SON, January 14, 2013.
Maria E. Fernandez, Ph.D. Associate Professor Health Promotion and Behavioral Sciences University of Texas, School of Public Health.
Laying the Foundation for Scaling Up During Development.
Third Sector Evaluation: Challenges and Opportunities Presentation to the Public Legal Education in Canada National Conference on “Making an Impact” 26.
Leadership Chapter 7 – Path-Goal Theory.  Path-Goal Theory Perspective  Conditions of Leadership Motivation  Leader Behaviors & Subordinate Characteristics.
M & E TOOLKIT Jennifer Bogle 11 November 2014 Household Water Treatment and Water Safety Plans International and Regional Landscape.
Nora Sabelli, NSF What could data mining and retrieval contribute to the study of education?
Queen’s Management & Leadership Framework
Organization Development and Change Thomas G. Cummings Christopher G. Worley Chapter Five: Diagnosing Organizations.
Researchers Without Borders Webinar 4 A Framework and Suite of Instruments for Examining Fidelity of Implementation Jeanne Century Center for Elementary.
THOUGHTFUL SUSTAINABILITY Teri Lewis Oregon State University Ronnie Detrich Wing Institute David StandifordOregon State University ABAI, San Antonio, Texas.
Why Community-University Partnerships? Partnerships Enhance quality of life in the region Increase relevance of academic programs Add public purposes to.
Victorian Curriculum: Introduction and overview
Course, Curriculum, and Laboratory Improvement (CCLI) Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics PROGRAM.
The Ohio STEM Learning Network: A Study of Factors Affecting Implementation Spread and Sustainability.
Spreading Improvements Heidi Johns, Quality Leader, BCPSQC July, 2013.
Organization Development and Change
A Multi-Level Framework to Understand Factors Influencing Program Implementation in Schools Celene E. Domitrovich, Ph.D. Penn State Prevention Research.
Cal-ABA 27th Annual Western Regional Conference Building a Data-based Decision Making Culture through Performance Management Randy Keyworth Ronnie Detrich.
Jeanne Century CEMSE, University of Chicago NRC Committee on Integrated STEM January 10, 2012 Defining iSTEM.
MANAGEMENT. ‘Managing is like holding a dove in your hand. If you squeeze too tight, you kill it. Open your hand too much, you let it go’. - T. Lasorda.
Stages of Research and Development
Introduction to Management and Organizations
Introduction to Management and Organizations
Coaching for Impact Susan Barrett
Health Systems Analysis
Implementation Science: Theories & Models
Introduction to Management and Organizations
Introduction to Management and Organizations
Introduction to Management and Organizations
Why are Managers Important?
INTRODUCTION TO MANAGEMENT
Figure 2 – Obstacles to Learning in Health markets
Explain why managers are important to organizations
Explain why managers are important to organizations
Introduction to Management and Organizations
Introduction to Management and Organizations
Fidelity of Implementation in Scaling-up Highly Rated Science Curriculum Units for Diverse Populations Carol O’Donnell and Sharon Lynch The George Washington.
Introduction to Management and Organizations
Introduction to Management and Organizations
Why are Managers Important?
Presentation transcript:

Improving Implementation Research Methods for Behavioral and Social Science Working Meeting Measuring Enactment of Innovations and the Factors that Affect Implementation and Sustainability: Moving toward Common Language and Shared Conceptual Understanding Jeanne Century Center for Elementary Mathematics and Science Education University of Chicago September 20 th, 2010

How and why do innovations spread and last? Two National Science Foundation supported projects: Accumulating Knowledge on Scaling and Sustaining Reform: A Foundation for Future Research Applying Research on Science Materials Implementation: Bringing Measurement of Fidelity of Implementation to Scale What is the innovation? What is spreading and lasting? Key Questions

Determine if innovations are effective Determine which parts of innovations are effective Determine why and under what conditions parts of innovations are effective Why Ask? We need to…. Determine how to support the spread of effective innovations with continued effectiveness Determine how to support the endurance of innovations

Use of multiple terms to mean the same thing Poor or absent definitions No conceptual frameworks Key Challenges Multiple conceptual frameworks Innovations don’t replicate; they translate

The Need for Common Language

A suite of instruments to measure enactment of innovations that can be used alone or in combination depending on resources and goals Instruments that provide clear and specific information about enactment of the innovation Outcomes of NSF Supported Projects Identification of factors affecting implementation (and sustainability) A conceptual framework that provides a foundation for instruments for measuring enactment of multiple innovations Clear, literature-based definitions of the factors affecting implementation (and sustainability) Conceptual organization of the factors that creates a starting point for measuring and modeling strength and direction of relationships

FOI The extent to which an enacted program is consistent with the intended program model Critical Components The elements of a program model that are essential to its implementation FOI Operationalized The extent to which the critical components of a program are present when the program is enacted Definitions for Fidelity of Implementation (FOI)

Combining Frameworks Approach Ruiz Primo (2005) – matrix of 5 dimensions and structure/process Lynch and O’Donnell (2005); Lastica and O’Donnell (2007) – matrix of 5 dimensions and structure/process Five Dimensions Approach Dane and Schneider (1998) 39/162 studies measured integrity In the 39 studies, the authors defined “integrity” along one or more of five dimensions the dimensions are: 1) adherence; 2) exposure; 3) quality of delivery; 4) participant responsiveness and 5) program differentiation Structure and Process Approach Mowbray et al. (1987), Wang et al. (1984) Structure (composition of the intervention) Process (human interactions that take place during delivery) FOI Approaches Critical Components Approach Hall and Hord (1987), Bond et al. 2000; Huntley, 2005; Sabelli & Dede (2001) and others Some called them model dimensions, fidelity criteria, critical parts

Combine critical component approach and structure process approach Show relationships to 5 dimensions approach Use combination of approaches to create categories of critical components and a framework that can apply to multiple interventions FOI Approaches CEMSE Approach

FOI Framework

C ategories of Differentiation (differentiation) Common Across Interventions Unique to Interventions CEMSE Framework Aligned with Other Approaches

Categories of Differentiation Common Program A Program B Program C Impact of a single program Compare two or more programs Compare programs to “business as usual” Compare two versions of a program Categories of Differentiation

Project on Factors Affecting Implementation and Sustainability Looked at economics, business, health, marketing, education Reviewed over 69,000 abstracts Read and coded over 600 reports Reconciled definitions used in the literature and conceptually organized them

Factors Affecting Implementation and Sustainability Implementation: Enactment of an innovation at a given point in time Maintenance: Lasting of an innovation characterized by a lack of change Sustainability: Lasting of an innovation characterized by its evolution and adaptation FOI: The extent to which an enacted program is consistent with the intended program model

Factors Affecting Implementation and Sustainability Ways of Lasting Sustainability Maintenance What Lasts The Innovation Ongoing Effectiveness Types of Movement Diffusion Dissemination Adoption Locus of Movement Movement: Originator to User Movement: User to User Movement Trust Time Phase/Stage Duration Strategies Planning Mandate Formative Evaluation Feedback Collaborative Change Process Properties of the Innovation Visibility Scope Complexity Adaptability Specificity Effectiveness Fit Fit: Current Practice Fit: Needs Fit: Values Elements of the Environment (Outside) External Climate: Social External Climate: Political Networks Resource Allocation Elements of the Environment (Inside) Internal Organizational Structure Internal Social Climate Resource Allocation Elements of the Environment Incentives Opportunities for Learning Locus of Decision-Making Characteristics of People in the Organization Sophistication of the User Quality of Leaders Characteristics of the Implementer Reasons for Decisions Buy-In Compliance Reward The engagement of the users in decisions related to an innovation Asking users for input, but not engagement them in decisions related to an innovation

The Innovation: An Example of Organizing Factors Ways of Lasting Sustainability Maintenance What Lasts The Innovation Ongoing Effectiveness Types of Movement Diffusion Dissemination Adoption Locus of Movement Movement: Originator to User Movement: User to User Movement Trust Time Phase/Stage Duration Strategies Planning Mandate Formative Evaluation Feedback Collaborative Change Process Properties of the Innovation Visibility Scope Complexity Adaptability Specificity Effectiveness Fit Fit: Current Practice Fit: Needs Fit: Values Elements of the Environment (Outside) External Climate: Social External Climate: Political Networks Resource Allocation Elements of the Environment (Inside) Internal Organizational Structure Internal Social Climate Resource Allocation Elements of the Environment Incentives Opportunities for Learning Locus of Decision-Making Characteristics of People in the Organization Sophistication of the User Quality of Leaders Characteristics of the Implementer Reasons for Decisions Buy-In Compliance Reward scope complexityspecificity critical components adaptability effectiveness visibility

Common Language What are the take-aways? Shared Conceptual Understanding Foundation for Accumulation of Knowledge Collaboration in Practice and Research “…if it were possible to make small experiments with wild ideas, while retaining the possibility of diffusing those that prove to be good ones, the adaptive position of exploration would be strengthened (Romano, 2002; Holahan, Weil and Wiener, 2003). Since structures that protect the system from the catastrophic consequences of wild ideas generally also inhibit the transfer of major discoveries (Cohen and Levinthal, 1989, 1990), there is no perfect solution to the problem.” March, 2005

Jeanne Century Center for Elementary Mathematics and Science Education University of Chicago rg Resources Learning Community Collaborative Problem Solving Contact Information