Design-Based Implementation Research to Study and Drive an Undergraduate STEM Education Improvement Project at Oregon State June 3, 2015 SMTI 2015 National Conference Jana Bouwma-Gearhart, Kathy Quardokus Fisher, Ann Sitomer, David Little, John Ivanovitch, Christina Smith, Milo Koretsky 1

Theory of action: Catalyze broad institutional change with respect to the use of Evidence- Based Instructional Practices (EBIPs) in large enrollment STEM classrooms through communities of practice (Wenger, 2009). Research goal: Study changes to STEM instructional practices and pedagogical knowledge via  Micro-level  educators’ and students’ practices  Meso-level  units’ routines, structures and culture  Macro-level  institutional routines, structures, and culture DBIR (Penuel et al., 2011)  Multiple stakeholders  Persistent problems of practice  Iterative and collaborative design Design-Based Implementation Research (DBIR) 2 Events Analysis Theory of Action

Enhancing STEM Education at Oregon State University  Within units  Communities of Practice (CoP)  Conference participation  Instruction-related data consultation  Across units  Community of Practice (CoP)  Collaboration with:  Center for Teaching and Learning (CTL)  STEM education research center  Other STEM change initiatives Events  Tenure track and non-tenure track faculty  Graduate teaching assistants (GTA) and undergraduate learning assistants (LA)  Chairs, deans, and vice provosts Participants 3

Identifying a Model for Organizational Change Research  Needs: Interpret a complicated system  Individual practices (cognition and action)  The social/cultural context  Aspects of their dynamic interaction  View of practitioners in light of change initiatives meant to foster more effective decision-making and other professional practices (Fumasoli & Stensaker, 2013)  “Building of [interventions] from practice” (Honig, 2003)  Past studies of organizational change  Focus on policy adoption with fidelity (Fumasoli & Stensaker, 2013)  Focus on mostly the macro scale or micro scale (Trowler et al., 2005; Trowler & Cooper, 2002) 4

Theoretical Frameworks Organizational Learning  Creating, retaining, or transferring knowledge in an organization Cultural-Historical Activity Theory  An organization understood as a collective, artifact-mediated, and object-oriented activity system Cultural Models  Shared information internalized through socialization, and used for guiding behavior in a complicated environment 5

Events Macro - Institution Meso - Department Micro - Individual 6 Center for Teaching and Learning Events Across Units CoP STEM Center Events Project Synthesis CoP on Specific Projects Conference Participation Instruction- related Data Consultation Conference Participation Organizational Structure

7 Data Sources Events Macro - Institution Meso - Department Micro - Individual Center for Teaching and Learning Events Across Units CoP STEM Center Events Project Synthesis CoP on Specific Projects Conference Participation Organizational Structure Instruction- related Data Consultation Conference Participation Administrative Interviews Policy Documents Project Team Interviews Event and CoP Observations Project Team Interviews CoP Observations Administrative and Educator Interviews GTA Practices Observation Educators Survey Educator InterviewsEducators Survey Classroom Observations Student Data

Events 8 Center for Teaching and Learning Events Across Units CoP STEM Center Events Project Synthesis CoP on Specific Projects Conference Participation Instruction- related Data Consultation Conference Participation Data Sources Administrative Interviews Policy Documents Project Team Interviews Event and CoP Observations Project Team Interviews CoP Observations Administrative and Educator Interviews GTA Practices Observation Educators Survey Educator Interviews Educators Survey Classroom Observations Student Data Macro - Institution Meso - Department Micro - Individual Events Collect Data based upon Theoretical Frameworks Analysis based upon Theoretical Frameworks Inform Theory of Action Design Design-Based Implementation Research Organizational Structure

Organizational Learning (Argote & Miron-Spektor, 2011) 9 Learning process: As an organization accomplishes tasks (experience) it interacts with its context and “learns” by creating, retaining, or transferring knowledge. Context is structure, culture, technology, identity, memory, goals, incentives, strategies, etc.  Active: members and tools  Latent: structure and culture Knowledge  Presented as: individuals’ schemas, routines, and the range of potential behavior  Distributed routines and cognition: Data sources and analysis  Educator and administrator interviews – qualitative analysis  Educator surveys – social network analysis  Analyzed at the macro and/or meso level and informed by the micro level

Cultural-Historical Activity Theory (CHAT) and Expansive Learning (Engestrom, 2001)  Learning Process: Expansive learning is driven by contradictions and accomplished when the object and motive are radically re-conceptualized (zone of proximal development of the activity)  Context: An activity system is collective, artifact-mediated, and object-oriented. It contains subjects, tools and signs, objects, rules, community, division of labor.  Activity system history is needed to understand a system’s problems and potential.  Historical context of division and labor: Data sources and analysis  Educator and administrator interviews – qualitative analysis  Policy documents – content analysis  Analyzed at the macro and/or meso level and informed by the micro level 10

Affordances of CHAT/Expansive Learning and Organizational Learning 11 TopicOrganizational LearningCHAT/Expansive Learning Sources of change and development Context interacts with experience Contradictions Types of ChangeCreating, retaining or transferring knowledge Mini-learning or expansive learning ContextActive: Members, tools and tasks Subjects, tools and signs, objects Latent: which individuals are members, what tools they have and which tasks they perform Rules, community, division of labor HistoryLens by which changes may be identified Lens by which changes may be understood Micro/Meso/MacroIncorporates what is happening at the micro level, but focusing more at meso and macro level Could be at any level, but tends to be based around an object at the meso level

Cultural Models (Ferrare & Hora, 2014)  Shared information internalized through socialization within and between groups.  Norms and practices are adopted, adapted and enacted by individuals to function in a complex environment  Cultural models of teaching and learning: Data sources and analysis  Educator and administrator interviews – qualitative analysis  Classroom observations - incidence/repertoires of practice  Analyzed at the micro level based upon individual’s judgments, perceptions and explanations of specific situations. 12

Events 13 Center for Teaching and Learning Events Across Units CoP STEM Center Events Project Synthesis CoP on Specific Projects Conference Participation Instruction- related Data Consultation Conference Participation Data Sources Administrative Interviews Policy Documents Project Team Interviews Event and CoP Observations Project Team Interviews CoP Observations Administrative and Educator Interviews GTA Practices Observation Educators Survey Educator Interviews Educators Survey Classroom Observations Student Data Macro - Institution Meso - Department Micro - Individual Events Collect Data based upon Theoretical Frameworks Analysis based upon Theoretical Frameworks Inform Theory of Action Design Design-Based Implementation Research

Events 14 Center for Teaching and Learning Events Across Units CoP STEM Center Events Project Synthesis Conference Participation Instruction- related Data Consultation Conference Participation Data Sources Administrative Interviews Policy Documents Project Team Interviews Event and CoP Observations GTA Practices Observation Project Team Interviews CoP Observations Administrative and Educator Interviews Educators Survey Educator Interviews Classroom Observations Student Data Macro - Institution Meso - Department Micro - Individual Analysis Examples CoP on Specific Projects Educators Survey

Potential for Organizational Learning in Networks Research Question: What is the potential for organizational learning at one research university regarding issues of teaching and learning/curriculum and instruction? 15 Organizational learning via transfer may be difficult because B_12 is not a participant CHAT question: How can the activity system of the unit make sense of the disconnect between B_04 and B_12 Legend: Non participant Participant Cultural models question: To what extent do individuals with many social connections have shared cultural models? DBIR: Given these analyses, what theory of action may be used to design events to improve the potential for organizational learning?

Faculty Teaching Climate Survey Data Ten items piloted from Knoreck (2012) Two items omitted due to lack of fit in Confirmatory Factor Analysis Analysis of eight climate items  N = 140 respondents  Reliability: α =.915  Unidimensional factor accounted for 68.6% of the variance Scale scores produced using item response theory  Equal variances assumed between climate scores of units,  Significant differences between means of climate scores of units,  Differences between units are of “large” effect size, η 2 16 Item Content (Knorek, 2012) Share teaching resources about how to improve their teaching with colleagues Discuss the challenges they face in the classroom with colleagues Consult with each other on teaching related issues Are encouraged by department administrators (e.g. department chair) to communicate with colleagues about their teaching Regularly discuss teaching related issues with department leadership Value faculty teaching development services available on campus as a way to enhance their teaching Believe that engaging in teaching improvement opportunities is part of their job Provides new faculty with teaching development opportunities and resources

DBIR Theory of Action with Low Climate Score 17 Legend: Non participant Participant Focus on the core group that has participated and discusses teaching regularly despite the low climate score

Strengths and Limitations of our Project and Meld of Theoretical Frameworks 18 StrengthsLimitations Our projects capacity 2 faculty members, 3 postdocs, 2 graduate students Are there too many details, as a result is the larger picture lost? Multi-perspective of various models Individuals, context, and interactions between the two When do we stop considering new frameworks? Potential for triangulationWill other change agents be able to use our model with less capacity for analysis?

Discussion and Questions How are you documenting and researching change in response to STEM education improvement initiatives in your institution or related organizations? What methodologies are you employing? What theoretical constructs concerning organizations and change guide your study? What is our model over-privileging/under-privileging? Per this discussion, what are some takeaways that can inform your plans for studying organizational change? What in our model may be most applicable, meaningful or replicable to your plans to study and document change? 19

20

DEPARTMENT CLIMATE IRT Item Parameters Item Content Itemab1b1 b2b2 b3b3 b4b4 b5b5 ShareRes Share teaching resources about how to improve their teaching with colleagues Challenges Discuss the challenges they face in the classroom with colleagues Consult Consult with each other on teaching related issues Encouraged Are encouraged by department administrators (e.g. department chair) to communicate with colleagues about their teaching Discuss Regularly discuss teaching related issues with department leadership Value Value faculty teaching development services available on campus as a way to enhance their teaching Believe Believe that engaging in teaching improvement opportunities is part of their job NewFaculty Provides new faculty with teaching development opportunities and resources 21

CLIMATE SCORES 22

References Argote, L., & Miron-spektor, E. (2011). Organizational Learning: From Experience to Knowledge. Organization Science, 22(5), 1123–1137. Engeström, Y. (2001). Expansive learning at work: Toward an activity theoretical reconceptualization. Journal of education and work, 14(1), Ferrare, J. J., & Hora, M. T. (2014). Cultural Models of Teaching and learning in Math and science: exploring the intersections of Culture, Cognition, and Pedagogical situations. The Journal of Higher Education, 85(6), Fumasoli, T., & Stensaker, B. (2013). Organizational studies in higher education: A reflection on historical themes and prospective trends. Higher Education Policy, 26(4), Honig, M. I. (2003). Building policy from practice: District central office administrators' roles and capacity for implementing collaborative education policy. Educational Administration Quarterly, 39(3), Knorek, J. K. (2012). Faculty teaching climate: Scale construction and initial validation. Unpublished PhD dissertation, University of Illinois, Urbana, IL Penuel, W. R., Fishman, B. J., Haugan Cheng, B., & Sabelli, N. (2011). Organizing Research and Development at the Intersection of Learning, Implementation, and Design. Educational Researcher, 40(7), 331–337. doi: / X Trowler, P. & Cooper, A. (2002) Teaching and learning regimes: implicit theories and recurrent practices in the enhancement of teaching and learning through educational development programmes, Higher Education Research and Development, 21(3), 221–240. Trowler, P et al. (2005). Freeing the chi of change: the Higher Education Academy and enhancing teaching and learning in higher education. Studies In Higher Education, 30(4), Wenger, E. (2009). Communities of practice. Communities, 22,

Institutional culture models, structures, routines, synergy across projects, artifacts of organizational history, leadership and other resources External organizational influences also accounted for Community activities and networks, distributed cognition/routines/felt affordances regarding pedagogy and improvements Departmental culture models, decision-making, training and other resources regarding pedagogy Educators’ schemas, sensemaking, routines, felt affordances and felt self- determination regarding pedagogical practices/knowledge/commitmen ts/ change activities/individual improvements MESO Interdisciplinary communities of practice concerning projects’ targeted evidence- based instructional practices and community-identified problems; Informal socials Project Activities Data Sources (and analytics) Intradisciplinary communities of practice concerning community-identified problems; Disciplinary conference participation Synergy with projects/stakeholders/suborganization s to revise structures to better reward/support pedagogical innovation and interdisciplinary work Data of educators’ practices and student impacts/factors provided for reflection; Support for individual pedagogical innovation (leadership) Data of educators’ practices and student impacts/factors provided for reflection; Support for individual pedagogical innovation (leadership) Administrator interviews (qualitative inductive analysis) Policy analysis (content analysis) Project(s) leadership interviews (qualitative inductive analysis) Administrator interviews (qualitative inductive analysis) Policy analysis (content analysis) Project(s) leadership interviews (qualitative inductive analysis) Community of practice observations (distributed routines/cognition analysis) Educator and administrator interviews (distributed routines/cognition analysis) Project leadership interviews (qualitative inductive analysis) Community of practice observations (distributed routines/cognition analysis) Educator and administrator interviews (distributed routines/cognition analysis) Project leadership interviews (qualitative inductive analysis) GTA practices and development video (discourse analysis) Educator and administrator interviews (qualitative inductive analysis) Educator surveys (department climate analysis, social network analysis, distributed routines/cognition analysis) GTA practices and development video (discourse analysis) Educator and administrator interviews (qualitative inductive analysis) Educator surveys (department climate analysis, social network analysis, distributed routines/cognition analysis) Educator interviews (qualitative inductive analysis) Educator surveys (knowledge/incidences self-report of practices, scaled metrics for determination, social network analysis) Class observations (incidence/repertoires of practice, characterization of students’ cognitive demand) Student data (qualitative and quantitative formative analysis) Course artifacts (document and content analysis) Educator interviews (qualitative inductive analysis) Educator surveys (knowledge/incidences self-report of practices, scaled metrics for determination, social network analysis) Class observations (incidence/repertoires of practice, characterization of students’ cognitive demand) Student data (qualitative and quantitative formative analysis) Course artifacts (document and content analysis) MACRO MICRO Possibilities for and evidence of organizational status and change centered in Cultural Models Theory, Organizational Learning, and Cultural Historical Activity Theory Frameworks Design-Based Implementation Research To Study and Drive An Undergraduate STEM Education Improvement Project at Oregon State 24