Effect of Implementation of JTF Engagement and Feedback Pedagogy on Change of Faculty Beliefs and Student Performance NSF Grant #1226325 Steve Krause (ASU), Terry Alford (ASU), Shariar Anwar (ASU), Adam Carberry (ASU), Candace Chan (ASU), Casey Ankeny (ASU), Bethany Smith (ASU), Brady Gibbons (OSU), Milo Koretsky (OSU), Bill Brooks (OSU), Deb Gilbuena (OSU), Cindy Waters (NCAT), Joe Stuart (OIT), Peggie Weeks (Ext. Evaluator), Eugene Judson (Internal Evaluator), Karl Smith (Advisory Bd.) and John Baglin (Advis Bd.) Just-in-Time-Teaching with Interactive Frequent Formative Feedback (JTF) Objective Results Student Attitude, Achievement, and Persistence with JTF JTF Student Course Persistence - 2nd week to final To test the effectiveness of JTF pedagogy on changing faculty beliefs and classroom practice and on changing associated student performance across diverse settings 97% Persistence (n = 227/235) Fa2013 - Four Institutions, five courses 95% Persistence (n = 311/328) Sp2014 - Four Institutions, five courses Roger’s Model – JTF Innovation Change Knowledge, Persuasion, and Evaluation of JTF - 9 out 9 faculty agreed that, “Being involved in the JTF project has motivated me to change my classroom practice.” “Increased awareness of confusing or poorly explained concepts.” “Students are made part of the learning team.” “I heard from several students directly that using muddiest points helped them engage in the class further than they normally would.” Implementation and Adoption of JTF - 7 out of 8 faculty agreed that “their views on teaching had changed somewhat or changed significantly” 8 out of 8 faculty agreed that “their classroom practice had changed Background Class Persistence Student Impact Value Survey JTF is a web-enabled, engagement and feedback pedagogy that uses muddiest points for two-way formative feedback and classroom engagement activities with online resources Shift to Student-Centered Classroom Practice Student Achievement with JTF Across 3 Universities and 4 Classes Cindy Waters; North Carolina A&T – Intro Materials Shariar Anwar; Arizona State – Math. Methods Materials Coburn’s Model – JTF Scaling of Innovation Depth: Change of Belief toward Student-Centered Learning- What is your role in the classroom today compared to before JTF? “More of a coach than a ‘lecturer’.” “More of a guide now.” “More as a coach and encourage and guide…” “…I am a guide and they must take on the learning.” “...Now I feel like, in addition to those, I am asking questions of EACH student during lecture. That’s not possible without what we are doing.” Sustainability: Adaptability of JTF in Many Classroom Settings - Six faculty rated, on a scale from 1-10: Frequency of contextualization for class content rated 9.3 Effectiveness of engagement strategies for learning rated 8.3 Adopted by multiple faculty in diverse settings at 4 institutions Spread: Creation of Community of Practice (CoP) - All faculty collected Muddiest Points and adjusted instruction 5 to 8 faculty participated in monthly web meetings over 2 years 6 faculty participated in two ASEE JTF workshops 2 faculty created 2 new survey tools used in the CoP Shift of Ownership: External Facilitators to Faculty Practitioners - All participants use web-enabled, two-way formative feedback All use web-based approaches to creating classroom materials Feedback pedagogy use increased from 3.9 pre-JTF to 8.3 post-JTF Pre-JTF: Mean = 68.0%; SD = 15.5% Post-JTF: Mean = 76.6%; SD = 19.5% Pre-JTF: Mean = 71.8%; SD = 17.9% Post-JTF: Mean = 81.2%; SD = 7.5% Traditional Shifted Classroom Roger’s Innovation Change Model[1] Characteristics of Change within Individuals: Knowledge or Awareness - Individual exposed to innovation Persuasion or Interest - Individual seeks more information Evaluation and Decision - Individual adopts or rejects innovation Implementation or Trial - Individual puts innovation to use Confirmation or Adoption - Innovation is continued/sustained Joe Stuart; Oregon Institute of Technology – Intro Materials Stephen Krause; Arizona State – Intro Materials Pre-JTF: Mean = 86.8%; SD = 7.8% Post-JTF: Mean = 92.0%; SD = 6.8% '09: Mn = 66.3%; SD = 13.8% '11: Mn = 71.1%; SD = 17.9% '13: Mn = 77.0%; SD = 13.0% Coburn’s Scaling of Innovation Model[2] Characteristics of Effective and Sustainable Scaling of Innovation: Depth – Faculty change beliefs at a deep level Sustainability – Innovation is adaptable to many settings Spread – Beliefs and norms change across an organization Shift of Ownership – Shifts from facilitator to faculty practitioner Conclusions Faculty shifted beliefs and adopted JTF pedagogy in classroom practice Faculty assumed ownership of JTF innovations in a community of practice These factors enhanced student attitude, achievement, and persistence [1] Rogers, E.M. (1962). Diffusion of innovations. Glencoe: Free Press. [2] Coburn, C.E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3-12. Project Outcomes Collaborating Institutions Implement web-based JTF instructional resources Build Community of Practice to implement JTF pedagogy. Investigate how instructors use feedback to adjust instruction Assess effect of JTF on student attitude, learning & retention Find potential of JTF strategies for other engineering domains Disseminate JTF strategies, findings, and products Dissemination of JTF Methods and Publications Granta, FIE (3 papers), and ASEE (3 posters, 2 workshops, 10 papers) Callister 9th ed. Materials Text – MP tutorial videos; e Vocab.; MPs skrause@asu.edu milo.koretsky@oregonstate.edu oitjoe@gmail.com kwaters@ncat.edu