Scaling a Cyber-Enabled, Just-in-Time-Teaching with Two-Way Formative Feedback (JTF) Pedagogy to the Multiple Disciplinary Program Level (JTFD) Stephen Krause, James Middleton, Keith Hjelmstad, Eugene Judson, Robert Culbertson, Casey Ankeny, Ying-Chi Chen, Bethany Smith, Yong-Seok Park, Lydia Ross, & Danielle Stevens Contact: jimbo@asu.edu NSF Grant#1226325 Objective Assessments Preliminary Data  To shift faculty practice towards student-centered instruction through the facilitation of faculty development workshops and the creation of disciplinary communities of practice Measuring Awareness (Roger’s Awareness) Evaluation and Results of VECTERS A short survey about awareness of various teaching practices was developed and administered 4-point Likert scale from very unfamiliar to very familiar “How familiar are you with the following topics? Research about effective teaching practices, Bloom’s Taxonomy, etc.” Administered by email to 6300 faculty from the top 20 engineering schools 283 responses  Evaluation of Survey Possible (Cronbach’s α = .90) Differences between male and female faculty responses were identified using Analysis of Covariance (ANCOVA) Background JTF is a web-enabled, engagement and feedback pedagogy that uses muddiest points for two-way formative feedback and classroom engagement activities with online resources JTFD involves training faculty in JTF pedagogy through a series of workshops. Using the train-the trainer-model[1] , disciplinary leader pairs are trained in JTF pedagogy during one semester. They then train cohorts in their disciplines in JTF pedagogy using the same materials the next semester.   F (1, 267) p Partial Eta-squared Formative Feedback – Value for students 8.90 0.003 0.033 Formative Feedback - Expect success due to students 7.26 0.008 0.027 Formative Feedback - Expect success based on self 11.66 0.001 0.043 Real-world applications – Value for students 6.53 0.011 0.024 Real-world app. - Expect success due to students 5.94 0.015 0.022 Student discussions - Expect success due to students 5.22 0.023 0.020 Measuring Classroom Practice (Roger’s Trial/Adoption) The RTOP (Reformed Teaching Observational Protocol) is a classroom observational protocol that quantifies the extent to which faculty exhibit student-centered behaviors in their classroom practice[5] Higher RTOP scores shown to correlate with higher student outcomes[6] Measuring Faculty Beliefs (Coburn’s Depth)   Year 1 (F15-Sp16) Year 2 (F16-Sp17) Year 3 (F17-Sp18) Year 4 (F18-Sp19) Cohort 1 Tier 1 Disciplinary Leader Pairs (DLPs) (AE, ME, CEE, CON) Being Trained by Project Leaders + Classroom Implementation Follow-on Assessment of DLP Faculty Individuals' Classroom Practice Cohort 1 Tier 2 Disciplinary Faculty Groups (DFGs) Being Trained by DLPs + Classroom Implementation Follow-on Assessment of DFG Faculty Individuals' Classroom Practice Cohort 2 Tier 1 Disciplinary Leader Pairs (DLPs) (CHE, MSE, BME) Cohort 2 Tier 2 Disciplinary Faculty Groups (DFGs) The ATI (Approaches to Teaching Inventory) is a self-reporting tool that measures extent of faculty teaching beliefs toward instructor-centered knowledge transmission vs. student-centered conceptual change[7] 4 subscales measuring extent of teacher focused strategy (TFS), teacher focused intention (TFI), student-focused strategy, (SFS) and student focused intention (SFI) 5-point Likert Scale from Strongly Disagree to Strongly Agree: “I think an important reason for running teaching sessions in this subject/course is to give students a good set of notes (TFI)” “I feel that it is better for students in this subject/course to generate their own notes rather than always copy mine (SFI)” A short free-response questionnaire about teaching beliefs and practices was developed and administered: “Describe the strategies you use to carry out your instruction.” “By what personal criteria do you judge the effectiveness of your instruction?” VECTERS (The Value, Expectancy, and Cost of Testing Educational Reforms Survey) was designed to assess dispositions of engineering faculty regarding student-centered instruction about the specific strategies of formative feedback, real-world applications, and student-to-student discussion. The instrument evaluated respondents’ considered value of a strategy, the success they expected from using the strategy, and the perceived cost to implement a strategy, using the Expectancy/Value Theory of Motivation[8] Few gender based differences for actual use of student-centered teaching strategies Significant gender-based differences were found for faculty dispositions towards student-centered teaching strategies Women were more positive regarding expectation of success and placed greater value on the strategies of formative feedback and real-world applications Women had more positive dispositions than men across all strategies Training Table for JTFD Disciplinary Communities of Practice (CoP) will be formed, which will increase the likelihood of adopting JTF pedagogy[2]. Roger’s Adoption of Innovation Model[3] Characteristics of Change within Individuals: Knowledge or Awareness - Individual exposed to innovation Persuasion or Interest - Individual seeks more information Evaluation and Decision - Individual adopts or rejects innovation Implementation or Trial - Individual puts innovation to use Confirmation or Adoption - Innovation is continued/sustained Disciplinary Leader Pairs: Changes in Classroom Practice Disciplinary Leader Pre-RTOP Min Max The day for Max RTOP Which workshop affects positive impact Max Gain (%) Avg. Avg Gain (%) AE1 22 51 2016-02-15 14:56 Session 4: Implementing Active Engineering 131.8% 34 54.5% AE2 27 59 2016-02-04 13:30 session 3: Interactive Classes 118.5% 46 70.4% CEE1 43 62 2016-02-23 14:54 Session 5: Cooperative Learning 44.2% 52.5 22.1% CEE2 49 37 88 2016-02-16 13:44 137.8% 61.7 66.8% CON1 31 69 2016-02-04 12:03 122.6% 55 77.4% CON2 29 54 2016-02-24 8:55 86.2% 40 37.9% ME1 17 60 2016-02-11 15:09 252.9% 32.8 92.9% ME2 67 64 70 2016-02-18 13:35 9.4% 66.6 4.1% Coburn’s Innovation Scaling Model[4] Characteristics of Effective and Sustainable Scaling of Innovation: Depth – Faculty change beliefs at a deep level Sustainability – Innovation is adaptable in many settings Spread – Beliefs and norms change across organization Shift of Ownership – Shifts from facilitator to faculty practitioner Measuring Social Networks (Coburn’s Spread) Project Outcomes For social network analysis, participants are asked who they speak with about teaching, curriculum, assessments, and technology/tools along with directionality. Previous work shows that higher social connectivity significantly correlated with higher RTOP scores[9] Promote change in faculty beliefs and practice toward student-centered instruction. Assess the impact of the faculty development program on the change in faculty beliefs and practice. Develop sustainable disciplinary CoPs during the processes of faculty training. Assess the effect of the classroom implementation of JTFD principles, by discipline, on student attitude, learning, and persistence. Disseminate JTFD strategies, tools, and findings. Sessions 3, 4 and 5 in the workshops had the most impact on RTOP score Overall average gain for Cohort 1 Disciplinary Leader Pairs was 53% We intend to measure change in these metrics by comparing pre-workshop data to data during and after the workshop sessions. Future Work Collect and analyze data from Disciplinary Leader Pairs and Faculty Continue following the Training Table for Fall 2016 participants Disseminate findings to the STEM education community [5] Piburn, M. et. al. (2000). Reformed Teaching Observation Protocol(RTOP). ACEPT http://PhysicsEd.BuffaloState.Edu/AZTEC/rtop/RTOP_full/PDF. [6] Lawson et. al. (2002). Reforming and evaluating college science and mathematics instruction. Journal of College Science Teaching, 31, 388–393. [7] Trigwell, K., and Prosser, M. (2004). Development and use of the approaches to teaching inventory. Educational Psychology Review, 16(4), 409-424. [8] Shah, J., & Higgins, E. T. (1997). Expectancy× value effects: Regulatory focus as determinant of magnitude and direction. Journal of personality and social psychology, 73(3), 447. [9] Middleton, J et. (2015). Examining the Relationship between Faculty Teaching Practice and Interconnectivity in a Social Network. Paper presented at the 122nd ASEE Annual Conference & Exposition. Seattle, WA. [1] Pimmel, R., McKenna, A., Fortenberry, N., Yoder, B., and Guerra, R. (2013). Faculty development using virtual communities of practice. 2013 ASEE Annual Conference Proceedings. [2] Pimmel, R., and McKenna, A. (2014).Faculty development using virtual communities of practice. 2014 ASEE Annual Conference Proceedings [3] Rogers, E.M. (1962). Diffusion of innovations. Glencoe: Free Press. [4] Coburn, C.E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3-12.