1. Scaling a Cyber-Enabled, Just-in-Time-Teaching with Two-Way Formative Feedback (JTF) Pedagogy to the Multiple Disciplinary Program Level (JTFD)
Stephen Krause, James Middleton, Keith Hjelmstad, Eugene Judson, Robert Culbertson, Casey Ankeny,
Ying-Chi Chen, Bethany Smith, Yong-Seok Park, Lydia Ross, & Danielle Stevens
Contact:
NSF Grant#
Objective
Assessments
Preliminary Data
 To shift faculty practice towards student-centered instruction through the facilitation of faculty development workshops and the creation of disciplinary communities of practice
Measuring Awareness (Roger’s Awareness)
Evaluation and Results of VECTERS
A short survey about awareness of various teaching practices was developed and administered
4-point Likert scale from very unfamiliar to very familiar
“How familiar are you with the following topics? Research about effective teaching practices, Bloom’s Taxonomy, etc.”
Administered by to 6300 faculty from the top 20 engineering schools
283 responses  Evaluation of Survey Possible (Cronbach’s α = .90)
Differences between male and female faculty responses were identified using Analysis of Covariance (ANCOVA)
Background
JTF is a web-enabled, engagement and feedback pedagogy that uses muddiest points for two-way formative feedback and classroom engagement activities with online resources
JTFD involves training faculty in JTF pedagogy through a series of workshops. Using the train-the trainer-model[1] , disciplinary leader pairs are trained in JTF pedagogy during one semester. They then train cohorts in their disciplines in JTF pedagogy using the same materials the next semester.
F (1, 267) p
Partial Eta-squared
Formative Feedback – Value for students
8.90
0.003
0.033
Formative Feedback - Expect success due to students
7.26
0.008
0.027
Formative Feedback - Expect success based on self
11.66
0.001
0.043
Real-world applications – Value for students
6.53
0.011
0.024
Real-world app. - Expect success due to students
5.94
0.015
0.022
Student discussions - Expect success due to students
5.22
0.023
0.020
Measuring Classroom Practice (Roger’s Trial/Adoption)
The RTOP (Reformed Teaching Observational Protocol) is a classroom observational protocol that quantifies the extent to which faculty exhibit student-centered behaviors in their classroom practice[5]
Higher RTOP scores shown to correlate with higher student outcomes[6]
Measuring Faculty Beliefs (Coburn’s Depth)
Year 1 (F15-Sp16)
Year 2 (F16-Sp17)
Year 3 (F17-Sp18)
Year 4 (F18-Sp19)
Cohort 1 Tier 1 Disciplinary Leader Pairs (DLPs)
(AE, ME, CEE, CON)
Being Trained by Project Leaders + Classroom Implementation
Follow-on Assessment of DLP Faculty Individuals' Classroom Practice
Cohort 1 Tier 2 Disciplinary Faculty Groups (DFGs)
Being Trained by DLPs + Classroom Implementation
Follow-on Assessment of DFG Faculty Individuals' Classroom Practice
Cohort 2 Tier 1 Disciplinary Leader Pairs (DLPs)
(CHE, MSE, BME)
Cohort 2 Tier 2 Disciplinary Faculty Groups (DFGs)
The ATI (Approaches to Teaching Inventory) is a self-reporting tool that measures extent of faculty teaching beliefs toward instructor-centered knowledge transmission vs. student-centered conceptual change[7]
4 subscales measuring extent of teacher focused strategy (TFS), teacher focused intention (TFI), student-focused strategy, (SFS) and student focused intention (SFI)
5-point Likert Scale from Strongly Disagree to Strongly Agree:
“I think an important reason for running teaching sessions in this subject/course is to give students a good set of notes (TFI)”
“I feel that it is better for students in this subject/course to generate their own notes rather than always copy mine (SFI)”
A short free-response questionnaire about teaching beliefs and practices was developed and administered:
“Describe the strategies you use to carry out your instruction.”
“By what personal criteria do you judge the effectiveness of your instruction?”
VECTERS (The Value, Expectancy, and Cost of Testing Educational Reforms Survey) was designed to assess dispositions of engineering faculty regarding student-centered instruction about the specific strategies of formative feedback, real-world applications, and student-to-student discussion.
The instrument evaluated respondents’ considered value of a strategy, the success they expected from using the strategy, and the perceived cost to implement a strategy, using the Expectancy/Value Theory of Motivation[8]
Few gender based differences for actual use of student-centered teaching strategies
Significant gender-based differences were found for faculty dispositions towards student-centered teaching strategies
Women were more positive regarding expectation of success and placed greater value on the strategies of formative feedback and real-world applications
Women had more positive dispositions than men across all strategies
Training Table for JTFD
Disciplinary Communities of Practice (CoP) will be formed, which will increase the likelihood of adopting JTF pedagogy[2].
Roger’s Adoption of Innovation Model[3]
Characteristics of Change within Individuals:
Knowledge or Awareness - Individual exposed to innovation
Persuasion or Interest - Individual seeks more information
Evaluation and Decision - Individual adopts or rejects innovation
Implementation or Trial - Individual puts innovation to use
Confirmation or Adoption - Innovation is continued/sustained
Disciplinary Leader Pairs: Changes in Classroom Practice
Disciplinary Leader
Pre-RTOP
Min
Max
The day for Max RTOP
Which workshop affects positive impact
Max Gain (%)
Avg.
Avg Gain (%)
AE1 22 51
:56
Session 4: Implementing Active Engineering
131.8% 34
54.5%
AE2 27 59
:30
session 3: Interactive Classes
118.5% 46
70.4%
CEE1 43 62
:54
Session 5: Cooperative Learning
44.2%
52.5
22.1%
CEE2 49 37 88
:44
137.8%
61.7
66.8%
CON1 31 69
:03
122.6% 55
77.4%
CON2 29 54
:55
86.2% 40
37.9%
ME1 17 60
:09
252.9%
32.8
92.9%
ME2 67 64 70
:35
9.4%
66.6
4.1%
Coburn’s Innovation Scaling Model[4]
Characteristics of Effective and Sustainable Scaling of Innovation:
Depth – Faculty change beliefs at a deep level
Sustainability – Innovation is adaptable in many settings
Spread – Beliefs and norms change across organization
Shift of Ownership – Shifts from facilitator to faculty practitioner
Measuring Social Networks (Coburn’s Spread)
Project Outcomes
For social network analysis, participants are asked who they speak with about teaching, curriculum, assessments, and technology/tools along with directionality.
Previous work shows that higher social connectivity significantly correlated with higher RTOP scores[9]
Promote change in faculty beliefs and practice toward student-centered instruction.
Assess the impact of the faculty development program on the change in faculty beliefs and practice.
Develop sustainable disciplinary CoPs during the processes of faculty training.
Assess the effect of the classroom implementation of JTFD principles, by discipline, on student attitude, learning, and persistence.
Disseminate JTFD strategies, tools, and findings.
Sessions 3, 4 and 5 in the workshops had the most impact on RTOP score
Overall average gain for Cohort 1 Disciplinary Leader Pairs was 53%
We intend to measure change in these metrics by comparing
pre-workshop data to data during and after the workshop sessions.
Future Work
Collect and analyze data from Disciplinary Leader Pairs and Faculty
Continue following the Training Table for Fall 2016 participants
Disseminate findings to the STEM education community
[5] Piburn, M. et. al. (2000). Reformed Teaching Observation Protocol(RTOP). ACEPT
[6] Lawson et. al. (2002). Reforming and evaluating college science and mathematics instruction. Journal of College Science Teaching, 31, 388–393.
[7] Trigwell, K., and Prosser, M. (2004). Development and use of the approaches to teaching inventory. Educational Psychology Review, 16(4),
[8] Shah, J., & Higgins, E. T. (1997). Expectancy× value effects: Regulatory focus as determinant of magnitude and direction. Journal of personality and social psychology, 73(3), 447.
[9] Middleton, J et. (2015). Examining the Relationship between Faculty Teaching Practice and Interconnectivity in a Social Network. Paper presented at the 122nd ASEE Annual Conference & Exposition. Seattle, WA.
[1] Pimmel, R., McKenna, A., Fortenberry, N., Yoder, B., and Guerra, R. (2013). Faculty development using virtual communities of practice ASEE Annual Conference Proceedings.
[2] Pimmel, R., and McKenna, A. (2014).Faculty development using virtual communities of practice ASEE Annual Conference Proceedings
[3] Rogers, E.M. (1962). Diffusion of innovations. Glencoe: Free Press.
[4] Coburn, C.E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3-12.