Investigation of Inverted and Active Pedagogies in STEM Disciplines: A Preliminary Report Reza O. Abbasian Texas Lutheran University, Department of Mathematics, Computer Science, and Information Systems rabbasian@tlu.edu Michael Czuchry Texas Lutheran University, Department of Psychology mczuchry@tlu.edu John T. Sieben Texas Lutheran University, Department of Mathematics, Computer Science, and Information Systems jsieben@tlu.edu

Overview Description of the project and grant Goals of the project Methodology and data gathering Preliminary results Future work Acknowledgements References

Description of the project NSF, IUSE grant Three years, $290K Study the effectiveness of Inverted classrooms and active learning in STEM disciplines Inverted versus Traditional courses in Statistics, Biology, Physics and Chemistry Faculty involvement: Six (plus a few unpaid volunteers) faculty managing the grant and teaching the inverted classes, one data analyst and one archivist/web site manager Courses: STAT 374 - 375, BIOL 143-144, CHEM 143- 144, PHYS 141-142

Goals and Objectives of the project The overall goal of the project is to develop, implement and evaluate the impact of the Inverted and Active Learning Pedagogies ( IALP) for Student Success and Retention in STEM disciplines Specific goals: 1. Develop IALP courses which optimize content coverage and maximize higher- order learning Summer 2017 workshop(s) to create videos Summer workshop/presentations for best practices on effective inverted classroom teaching Summer 2017 : Develop data collection instruments and the management tools to analyze data Summer 2017: develop pre- and post-tests for each course Fall 2017 and 2018: Discussion sessions each semester on sharing experiences December 2017 and 2018, May 2018 and 2019 : Collect data on the results of pre- /post-tests for Inverted and traditional courses and conduct faculty and student surveys.

2. Determine the impact of IALP on the success and retention of STEM majors Analyze the data at the end of each semester and the cumulative results Analyze registrar’s data by academic level and major Determine the impact of IALP courses on the retention rate Analyze the student surveys to determine the impact of ILAP course satisfaction

3. Disseminate to the national STEM community the projects findings and course material Disseminate faculty work through presentations and publications Create an archive of IALP material ( videos, projects, etc.) Serve as mentors to other universities to replicate and implement the strategies and materials

Methodology and Data Gathering (Fall 2017 and Spring 2018) We examined faculty teaching traditional or inverted/hybrid classes in Statistics, Biology (Fall only), and Chemistry We used Bloom’s Taxonomy to create questions at multiple cognitive levels Students in all classes were administered the exams at the beginning and end of each semester Grades and withdrawals over the course of the semester were also examined

Approach for Question Generation Cognitive Dimensions for Multiple-Choice Questions 18 Q Knowing 6 Q Doing 6 Q Interpreting = 30 Questions total Remember Understand Apply Analyze Evaluate Cognitive Dimension Knowledge Dimension 1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create A. Factual B. Conceptual C. Procedural D. Metacognitive 3 3 2 1 1 3 2 1 1 3 3 3 2 1 1

Preliminary Results (Pre- Post-Tests) 2X3X2 Mixed MANOVAs were conducted examining pre- post- tests Condition (Inverted/Hybrid vs. Traditional) Discipline (Statistics vs. Biology* vs. Chemistry) Time (Beginning of Semester vs. End of Semester) *Fall 2017 only Dependent variables (in separate MANOVAs or ANOVAs) included: Total score Factual, Conceptual, and Procedural Knowledge Remembering and Understanding Application Analyzing and Evaluating

Sample Sizes (Fall 2017) Discipline Statistics Biology Chemistry Totals Inverted/Hybrid N = 20 N = 28 N = 24 N = 72 Class Type Traditional N = 25 N = 19 N = 28 N = 72 Totals N = 45 N = 47 N = 52 N = 144

Representative Findings (Fall 2017)

Figure 1. Means and Standard Errors for the Percent Correct at the Beginning and End of the Semester. * * * % Correct Notes. *p < .05; Fall 2017

Figure 2. Means and Standard Errors for the Percent Correct for Procedural Knowledge at the Beginning and End of the Semester in Statistics, Biology, and Chemistry Classes. * * * % Correct Procedural Knowledge Notes. *p < .05; Fall 2017; Statistically significant interaction

Figure 3. Means and Standard Errors for the Percent Correct for Procedural Knowledge at the Beginning and End of the Semester in Inverted/Hybrid and Traditional Classes. * * * % Correct Procedural Knowledge Notes. *p < .05; Fall 2017; Statistically significant interaction

Summary of Findings (Fall 2017) Students did better over time demonstrating important learning Students did better in Statistics classes over time compared to Biology and Chemistry classes Students in traditional classes did better than inverted/hybrid classes on procedural knowledge over time as well as understanding* * Results not depicted in prior Figures

Representative Findings (Spring 2018)

Sample Sizes (Spring 2018) Discipline Statistics Chemistry Totals Inverted/Hybrid N = 35 N = 28 N = 63 Class Type Traditional N = 26 N = 13 N = 39 Totals N = 61 N = 41 N = 102

Figure 4. Means and Standard Errors for the Percent Correct at the Beginning and End of the Semester. * * * % Correct Notes. *p < .05; Spring 2018

Figure 5. Means and Standard Errors for the Percent Correct for Procedural Knowledge at the Beginning and End of the Semester in Statistics, Biology, and Chemistry Classes. * * % Correct Procedural Knowledge Notes. *p < .05; Spring 2018; Statistically significant interaction

Figure 6. Means and Standard Errors for the Percent Correct for Procedural Knowledge at the Beginning and End of the Semester in Inverted/Hybrid and Traditional Classes. * * % Correct Procedural Knowledge Notes. *p < .05; Spring 2018; Interaction p = .158

Summary of Findings (Spring 2018) Again, students did better over time demonstrating important learning Again, students did better in Statistics classes over time compared to Chemistry classes Students in traditional classes did better than inverted/hybrid classes on evaluating information over time* * Results not depicted in prior Figures

Preliminary Results (Final Grades and Withdrawals) 2X3 ANOVAs were conducted examining final grades and withdrawals Condition (Inverted/Hybrid vs. Traditional) Discipline (Statistics vs. Biology* vs. Chemistry) *Fall 2017 only Dependent variables (in separate ANOVAs) included: Final grade in courses Withdrawals from course (Fall 2017 only)

Sample Sizes for Final Grade (Fall 2017) Discipline Statistics Biology Chemistry Totals Inverted/Hybrid N = 24 N = 29 N = 31 N = 84 Class Type Traditional N = 26 N = 19 N = 33 N = 78 Totals N = 50 N = 48 N = 64 N = 162

Figure 7. Means and Standard Errors for Final Grades in Statistics, Biology, and Chemistry. * * Average Grade Received Notes. *p < .05; Fall 2017

Figure 8. Means and Standard Errors for Final Grades in Inverted/Hybrid and Traditional Classes. Average Grade Received Note. Fall 2017

Sample Sizes Final Grades (Spring 2018) Discipline Statistics Chemistry Totals Inverted/Hybrid N = 45 N = 30 N = 75 Class Type Traditional N = 28 N = 14 N = 42 Totals N = 73 N = 44 N = 117

Figure 9. Means and Standard Errors for Final Grades in Statistics and Chemistry. * Average Grade Received Notes. *p < .05; Spring 2018

Figure 10. Means and Standard Errors for Final Grades in Inverted/Hybrid and Traditional Classes. * Average Grade Received Notes. * p < .05; Spring 2018; Effect holds with Statistics N = 44 and Chemistry N = 44

Sample Sizes for Withdrawals (Fall 2017) Discipline Statistics Biology Chemistry Totals Inverted/Hybrid N = 25 N = 31 N = 31 N = 87 Class Type Traditional N = 27 N = 25 N = 37 N = 89 Totals N = 52 N = 56 N = 68 N = 176

Figure 11. Means and Standard Errors for Withdrawals in Statistics, Biology, and Chemistry. * * Proportion of Withdrawals Note. *p < .05; Fall 2017

Figure 12. Means and Standard Errors for Withdrawals in Inverted/Hybrid and Traditional Classes. * Proportion of Withdrawals Note. *p < .05

Summary of Primary Findings (Final Grades and Withdrawals) Students had higher grades in Statistics than in Biology or Chemistry Students withdrew more from Biology than Statistics or Chemistry in Fall 2017 Students withdrew more from traditional classes than inverted/hybrid classes in Fall 2017 Students had higher grades in inverted/hybrid classes than traditional classes in Spring 2018

Key Points Students demonstrated important learning over time Traditional lectures may benefit learning to a degree (at least as evaluated here) Decreased withdrawals from inverted/hybrid classes warrants further attention Higher grades in Spring 2018 for inverted/hybrid classes warrants further attention

Limitations Students were not randomly assigned to classes and thus many factors influence which class they end up in Cannot rule out potential impact of professor (although some of the findings collapse across professors, and across disciplines) The degree to which the class was hybrid or inverted could influence what was observed

Future Work Examine transfer of learning in sequential classes (e.g., STAT 374 – STAT 375) Examine faculty surveys on number, type, and length of instructional videos Examine content that is inverted vs. traditional in the same classes (to control for potential effects of professor) Examine the role of technology

Acknowledgements We would like to thank NSF for the generous funding, all the participating faculty in this grant ( Drs. Sauncy, Sieben, Ruane, Davis, Jonas, Hijazi, Owenby ) and our colleagues at Texas Lutheran University for their support

References 1. Maureen Lage, Glenn Platt, Michael Treglia (2000), Inverting the Classroom: A gateway to Creating an Inclusive Learning Environment, Journal of Economic Education 2. A. Kaw, M. Hess (2007). Comparing Effectiveness of Instructional Delivery Modalities in an Engineering Course Autar Kaw and Melinda Hess, International Journal of Engineering Education, Vol. 23, No. 3, pp. 508-516 3. Bergmann, J., & Sams, A. (2012). Flip your classroom: reach every student in every class every day. Washington, DC: International Society for Technology in Education 4. Clive Thompson (15 Jul 2011), "How Khan Academy is Changing the Rules of Education", Wired 5. Reza Abbasian & John Sieben“Creating an Inverted Classroom”, ICTCM , Boston, MA, March 2013. 6. Reza Abbasian & John Sieben“Inverted Classrooms: the What, the Why, the How”, Annual MAA-Texas Section, Lubbock, TX, April 2013. 7. Reza Abbasian and John Sieben” Inverted classroms: Videos, Sofware, Hardware, Class Activities and Best Practices”