The Effect of Teaching on Student Learning in the Onsite and MOOC Version of the Nonprofit Governance Course June 1, 2016 Research Presentation 2016 CIT Conference Yvonne D. Harrison, PhD, University at Albany, SUNY Vic Murray, PhD, University of Victoria Alena Rodick, MS, Empire State College LONGTIDUDINAL

Overview Research Objective Overview of the Course and Teaching and Learning Methods Applied Learning Theory and Hypothesis Method and Sample Results and Observations Preliminary Conclusions and Next Steps

Research Objective Grants to transform an onsite course into a MOOC Research Objective: To evaluate the effect of the course pedagogy on student learning in the onsite and MOOC versions of the course.

Course Overview Onsite: Nonprofit Governance 17 week semester-based graduate level course January 2015; met weekly Online: The Governance of Nonprofit Organizations MOOC 17 week semester-based course on Coursera’s legacy platform January 2015; launched weekly content and engagement structure

Course Structure 1 Week 3, 5 Week Modules in 2 Weeks 15 Week Course Preview Week Module 1 Module 2 Module 3 Final Project 1 Week 3, 5 Week Modules in 15 Week Course 2 Weeks

Instructional Design Module Week 1 Week 2 Week 3 Week 4 Week 5 Content & Formative Assessment Week 2 Week 3 Week 4 Week 5 Summative Assessment Instructor Lectures Expert Interviews Case Study Quiz Journal Articles Open Textbook, Purchased textbooks Additional resources Discussion Forums Lecture Commentary Take 2 Tests and Submit Assignment Semester Project Discuss Tests in Forum/ Peer Review Assignments Take Same Tests Again/Receive Assignment

Assessment of Learning Formative—3 weeks Quizzes, written responses to cases, and lecture commentaries not factored into grade Summative—2 weeks Team Based Learning—knowledge testing process that mirrors how board members share information and make decisions Academic Service Learning—assessment process designed to resolve real world governance problems and develop leadership competency

Implementation Onsite Course Online Course TBL ASL TBL ASL Individual knowledge and application exercise tests Same tests in open forum Delayed feedback (1 week) ASL Work with real board or fictional case to assess performance Three written reflections and final report Peer Reviewed and Graded TBL Individual knowledge and application exercise tests Same tests in small group Immediate feedback ASL Work with real board to assess performance Three written reflections and final report Instructor Reviewed and Graded

Learning Theory and Hypothesis Methods integrated into course design to move students along Bloom’s Taxonomy of critical thinking (lower to higher order thinking) We expect learning in the onsite and online versions of the course Null Operational Hypothesis: There is no relationship between course delivery type and student learning

Measures Academic Performance Course Design and Learning Process Pre and Post RAT test scores (individual and group) BEAR and Final Report assignment grades Course Design and Learning Process Perceived value of the teaching methods applied, effectiveness of course design and instructional team in helping students meet learning outcomes, satisfaction with course, and motivation to take additional courses

Methods and Sample Methods Sample Machine and Instructor grading of tests Instructor and peer review of assignments TA observations of TBL and ASL learning Online course evaluation survey Sample 2015 Spring Semester Performance Online (n = 68); Onsite (n = 10) Learning Process Online (n = 259); Onsite (n = 9)

Results: TBL Performance Onsite Online 1-10 M/C Test Knowledge Mean Application Individual RATs Pre Post Module 1 7.10 7.9 6.13 8.70* 6.25 8.76* Module 2 6.00 6.90 4.27 7.22* 6.22 7.80* Module 3 7.30 5.61 9.19* 6.51 8.38* Group RATs SG1 n = 4 SG2 RAT Forum 9* 10* 8 9 * = significant increase beyond p<.05

Student Perception Likert 1-5 Scale Onsite % (N) Online How valuable were the Readiness Assurance Tests to the learning process? M = 3.63, SD = 1.06 M = 3.95, SD = 1.02 Not at All 3 (5) Slightly 25 (2) 6.7 (11) Moderately 16.5 (27) Very 62.5 (5) 40.2 (66) Extremely 12.5 (1) 33.5 (55) No significant perceptual difference between student groups “Required me to review material more thoroughly, which helped retention” “Forced me to look over my notes; discussing content with peers/arguing points of view helpful” “Helped me to realize what I was unsure of after I completed the readings and then we would usually go over those topics as a class” Onsite “Required me to review material more thoroughly, which helped retention” MOOC evaluation: “Forced me to look over my notes; discussing content with peers/arguing points of view helpful” “Helped me to realize what I was unsure of after I completed the readings and then we would usually go over those topics as a class”

TA Observations Team-Based Learning: Process mirrored onsite class small group behavior Student information sharing--content and evidence in similar ways Students challenging questions or identifying how they misread question or did not have all the information Consensus reached in discussions resulting in correct answers surfacing or appeals

Results: ASL Performance Onsite Online Rubric Max 10 Instructor Review (1) Mean Peer Reviews (3) Board Effectiveness Assessment Reflection (BEAR) Evaluation Rubric with Feedback Peer Review Grading and Feedback Guidelines Module 1 Assignment 9.54* 8.61 Module 2 Assignment 8.66* 7.85 Module 3 Assignment 8.98* 8.14 Semester Project Final Report 9.00 8.90 * = significant difference beyond p<.05

Student Perception Likert 1-5 Scale Onsite % (N) Online How valuable was it to work with a real board? M = 4.62, SD = .51 M = 4.36, SD = .98 How valuable was it to work with information from a board? NA 4.13, SD = .78 Real Board Board Info Not at All 2.9 (2) Slightly 4.3 (3) 3.4 (3) Moderately 5.8 (4) 13.8 (12) Very 37.5 (3) 27.5 (19) 48.3 (42) Extremely 62.5 (5) 59.4 (41) 34.5 (3) No significant perceptual difference between student groups Onsite “Required me to review material more thoroughly, which helped retention” MOOC evaluation: “Forced me to look over my notes; discussing content with peers/arguing points of view helpful” “Helped me to realize what I was unsure of after I completed the readings and then we would usually go over those topics as a class”

TA Observations Academic Service Learning Assignment and semester project questions were resolved as students became more familiar with the course and comfortable communicating in the platforms Students appreciated the opportunity to appeal peer reviewed grades and have written work reviewed by a TA Very few students appealed grades. Of those that did, most had passing grades Learning process mirrored onsite course ASL worked in this MOOC Real and demo boards drawing from research Scaffold peer-reviewed assignment resulting in self-awareness and feedback to improve final report

Results: Learning Process 1-5 Likert Scales Onsite N = 8 Online N = 164 Course Design Development of Leadership Competency 4.12 .35 3.90 1.12 Instructional Team in Facilitating Leadership Competency Dev. 4.37 .91 3.98 1.18 Satisfaction with Learning Experience 4.50 .53 2.89*** 1.85 Inspire nonprofit education 4.25 .70 3.72 .99 ***Significant Difference Between Groups Beyond p<.001 level

TA Observations Course Forums and Blogs Engagement. All commenters engaged in a respectful manner, and only one of the over 100 posts was not constructive. Deep reflection. A vast majority of comments demonstrated that students were thinking critically about the issues, considering implications, and offering their own perspective. Connecting learning to experiences. Many students work for NPOs and/or serve on boards. Many shared their experiences and related them to the concepts and ideas in the course. Consistent with onsite course engagement

Preliminary Conclusions Evidence of learning in both courses (i.e. increases in test scores and grades on written assignments) Results are consistent with TA observations of learning process Further research Content analysis of written assignments and final report Exploring comments and suggestions for improving the learning process Some methods may return more value Working with a real board versus fictional Online group perceived TBL had more value on learning than onsite group though not statistically significant Online provides a safer space to fail? No accountability

Next Steps in the Research Compare motivation across onsite and online groups Examine the impact of learning on student, board, and organization performance through the nonprofit governance sponsored research Align Coursera data with pre and post course survey data

Thank You! Questions