1. The Effect of Teaching on Student Learning in the Onsite and MOOC Version of the Nonprofit Governance Course June 1, 2016 Research Presentation CIT Conference
Yvonne D. Harrison, PhD, University at Albany, SUNY
Vic Murray, PhD, University of Victoria
Alena Rodick, MS, Empire State College
LONGTIDUDINAL

2. Overview Research Objective
Overview of the Course and Teaching and Learning Methods Applied
Learning Theory and Hypothesis
Method and Sample
Results and Observations
Preliminary Conclusions and Next Steps

3. Research Objective Grants to transform an onsite course into a MOOC
Research Objective: To evaluate the effect of the course pedagogy on student learning in the onsite and MOOC versions of the course.

4. Course Overview Onsite: Nonprofit Governance
17 week semester-based graduate level course
January 2015; met weekly
Online: The Governance of Nonprofit Organizations MOOC
17 week semester-based course on Coursera’s legacy platform
January 2015; launched weekly content and engagement structure

5. Course Structure 1 Week 3, 5 Week Modules in 2 Weeks 15 Week Course
Preview Week
Module 1
Module 2
Module 3
Final Project
1 Week
3, 5 Week Modules in
15 Week Course
2 Weeks

6. Instructional Design Module Week 1 Week 2 Week 3 Week 4 Week 5
Content & Formative Assessment
Week 2
Week 3
Week 4
Week 5
Summative Assessment
Instructor Lectures
Expert Interviews
Case Study
Quiz
Journal Articles
Open Textbook, Purchased textbooks
Additional resources
Discussion Forums
Lecture Commentary
Take 2 Tests and Submit Assignment
Semester Project
Discuss Tests in Forum/ Peer Review Assignments
Take Same Tests Again/Receive Assignment

7. Assessment of Learning
Formative—3 weeks
Quizzes, written responses to cases, and lecture commentaries not factored into grade
Summative—2 weeks
Team Based Learning—knowledge testing process that mirrors how board members share information and make decisions
Academic Service Learning—assessment process designed to resolve real world governance problems and develop leadership competency

8. Implementation Onsite Course Online Course TBL ASL TBL ASL
Individual knowledge and application exercise tests
Same tests in open forum
Delayed feedback (1 week)
ASL
Work with real board or fictional case to assess performance
Three written reflections and final report
Peer Reviewed and Graded
TBL
Individual knowledge and application exercise tests
Same tests in small group
Immediate feedback
ASL
Work with real board to assess performance
Three written reflections and final report
Instructor Reviewed and Graded

9. Learning Theory and Hypothesis
Methods integrated into course design to move students along Bloom’s Taxonomy of critical thinking (lower to higher order thinking)
We expect learning in the onsite and online versions of the course
Null Operational Hypothesis: There is no relationship between course delivery type and student learning

10. Measures Academic Performance Course Design and Learning Process
Pre and Post RAT test scores (individual and group)
BEAR and Final Report assignment grades
Course Design and Learning Process
Perceived value of the teaching methods applied, effectiveness of course design and instructional team in helping students meet learning outcomes, satisfaction with course, and motivation to take additional courses

11. Methods and Sample Methods Sample
Machine and Instructor grading of tests
Instructor and peer review of assignments
TA observations of TBL and ASL learning
Online course evaluation survey
Sample
2015 Spring Semester Performance
Online (n = 68); Onsite (n = 10)
Learning Process
Online (n = 259); Onsite (n = 9)

12. Results: TBL Performance
Onsite
Online
1-10 M/C Test
Knowledge
Mean
Application
Individual RATs
Pre
Post
Module 1
7.10
7.9
6.13
8.70*
6.25
8.76*
Module 2
6.00
6.90
4.27
7.22*
6.22
7.80*
Module 3
7.30
5.61
9.19*
6.51
8.38*
Group RATs
SG1
n = 4
SG2
RAT Forum 9*
10* 8 9
* = significant increase beyond p<.05

13. Student Perception Likert 1-5 Scale Onsite % (N) Online
How valuable were the Readiness Assurance Tests to the learning process?
M = 3.63, SD = 1.06
M = 3.95, SD = 1.02
Not at All
3 (5)
Slightly
25 (2)
6.7 (11)
Moderately
16.5 (27)
Very
62.5 (5)
40.2 (66)
Extremely
12.5 (1)
33.5 (55)
No significant perceptual difference between student groups
“Required me to review material more thoroughly, which helped retention”
“Forced me to look over my notes; discussing content with peers/arguing points of view helpful”
“Helped me to realize what I was unsure of after I completed the readings and then we would usually go over those topics as a class”
Onsite
“Required me to review material more thoroughly, which helped retention”
MOOC evaluation:
“Forced me to look over my notes; discussing content with peers/arguing points of view helpful”
“Helped me to realize what I was unsure of after I completed the readings and then we would usually go over those topics as a class”

14. TA Observations Team-Based Learning:
Process mirrored onsite class small group behavior
Student information sharing--content and evidence in similar ways
Students challenging questions or identifying how they misread question or did not have all the information
Consensus reached in discussions resulting in correct answers surfacing or appeals

15. Results: ASL Performance
Onsite
Online
Rubric Max 10
Instructor Review (1)
Mean
Peer Reviews (3)
Board Effectiveness Assessment Reflection (BEAR)
Evaluation Rubric with
Feedback
Peer Review Grading and Feedback Guidelines
Module 1 Assignment
9.54*
8.61
Module 2 Assignment
8.66*
7.85
Module 3 Assignment
8.98*
8.14
Semester Project
Final Report
9.00
8.90
* = significant difference beyond p<.05

16. Student Perception Likert 1-5 Scale Onsite % (N) Online
How valuable was it to work with a real board?
M = 4.62, SD = .51
M = 4.36, SD = .98
How valuable was it to work with information from a board? NA
4.13, SD = .78
Real Board
Board Info
Not at All
2.9 (2)
Slightly
4.3 (3)
3.4 (3)
Moderately
5.8 (4)
13.8 (12)
Very
37.5 (3)
27.5 (19)
48.3 (42)
Extremely
62.5 (5)
59.4 (41)
34.5 (3)
No significant perceptual difference between student groups
Onsite
“Required me to review material more thoroughly, which helped retention”
MOOC evaluation:
“Forced me to look over my notes; discussing content with peers/arguing points of view helpful”
“Helped me to realize what I was unsure of after I completed the readings and then we would usually go over those topics as a class”

17. TA Observations Academic Service Learning
Assignment and semester project questions were resolved as students became more familiar with the course and comfortable communicating in the platforms
Students appreciated the opportunity to appeal peer reviewed grades and have written work reviewed by a TA
Very few students appealed grades. Of those that did, most had passing grades
Learning process mirrored onsite course
ASL worked in this MOOC
Real and demo boards drawing from research
Scaffold peer-reviewed assignment resulting in self-awareness and feedback to improve final report

18. Results: Learning Process
1-5 Likert Scales
Onsite
N = 8
Online
N = 164
Course Design
Development of Leadership Competency
4.12
.35
3.90
1.12
Instructional Team in Facilitating Leadership Competency Dev.
4.37
.91
3.98
1.18
Satisfaction with Learning Experience
4.50
.53
2.89***
1.85
Inspire nonprofit education
4.25
.70
3.72
.99
***Significant Difference Between Groups Beyond p<.001 level

19. TA Observations Course Forums and Blogs
Engagement. All commenters engaged in a respectful manner, and only one of the over 100 posts was not constructive.
Deep reflection. A vast majority of comments demonstrated that students were thinking critically about the issues, considering implications, and offering their own perspective.
Connecting learning to experiences. Many students work for NPOs and/or serve on boards. Many shared their experiences and related them to the concepts and ideas in the course.
Consistent with onsite course engagement

20. Preliminary Conclusions
Evidence of learning in both courses (i.e. increases in test scores and grades on written assignments)
Results are consistent with TA observations of learning process
Further research
Content analysis of written assignments and final report
Exploring comments and suggestions for improving the learning process
Some methods may return more value
Working with a real board versus fictional
Online group perceived TBL had more value on learning than onsite group though not statistically significant
Online provides a safer space to fail?
No accountability

21. Next Steps in the Research
Compare motivation across onsite and online groups
Examine the impact of learning on student, board, and organization performance through the nonprofit governance sponsored research
Align Coursera data with pre and post course survey data

22. Thank You!
Questions