1. Karen Swan, Scott Day, Leonard Bogle & Traci van Prooyen University of Illinois at Springfield Exploring MOOC Pedagogies

2. American Council on Education’s (ACE) College Credit Recommendation Service is evaluating MOOCs for content and approving them for specific college credits if students pass ACE CREDIT tests ($150). ACE CREDIT evaluates MOOCs for their content; we are reviewing MOOCs for their pedagogical approaches. CONTEXT

3. We developed an instrument to characterize the pedagogical approaches taken in MOOCs -- modeled on a similar instrument proposed by Tom Reeves (1996) for describing the pedagogical approaches found in computer-based instruction -- “ Pedagogical dimensions are concerned with those aspects of design and implementation... that directly affect learning.” AMP (Assessing MOOC Pedagogy) tool

4. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) AMP TOOL

5. METHODS By Fall 2013, ACE evaluated and approved 5 Coursera, 1 EdX, and 7 Udacity MOOCs -- College Algebra, BioElectricity, Genetics, Pre-Calculus, and Single Variable Calculus, Circuits, Introduction to Artificial Intelligence, Introduction to Computer Science, Introduction to Physics, Introduction to Statistics, Introduction to Parallel Programming, 3-D Modeling and HTML 5 Game Development We reviewed the same 13 MOOCs and four other Coursera MOOCs that covered non-STEM courses.

6. METHODS We iteratively revised the AMP tool as we tested its efficacy on the approved MOOCs, first by reducing the scale of each dimension from ten to five, and then by establishing specific criteria for each dimension, to improve inter-rater reliability. We reviewed all the approved MOOCs using the revised instrument. Four researchers reviewed each MOOCs. Inter-rater reliability was >80% on all the reviews before meeting to achieve consensus on the ratings.

7. The approved COURSERA courses followed a format that resembled the traditional lecture/text/testing routine of traditional university courses, including cohorts and weekly hard deadlines.

8. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) college algebra

9. bioelectricity pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

10. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) genetics & evolution

11. pre-calculus pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

12. single variable calculus pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

13. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) college algebra bioelectricitygenetics & evolution pre-calculus single variable calculus

14. circuits pedagogical categoryratings epistemology (objectivist)12345(constructivist) role of teacher (teacher- centered) 12345(student-centered) focus of activities (convergent)12345(divergent) structure (less structure)12345(more structure) approach to content (concrete)12345(abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported)12345(integral) accommodation of individual differences (unsupported)12345(multifaceted) activities/assessment (artificial)12345(authentic) user role (passive)12345(generative)

15. The Coursera courses approved for ACE credit, as well as EdX’s Circuits, tended to be objectivist, teacher- centered, convergent, highly structured, more abstract than concrete, with minimal feedback. The courses tended to fall somewhere in the middle between supporting and not supporting cooperative learning, accommodating and not accommodating individual differences, artificial and authentic activities/assessment, and passive and active user roles.

16. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) college algebra bioelectricity genetics & evolution pre-calculus single variable calculus circuits

17. UDACITY courses followed a format that resembled programmed learning, including self-pacing and embedded quizzes.

18. interactive 3-d graphics pedagogical categoryratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 1 2345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

19. html 5 graphics pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

20. pedagogical categoryratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student-centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) introduction to artificial intelligence

21. introduction to computer science pedagogical categoryratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student-centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

22. introduction to parallel processing pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

23. introduction to physics pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

24. introduction to statistics pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

25. Udacity courses tended to be neither objectivist nor constructivist, slightly less teacher-centered and convergent than Coursera courses, highly structured, halfway between abstract and concrete, with immediate, clear, and constructive feedback. They also tended not to support cooperative learning, but because of their self-directed approach were quite accommodating of individual differences. Udacity also made an effort to develop authentic activities, and perhaps because of the large number of CS courses, supported a more generative user role than the ACE- approved courses from Coursera and EdX.

26. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) interactive 3-d graphics HTML 5 graphics intro to artificial intelligence intro to computer science intro to parallel processing intro to physics intro to statisics

27. EDX COURSERAUDACITY epistemology 1.0 2.4 role of teacher 1.0 1.42.0 focus of activities 1.0 1.9 structure 5.0 4.9 approach to content 4.0 3.63.0 feedback 3.0 2.04.3 cooperative learning 2.0 2.82.1 accommodation of individual differences 2.0 2.63.0 activities/assessment 1.0 2.63.3 user role 2.0 3.03.1 average ratings across platforms

28. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) Coursera EdX Udacity comparisons of ACE approved courses across platforms

29. art & inquiry pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative)

30. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) comic books & graphic novels

31. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) jazz improvisation

32. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) the music of the Beatles

33. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) art & inquiry comic books & graphic novels jazz improvisation the music of the Beatles non-stem

34. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) art & inquiry comic books & graphic novels jazz improvisation non-stem

35. STEMBEATLESNON-STEM epistemology 1.0 4.7 role of teacher 1.41.03.0 focus of activities 1.0 4.3 structure 5.04.03.7 approach to content 3.62.02.3 feedback 2.0 3.7 cooperative learning 2.82.03.0 accommodation of individual differences 2.62.02.7 activities/assessment 2.61.04.7 user role3.02.04.3 average ratings across domains

36. pedagogical category ratings epistemology (objectivist) 12345 (constructivist) role of teacher (teacher- centered) 12345 (student- centered) focus of activities (convergent) 12345 (divergent) structure (less structure) 12345 (more structure) approach to content (concrete) 12345 (abstract) feedback (infrequent, unclear) 12345 (frequent, constructive) cooperative learning (unsupported) 12345 (integral) accommodation of individual differences (unsupported) 12345 (multifaceted) activities/assessment (artificial) 12345 (authentic) user role (passive) 12345 (generative) STEM Beatles non-STEM

37. (objectivist) 12345 (constructivist) (teacher- centered) 12345 (student- centered) (convergent) 12345 (divergent) (less structure) 12345 (more structure) (concrete) 12345 (abstract) (infrequent, unclear) 12345 (frequent, constructive) (unsupported) 12345 (integral) (unsupported) 12345 (multifaceted) (artificial) 12345 (authentic) (passive) 12345 (generative) two metaphors for learning (Sfard, 1998) acquisitionparticipation

38. CONCLUSIONS AMP tool can effectively characterize MOOC pedagogies At least two distinct pedagogical approaches can be distinguished Future research will explore these distinctions, as well as MOOCs offered on different platforms & differently organized Future research will try to link pedagogical to learning outcomes

39. lbogl1@uis.edu sday1@uis.edu kswan4@uis.edu tvanp2@uis.edu