1. Teaching & Learning Research in the Era of Big Data Rebecca Petersen, Sr. Research Manager, HarvardX

2. Or: Rebecca’s collected thoughts on Research & MOOCs

3. Hi. We’re HX Research http://harvardx.harvard.edu/research

4. We make sense of large data sets generated from open online learning environments

5. We do this through…. Producing rigorous scholarship in the field of learning and analytics Create tools such as Insights, and analytics dashboards for faculty and course teams Provide consultative support to HX faculty and course teams on data interpretation and experimental design Match researchers and course teams to work together

6. MOOC Big Data Hype

7. HarvardX & MITx 2013-2014* 54 Courses 2.1MM reg 1.1MM “viewers” ~40K reg/course * Year 2 Benchmark data of all courses from 2012-September 21, 2014

8. This is a talk about building a research community in the era of big data in teaching and learning

9. The day-to-day work http://harvardx.harvard.edu/research Maven of Research Administration The Data Guy: Gets the Data clean and useable Amazing Post Docs who analyze the data and run studies

10. HarvardX Research Community HarvardX Faculty HarvardX Course Teams * Collaborate Evaluate Inspire Serve *

11. Our early day work is very much about sensemaking and wayfinding with the data through secondary analysis Who are the learners? How do they spend their time in the course? Why are they here?

12. Our Study Focus Areas Motivation Commitment Background Learning conditions

13. We also navigate the day-to-day tension of ambitious scope and ideals with course production realities

14. Research Vignettes Towards Research Enablement Discussion

15. Evidence from MOOCs supporting instructional #advancedcommonsense

17. A Cascade of Completion Rates: Years 1 and 2, Across 63 Courses Of survey responders who “do it all” 25% say that they are not interested in certification 50% say that are or have been teachers 20% say that are or have been teachers in this subject 74% have a Bachelor’s degree The median age is 32.

19. In the MOOC world: Small percentages still represent big numbers Carefully choose your denominator Practice meticulous methodology and know the literature

20. JusticeX Forum Experiment Careful Methodology

21. Choose your denominator Registrants v. participants when factoring completion

22. Toward Research Enablement: Policy, Protocol and Learner Data

23. Facilitating Collaboration

24. Legend Research Enablement: Proposal Decision Making Process 24 Proposal submitted Is it possible using Hx platform? Is there available capacity? Will it require substantial Hx resources? Can some version be done using existing data? Provide access to data (or other support) Aligned with priorities? Can the research be aligned with priorities? Execute! Connect to other resources, try to “schedule” Not aligned with priorities Not feasible Yes No #2#3 #1 Pending IRB approval The Research Committee, or a subcommittee, will review unsuccessful proposals to inform development of the edX platform (“not feasible”) as well as research priorities (“not aligned with priorities”)

25. On Privacy and Policy http://queue.acm.org/detail.cfm?id=2661641

28. http://chronicle.com/article/Are-MOOC-Takers-Students-/150325/

29. How does your individual school treat MOOC learner data? How knowledgeable is your IRB about open online learning data? Will all of the research you do be scholarly (for publication) or is it more closely aligned with Institutional Research (IR)

30. The Future

32. Onward!