Analysis of Forum Discourse in Large Online Classes David Joyner Associate Director of Student Experience College of Computer, Georgia Tech

Background

Online Master of Science OMSCS Online Master of Science in Computer Science The Georgia Tech Online Master of Science in Computer Science program began as a collaboration between Georgia Tech, Udacity, and AT&T.

OMSCS Spring 2018: 6500 students; 8900 "seats" Two years in, the program enrolls 3400 students from every state and over 100 countries.

Student-Initiated Discussions Expert-Level Peer Review Extended Self-Documenting Student-Student Feedback De Facto Teaching Assistants Parallel Organic Student-Instructor Communication

20% instructor or teaching assistant; 80% student 10,000 × 207 2,070,000 posts per class classes offered total posts 20% instructor or teaching assistant; 80% student

Project Themes Automated Q&A and Answer Curation Gender and Social Dynamics Learner Sentiment Analysis Predicting Student Success and Retention

Automated Q&A and Answer Curation

Study #1: Jill Watson and Noelle King By Question Method Findings Design & Intelligence Lab Does there exist a typology of common question types an AI can address? Human-annotation of forum questions There exist general top-level question categories (technical support, social, rules and procedures), but next-level questions are highly class-specific.

Study #2: Clustering Similar Questions By Question Method Findings Chapman Siu Can we intelligently detect and cluster similar questions for student support? Latent Dirichlet Allocation, Latent Semantic Indexing, Matrix Cosine Similarity Many common student questions have individual wrinkles that make automated clustering difficult; nonetheless, there is opportunity.

Study #3: Wiki Curation from Forums By Question Method Findings Jay Haksar Can FAQ-style wikis be automatically generated from class discussion forums? Use NLTK, MoinMoin, and Python to auto- generate wikis from Piazza forums A rough wiki can be generated based on past forum usage, offloading portions of the responsibility onto the system and leaving polish and error-checking to course staff.

Gender and Social Dynamics

Study #4: Social Engagement Traits By Question Method Findings Sohyun Sung How do social relationships and behaviors manifest in online courses? Statistical analysis of activity in instructor-, TA-, and student-led threads On average, instructor- and TA-led threads garner the most passive and active engagement, but individual students may also garner similar activity in their threads.

Study #5: Subgroup Performance in Forums By Question Method Findings Sandra Davis How do gender and professional subgroups differ in forum participation and feelings of belongingness? Analysis of past forums and survey on forum usage Women answer questions less often and post shorter replies than men; former CS majors feel more like they belong; overall forum participation rates are low, suggesting a significant skew.

Learner Sentiment Analysis

Study #6: Identifying Struggling Students By Question Method Findings Damian Durruty How can we reliably detect individual student sentiment in large online courses? Apply IBM Watson's Tone Analyzer to forum data Tone Analyzer can identify struggling students based on forum patterns, but is not well-suited to evaluating overall classroom sentiment due to sampling bias and a bias toward negativity.

Study #7: Overall Classroom Sentiment By Question Method Findings Michael Schubert How can we reliably detect student sentiment in large online courses? Apply Microsoft Text API to forum data Student sentiment can be automatically extracted, and Microsoft Text API potentially limits negative biases; however, sampling bias is still present.

Predicting Student Success and Retention

Study #8: Predicting Student Success By Question Method Findings Nate Payne & Daniel Hegberg Can forum participation data help predict success rates? Multiple linear regression on forum participation rates (and other data) Course success can be predicted in part by forum participation, as well as prior programming experience, prior online learning experience, prior education level, and video-watching pace.

Study #9: Predicting Student Retention By Question Method Findings Mason Gallo Can forum participation data help predict retention rates? Multiple linear regression on forum participation rates (and other data) Course retention can be predicted in part by forum participation, as well as current course grade and student age. Gender, geography, and prior experience were not strong predictors.

Recap Automated Q&A and Answer Curation Gender and Social Dynamics Learner Sentiment Analysis Predicting Student Success and Retention

Questions? For more: lucylabs.gatech.edu DavidJoyner.net Contact me: david.joyner@gatech.edu