Weakly Supervised Models of Aspect-Sentiment for Online Course Discussion Forums ARTI RAMESH SHACHI H. KUMAR JAMES FOULDS LISE GETOOR

2 Massive: attracts thousands of participants Open: open access, content, and assessment Online: hosted online by education companies in partnership with top universities

3 Classroom – Face-to-face interaction between instructor and students MOOCs MOOC Discussion Forums – Primary means of interaction between instructor and students Large number of students, posts: Hard to monitor manually Posts discuss problems in course - course material, errors, feedback

4 Example MOOC Posts MOOC PostFine-grained Topic The video is very choppy. Can somebody fix this? Lecture-Video Will subtitles be made available for the lectures for this week? I liked the transcripts from last week. Lecture-Subtitles Will everyone get a certificate or only people in the signature track? Certificate When is quiz 4 due?Quiz-Deadlines

5 Predicting fine-grained problems: Challenges Labeled data hard to obtain – 5-10% posts contain problems – Privacy concerns around data sharing – Problems differ across courses Unsupervised/weakly supervised approaches desirable – System not fine-tuned to one course, but can adapt across courses

6 Related Work Aspect-sentiment in Online Reviews Semi-supervised generative model, with seed words to identify aspect clusters [Mukherjee et al., 2012] Unsupervised Aspect-Sentiment Model for Online Reviews [Brody et al., 2012] Hierarchical Aspect-Sentiment Model for Online Reviews [Kim et al. 2013] MOOCs Predicting Instructor Intervention in MOOC Forums [Chaturvedi et al., 2014]

7 SeededLDA for MOOC Forums SeededLDA Guide topic discovery by specifying representative seed words seededLDA uses seeds to bias topic-word and word- document distributions seededLDA gathers words related to seed words SeededLDA for MOOCs Many classes but a common set of seed words Seed words for MOOCs from syllabus and forums Jagarlamudi et al. 2010

8 Hinge-loss Markov Random Fields & Probabilistic Soft Logic Hinge-loss Markov Random Fields (HL-MRFs) – Logic-based MRFs that can reason about both discrete and continuous graph data scalably and accurately – Efficient Inference: convex optimization in continuous space Probabilistic Soft Logic (PSL) – Templating language for HL-MRFs – Weighted logical rules to model dependencies – Continuous variables in [0,1] Bach et al. 2012

9 Analogous to predicting aspect-sentiment in online reviews Aspect hierarchy connecting course elements HL-MRF framework – Combining different features – Encoding coarse-to-fine aspect hierarchy – Encoding dependencies between aspect and sentiment Jointly modeling aspect and sentiment Predicting fine-grained problems and sentiment: Joint Prediction Problem

10 Our Contributions Identify fine-grained aspects in online courses Extract course-specific features from posts using SeededLDA Construct coarse-to-fine aspect hierarchy to model aspect dependencies Construct weakly-supervised joint model for aspect-sentiment using HL-MRFs Validate system using crowdsourced posts sampled from 12 courses

11 MOOC Aspect-Sentiment Models: SeededLDA LECTURE: lecture, video, download, transcript, slide, note QUIZ: quiz, assignment, question, midterm, exam, submission CERTIFICATE: certificate, score, statement, signature SOCIAL: name, course, introduction, study, group Coarse Aspect seeds Sentiment seeds POSITIVE: interest, exciting, thank, great, happy, glad, enjoy NEGATIVE: problem, difficult, error, issue, unable, misunderstand NEUTRAL: coursera, class, hello, everyone, greet, name

12 SeededLDA Model Fine Aspect seeds LECTURE-VIDEO: video, problem, download, play, player, LECTURE-AUDIO: volume, low, headphone, sound, audio, hear LECTURE-LECTURER: professor, fast, speak, pace, follow, speed LECTURE-SUBTITLES: transcript, subtitle, slide, note, lecture, LECTURE-CONTENT: typo, error, mistake, wrong, right, incorrect QUIZ-CONTENT: question, challenge, difficult, understand, typo QUIZ-SUBMISSION: submission, submit, quiz, error, unable, resubmit QUIZ-GRADING: answer, question, answer, grade, assignment, quiz QUIZ-DEADLINE: due, deadline, miss, extend, late

13 PSL-Joint: Combining Features SeededLDA score for fine aspect and coarse aspect to predict fine aspect of post P

14 PSL-Joint: Combining Features SeededLDA score for sentiment and fine aspect to predict fine aspect

15 PSL-Joint: Encoding Dependencies Dependency between coarse aspect and fine aspect

16 PSL-Joint: Encoding Dependencies Dependency between sentiment and fine aspect

17 Experimental Evaluation ModelLectureQuizCertificateSocial SeededLDA PSL-Joint ModelPositiveNegativeNeutral SeededLDA PSL-Joint SeededLDA and PSL-Joint for sentiment F-1 scores for SeededLDA and PSL-Joint for coarse aspects

18 Experimental Evaluation ModelLectureQuizCertificateSocial SeededLDA PSL-Joint ModelPositiveNegativeNeutral SeededLDA PSL-Joint SeededLDA and PSL-Joint for coarse aspects SeededLDA and PSL-Joint for sentiment PSL-Joint outperforms SeededLDA for most coarse aspects and sentiment

19 Experimental Evaluation ModelContentVideoAudioLecturerSubtitles SeededLDA PSL-Joint ModelContentSubmissionDeadlinesGrading SeededLDA PSL-Joint Fine-grained aspects under coarse aspect lecture Fine-grained aspects under coarse aspect quiz

20 Experimental Evaluation ModelContentVideoAudioLecturerSubtitles SeededLDA PSL-Joint ModelContentSubmissionDeadlinesGrading SeededLDA PSL-Joint Fine-grained aspects under coarse aspect “lecture” Fine-grained aspects under coarse aspect “quiz” PSL-Joint distinguishes between lecture- content and quiz- content

21 Experimental Evaluation ModelContentVideoAudioLecturerSubtitles SeededLDA PSL-Joint ModelContentSubmissionDeadlinesGrading SeededLDA PSL-Joint Fine-grained aspects under coarse aspect “lecture” Fine-grained aspects under coarse aspect “quiz” Significant improvement in scores for lecture-lecturer and quiz-deadlines

22 Interpreting PSL-Joint Predictions “There is a typo or other mistake in the assignment instructions (e.g. essential information omitted).” SeededLDA Prediction: Lecture-content PSL-Joint Prediction: Quiz-content “Thanks for the suggestion about downloading the video and referring to the subtitles. The audio is barely audible, even when the volume is set to 100%” SeededLDA Prediction: Lecture-subtitles PSL-Joint Prediction: Lecture-audio

23 Conclusion: Fine-grained aspect- sentiment in MOOC forums Automatically detecting problems in forum posts useful for instructors Weakly supervised probabilistic framework to automatically detect aspect and sentiment in online courses – SeededLDA and PSL-Joint models as means to encode domain information and predict aspect and sentiment PSL-Joint significantly outperforms SeededLDA for many fine aspects, coarse aspects, and sentiment – Structural dependencies among aspect and sentiment helps in prediction