Smart Subtitles with Personalized Quizzes for Vocabulary Building CS2610: Course Project Fall 2017 Ahmed Magooda, Spencer Gray, Kadie Clancy

Problem and Background Language learners often use subtitled videos to help them learn Standard subtitles are geared more toward comprehension than vocabulary learning Translations are nonlinear and are provided only for phrases, not vocabulary terms Previous work to address this problem: Smart Subtitles (Kovacs and Miller) Automatically generated from common video sources such as subtitled DVDs. Provide features such as vocabulary definitions on hover, and dialog-based video navigation

Further Improvement on Smart Subtitles Addition of Personalized Quizzes We hypothesize that more exposure to vocabulary terms that users were unsure of will increase their retention rate of those terms Personalized Quizzes: quizzing the users on terms that they are unsure of Periodically: After a certain interval of time, quizzes will be presented This also helps ensures that the user is actively watching the video

Smart Subtitles with Personalized Quizzes Assuming that when a user clicks on a word to get the translation and synonyms that they do not actively know that term. Quiz Content: words that the user has clicked on while watching the video, plus other “important” words from the video using part of speech tagging Quiz Format: varied formats to determine which style works best with users in study Matching Multiple Choice

Demo

User Study Study Participants 5 participants (2 Female, 3 male) All spanish learners, varied levels 4/5 participants have used videos to study Spanish in the past Study Format Debriefed on purpose of experiment and how to use video viewer Pre and post test to gather data on new terms learned New terms learned: words specific to each video segment in study, can infer that new words were learned in a specific video

Experimental Design Independent Variable: Quiz Type {none, matching, multiple choice} Dependent Variable: number of new words learned Each participant is shown 3 different video segments (5 min each) in randomized order, each utilizing a different quiz type. Each quiz type showed two quizzes per video segment.

Experimental Design Data generated via pre and post tests Same test given before the study and after the study (outside the system) Multiple choice vocabulary test Each word in the test was unique to a video Compare the scores and we consider an improvement in score as a new word learned Words learned per quiz type Users were not shown correct answers after the pre test Users did not know they were going to take the same test again as a post test Also asked participants which quiz type they prefered Without knowing score

Results Results were not significant - which we expected Problems with guessing in the pre and post test if they don’t actually know a word Some scored better in the pre test than in the post test Participants commented that the videos were too difficult for beginners

Future Work Improved user study Better method to gauge how many words have been learned Short-term versus long-term memory Equivalent fluency levels with level appropriate video segments Incorporate more interesting quiz types into our system Improve translation and stemming Improve content of quizzes Both static and dynamic words were chosen pseudo-randomly Need to discover an algorithm to pick words based on various metrics

References Kovacs, Geza. “Smart Subtitles for Language Learning”. CHI 2013 Extended Abstracts.

Thanks Questions ?