Using Short-Answer Format Questions for an English Grammar Tutoring System Conceptualization & Research Planning Jonggun Gim

Research Objective in one sentence ► To propose a software designed to generate short answer format question and to analyze the answer to assess a student’s English grammar knowledge and plan for further tutoring.

Intelligent Tutoring System ► software packages that either teach or help human tutors to teach closed sets of knowledge to students ► Shortcomings: Less Interactive, Less adaptive, Less Fun ► Advantages  it can remember exactly what each student knows and what should be taught next,  it can employ limitless variation of approach to teach same lesson  Cheaper

Automatic Question Generation ► [Mitkov] What type of verb form has to come after the verb, ‘enjoy’? ① gerund ② infinitive ► [Chen] Complete the sentence by choosing the most appropriate verb form: “I intend ______ you that we cannot approve your application..” ① telling ② to tell ③ told ④ tell

Originality in this research ► Short Answer format Question Generation ► Change the following active voice sentence into a passive one: “All students admire Principal Lee.” ► We believe this approach is particularly efficient in tutoring grammar because an instruction like English grammar requires not only the knowledge but also fluency that can be acquired through practice with varying usage.

Approach ► Five Steps for using short-answer format for English Grammar Tutoring are;  1) defining the set of grammatical rules to be mastered,  2) selecting a grammatical rule to be questioned/practiced,  3) selecting an example sentence,  4) generating a question, and finally,  5) evaluating the answers for generating appropriate advice.

Approach – Grammar Rule set R2Tense a Simple Present tense changes to a Present Continuous tense with changing the verb into appropriate ‘be’ + the verb inflected to a present participle R3 Passive Voice an Active voice sentence changes to passive one with swapped S and O + ‘be’ + past participle + ’by’ R4Interrogation a yes/no question can be constructed from a descriptive sentence with a ‘be’ verb by swapping S and V and changing the period to a question mark. R5Idiom the verb ‘enjoy’ is followed by gerund ► Criteria for defining Subset of English Grammar for the system  Rules that can be easily questioned by short answer format  Rules to be mastered by students who learns English as a second language

Approach – Question Patterns ► Question are made of a Direction and a Target Sentence ► Directions are decided by the rule to be tested (one-to-many)  R2-P2, P2’, R3-P3, R4-P4, ► Target sentences are selected from existing corpus or reliable web documents Direction Target Sentences P2 Change the following sentence to present continuous tense: “I play with my dog.” P3 Change the following active voice sentence to a passive one: “All students admire Principal Lee.” P4 Change the sentence to a yes/no question: “She wore a black sweater yesterday.” P5 Complete the sentence: “I enjoy _____ as a new hobby.”

Approach – Analysis of Student’s Answers ► Students’ knowledge on a grammatical rule is remembered as a probability between 0 and 1, not as a True/False decision ► For a question aimed to test the knowledge of rule A can be used also to test knowledge of rule B by analyzing the student’s answer  “Principal Lee is admiren by all students.”

Functional Requirements ► Tutors shall be able to define Grammatical Rules to be tested and Prerequisite relationship among them ► Tutors shall be able to choose a rule and command the system to generate certain numbers of questions based on the rule and accompanying answer (or answer pattern) ► The system shall be able to parse the students’ answer to:  Tell if the answer is correct or not  Update the knowledge database of the student  To explain the grammatical rule that the student seems not to understand

Design and Implementation ► Grammatical Rule Representation  GraviaX grammar checker (?) ► Target sentence retrieval  Part of Speech analysis – NLTK (Natural Language ToolKit) ► Answer parsing  POS analysis – NLTK  English Semantic Dictionary - WordNet

Evaluation of the approach ► Things to be evaluated from the suggested approach  1) how much the generated questions are similar to the ones generated by human experts  2) how much the provided advices based on the student’s answers are relevant  3) whether the questions are more diverse than the ones generated by human  4) whether questions are more adaptively generated than the ones developed by human

Results ► Score Correlation (chart on the right) ► Survey on the relevance of the advices, adaptiveness & diversity of questions will be collected

Discussion ► Factors that determines the quality of tests will be reevaluated ► Will try to find more factors that determine the quality of tests ► Will try to devise better ways to measure the current factors ► Other Components of Learning English  Short answer format question based Reading Comprehension Test

References ► Mitkov, R. & Ha, L.A. (2003). Computer-Aided Generation of Multiple-Choice Tests. In Proceedings of the HLT- NAACL 2003 Workshop on Building Educational Applications Using Natural Language Processing, Edmonton, Canada, May, p ► Chia-Yin Chen, Hsien-Chin Liou, Jason S. Chang, (2006), FAST – An Automatic Generation for Grammar Tests ► Steven A. Stahl, (1986), Three principals of effective vocabulary instruction. Journal of Reading, 29 ► NLTK (Natural Language ToolKit) documentation,