Speech Analysing Component in Automatic Tutoring Systems Presentation by Doris Diedrich and Benjamin Kempe

Survey A Short Introduction to LSA (Latent Semantic Analysis) What LSA does How effective is LSA How is an LSA representation created Some Caveats Auto Tutor Course in computer literacy at the U. of Memphis Interface Simulating human tutoring use of LSA working system Why2 Why Why2 ? Evaluation: KCDs Statistic and Symbolic Representations

LSA (Latent Semantic Analysis) " Motivation: An automatic tutors „understanding“ of what a student says is limited => reduction on topics/systems where multiple choice or other simple selections make sense. " LSA is used to analyse students natural language answers: " what topics are hit  how good is the answer

In Detail LSA is used to analyse texts on a topic, to create a statistical representation of keywords with word occurrences (=text). When a student types in a text, LSA computes a representation of that students text (students answer) which can be compared to the reference text's representation. So natural language answers are evaluated and if needed, it can be found out which topics are lacking in the answer text.

In an example on texts are presented. Students are asked to write summaries (essaies). Essay 1: A list of keywords, no relation between them Essay 2: A copy of the original text Essay 3: A perfect essay about a completely different topic Essay 4: A good essay about the topic, containing the right keywords

LSA: How does it work? The text is represented as a matrix in which each row stands for a unique word and each column stands for a text passage or other context. Each cell contains the frequency with which the word of its row appears in the passage denoted by its column Next, the cell entries are subjected to a preliminary transformation, in which each cell frequency is weighted by a function that expresses both the word's importance in the particular passage and the degree to which the word type carries information in the domain of discourse in general Next, SVD (Single Value Decomposition) is applied to the matrix. This reduces dimensionality.

LSA Caveats " LSA is a pure statistical representation of words and their common appearance " LSA lacks information such as that expressed by synatx and used in logic " LSA mimics lexical and semantical knowledge but has none " LSA can only capture a limited range of topics at a time: to many topics in one representation lead to a less detailed analysis of students texts

An Introduction to Auto Tutor A Tutoring system which uses LSA as its Analysing Component Benjamin Kempe

Why2 Another intelligent tutoring system which uses LSA The goal of Why2 is to coach students through the process of constructing explanations that are complete and do not contain any misconceptions. To do so, Wy2 uses KCDs (Knowledge Construction Dialogues). Those KCDs are eather interactive directed lines of reasoning. Either to elicit a specific idea, or to remediate a specific misconception.

It is important to select exactly the appropriate KCDs both to give students the KCDs that they do need, and to avoid giving the extraneous KCDs that they do not need. Why2 uses a measurement system, which is able to evaluate (and so to better) the quality of KCD selection in terms of KCD Precision: KCDs correctly given / total KCDs given KCD Recall: KCDs correctly given / KCDs needed KCD false alarm rate: KCDs incorrectly given / KCDs not needed KCD precision and recall vary with essay quality: the better a students essay, the more difficult it gets to find good KCDs.

To select appropriate KCDs, Why2 uses a special system to select the dialogues: students answers are evaluated with a statistical system (Rinbow, a Bayes classifier) as well as with a symbolic system (CARMEL et al.). Additional Symbolic Analysis gives better results (Better selected KCDs: more precision, less false alarms, better recall rate)

Object of research: Combination of diffrent techniques for good analysis of students essaies. „Bag of words“ i.e. statistical approaches are fast and low cost but can be tricked. Knowledge based approaches are slow and brittle but more precise and capture nuances. Natural langage -> CARMEL: symbolic sentence level language understanding -> set of first oder logical forms Natural Language->RAINBOW: naive Bayes classifier.Assigns sentences to classes that are associated with set of logical forms Logical forms -> DLU: discoures language understanding -> proof trees Proof treef ->which points are missing, which misconceptions

Literature " An Introduction to Latent Semantic Analysis. Landauer, T.K., Folz, P.W., Laham, D. (1998). Discourse Processes, 25, " Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor. Graesser, A.C., Wiemer-Hastings, P., Wiemer-Hastings, K. Harter, D. " Improving an intelligent tutor's comprehension of students with Latent Semantic Analysis. Wiener-Hastings, P., Wiemer-Hastings, K., Graesser, A.C. " A Hybrid Language Understanding Approach for Robust Selection of Tutoring Goals. Rosé, C.P., Bhembe, D., Roque, A., Siler, S., Srivastava, R., VanLehn, K. " "

Thank you! Questions?