Intelligent Tutoring Systems

What is an ITS? Proposal: Start with a bunch of exercises for a topic, where each exercise is given a difficulty rating and subtopic. Start students with a series of exercises at the easiest rating (perhaps with training materials) for the first topic. When the student has “proved proficiency” at the lowest rating, start giving them the next difficulty (perhaps with more training materials). When they complete the various levels for the current subtopic, move to the next one. Is this an ITS?

A Definition: To be classified as “intelligent”, they must present “human-like” tutoring capabilities, that means to be able to adjust the content and delivery of the lesson to the students’ characteristics and needs by analysing and/or anticipating theirs responses and behaviours. -Roger Nkambou

Q: Does a (human) tutor make much difference?

Effects of (Human) Tutoring Two-sigma “problem”: Bloom observed that students who receive good 1-on-1 tutoring perform two standard deviations better than students who receive traditional classroom instruction. The ideal ITS would perform as well as a good human tutor, providing equivalent 1-on-1 instruction. Today’s online courses are at the performance level of traditional classroom instruction.

Key problem “I am stuck, what do I do now?” Online courses do no better than traditional lecture to solve this problem. Can ITS do better? Even if an ITS only handled the easy situations, that would still leave more time for a human instructor to handle the more difficult cases. Work ongoing with ITS since the 1970s

Early Successes 1980’s: CMU LISP Tutor students performed 43% better on final than traditional instruction completed complex programming task 30% faster Early 1990’s: Sherlock (jet aircraft troubleshooting procedures) Students using ITS did significantly better than control group. After 20 hours of instruction, performed at the level of technicians with 4 years of experience. Many successful military training systems Successful math instruction: Cognitive Tutors

Traditional Architecture

Primary Components Expert Model (Domain Model): Subject matter knowledge Student Model: Represents what the student does and does not know Instructor Model (Tutoring Model): ITS strategy for how to teach User Interface Model

Rule-based Approach Contains domain expert’s subject matter knowledge and problem- solving strategies Problem-solving strategy can be captured by rules ITS can compare learner’s process to the expert model’s actions. Ideally, can generate problems on-the-fly from the rules. Expensive to build

Scenario-based Approach A given training scenario incorporates the necessary knowledge. A template task incorporates the necessary steps to solve the problem Limits the necessary knowledge to the one problem type A tutorial is then a series of such problem types, each with its knowledge and rules

Student Model Evaluates student performance The key problem is deducing what misconception led to a particular wrong answer, and then addressing that misconception. Example: Mistakes in addition Fail to carry Always carry Mistakes in single-digit addition

Example: Tactical Action Officer

Instructor Model Encodes instructional process Integrates information from the Student and Expert Models to make decisions Attempts to decide the “next step” Supply instruction (new content or hints) Supply problems (of right difficulty)

ITS Systems While the picture describe of the gross architecture is common, every ITS system is different in the details. Typically an ITS works for only some tutoring strategies. Each typically focuses on one model (expert, student, or instructor) over the others.

Rules One approach to the instructor model is as a large body of production rules. The Student Model stores values for probability that the student knows each rule. When the student applies a rule to the current problem state, the probability value for that rule goes up.

Instructor Model: Guidance A major issue is how much guidance to give the student. Too much leads to hints-driven behavior, shallow learning Too little leads to student not knowing what to do, making no progress. Direct Instruction vs. Discovery Learning Discovery learning is in favor “Minimally guided discovery” or “pure discovery” is out of favor

Cognitive Tutor Example of an experiment (Brunstein, Betts, and Anderson 2009). Also illustrates a cognitive math tutor.

Experiment

Results Conditions: (a) Written instruction, (b) direct demonstration, (c) both, (d) neither (discovery) For the first two problems in a set, discovery was the slowest But for the remaining problems, discovery and direct demonstration were significantly faster than the others (total time) This was true ONLY when there was significant practice during each section. When the amount of practice was cut, the discovery treatment participants effectively could not continue past a certain point. This represents a typical example of the type of design issues that ITS developers face. In this case, I question if they got the right interpretation. This could be true that in isolation, discovery and lots of practice does best. But over the entire series, direct demonstration might do the best.

Development Process for ITS Needs Assessment Figure out the specific content pieces, their difficulty, etc. Cognitive Task Analysis Build computational model of the required problem solving knowledge Lots of domain expert interviews for how to solve problems Initial Tutor Implementation Set up guidance system to support student working problems. Evaluation

Are ITS used? Recent evaluations of ITS have found some that have effect size comparable to one-on-one human tutors. Algebra Cognitive Tutor Such cognitive math tutors used in thousands of schools each year, with hundreds of thousands of students If they have been studied so long, and work so well, why do they not make a bigger impact? Does Khan Academy use ITS? Code Academy, etc?

Cost Issues Development times are typically 200-300 hours for 1 hour of instruction. Not considered commercially viable at this ratio This does not actually make sense to me. That cost does not seem at all excessive.

Pedagogical Limitations Might not develop “deep” learning in students Being able to request hints on demand might harm learning Some students use a “hints driven” approach to get the exercises done ITS does not ask questions of students to get them to explain their problem solving approach The result is that they don’t learn the language, and so cannot communicate about the content.

Current research Better dialog systems (Socratic Method, Discovery Driven) Better detection of student emotional affect Observation: The best learning comes with some --- but not too much --- tension or frustration.