1. What can CTAT do for you? Overview of the CTAT track Vincent Aleven, Bruce McLaren and the CTAT team 3rd Annual PSLC LearnLab Summer School Pittsburgh, June 18-22, 2007

2. What can CTAT do for you? - 2 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 If you are not in the CTAT track, why would you listen to this talk? CTAT relevant to most other tracks: –TuTalk: TuTalk dialogue tutor agent and CTAT intelligent tutor would make an awesome duo! –In Vivo: could do an in vivo experiment with CTAT-based tutors (happens all the time!) –Data Shop: many data sets in the Data Shop were generated using CTAT-built tutors –TagHelper: Hmm…. maybe you are naturally curious about CTAT Track hopping is allowed! –E.g., if in the In Vivo track, could attend (some) CTAT sessions

3. What can CTAT do for you? - 3 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Overview What is “a tutor?” –And would you want to build one? Why do you need CTAT to build one? What more precisely can you do with CTAT? –Short movie of authoring with CTAT –Examples of projects that have used CTAT –Key choices to make when authoring with CTAT Planned activities in the CTAT track

4. What can CTAT do for you? - 4 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 A simple example: Fraction Addition Interact with the fraction addition tutor on the CTAT web site …

5. What can CTAT do for you? - 5 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Key elements of interactivity in CTAT Tutors 1.Problem-solving environment, often with interactive tools 2.Tutorial guidance, in the form of a. implicit yes-no feedback on correctness of problem steps b. feedback messages for commonly-occurring errors c. next-step hints (on demand or tutor initiated) ==> can handle multiple student strategies for solving problems 3.Adaptive problem selection based on student performance solving problems with the tutor Koedinger, K. R., & Aleven V. (in press). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review.

6. What can CTAT do for you? - 6 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Kinds of Computer Tutors Intelligent tutoring systems e.g., Sherlock Model-tracing tutors e.g., Andes Cognitive Tutors e.g., Algebra Tutoring systems CAI e.g., Microsoft’s Personal Tutor Constraint- based tutors e.g., SQL Tutor Example-tracing tutors e.g., Stoichiometry, French Culture Tutor Can be built with CTAT

7. What can CTAT do for you? - 7 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Cognitive Tutors: Real-world success of Intelligent Tutoring Systems technology Cognitive Tutor Courses –Goal: Improve quantitative literacy & fluency –Computer-based one-to-one tutoring Based on computational models of student thinking Based on the concept of Model Tracing Used by many students in many schools –Algebra Cognitive Tutor: In over 2,000 schools in the USA, 300,000 students per year –Geometry Cognitive Tutor: In 350 schools –Most widely used intelligent tutoring system? University created a company, Carnegie Learning, to disseminate

8. Algebra Cognitive Tutor Sample Use graphs, graphics calculator Analyze real world problem scenarios Use table, spreadsheet Use equations, symbolic calculator Tutor learns about each student Tutor follows along, provides context-sensitive Instruction

9. What can CTAT do for you? - 9 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Replicated Field Studies Full year classroom experiments Replicated over 3 years in urban schools In Pittsburgh & Milwaukee Results: 50-100% better on problem solving & representation use. 15-25% better on standardized tests. Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.

10. What can CTAT do for you? - 10 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 ACT-R: A Cognitive Theory of Learning and Performance Big theory … key tenets: –Learning by doing, not by listening or watching –Production rules represent performance knowledge: These units are: Instruction implications: modular context specific isolate skills, concepts, strategies address "when" as well as "how" Anderson, J.R., & Lebiere, C. (1998). The Atomic Components of Thought. Erlbaum.

11. What can CTAT do for you? - 11 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Cognitive Model: A system that can solve problems in the various ways students can Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d Cognitive Tutor Technology: Use ACT-R theory to individualize instruction

12. What can CTAT do for you? - 12 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology: Use ACT-R theory to individualize instruction Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

13. What can CTAT do for you? - 13 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology: Use ACT-R theory to individualize instruction Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction Hint message: “Distribute a across the parentheses.” Bug message: “You need to multiply c by a also.” Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing Known? = 85% chanceKnown? = 45%

14. What can CTAT do for you? - 14 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 CTAT motivation: Make tutor development easier and faster! Development costs of instructional technology are high –E.g., ~300 dev hours per hour of instruction for Computer Aided Instruction (Murray, 1999) Cognitive Tutors: –Large student learning gains as a result of detailed cognitive modeling –~200 dev hours per hour of instruction (Koedinger et al, 1997) –Requires PhD level cog scientists and AI programmers Solution: Easy to use Cognitive Tutor Authoring Tools (CTAT) Murray, T. (1999). Authoring Intelligent Tutoring Systems: An Analysis of the state of the art. The International Journal of Artificial Intelligence in Education, 10, 98-129. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. The International Journal of Artificial Intelligence in Education, 8, 30-43.

15. What can CTAT do for you? - 15 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 CTAT goal: broaden the group of targeted authors Instructional technology developers (e.g., instructional media dept. at university, or developers of on-line courses) Researchers interested in intelligent tutoring systems Instructors (e.g., computer-savvy college professors) Learning sciences researchers interested in using computer-based tutors in their experiments –Within the PSLC, CTAT-based tutors are often used in in vivo experiments

16. What can CTAT do for you? - 16 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 How to reduce the authoring cost? Less programming, more automation –Drag & drop interface construction –Demonstration-based programming –New: Machine learning and data mining Human-Computer Interaction (HCI) methods –User studies, summer schools, informal & formal comparison studies Exploit tools already in use –Component-based architecture with “standard” tool-tutor protocol –Off-the-shelf tools and languages (e.g., Netbeans, Eclipse, Flash, Jess)

17. What can CTAT do for you? - 17 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 CTAT supports two types of tutors Cognitive Tutors and Example-Tracing Tutors Behavior is similar (though not identical), both support the same elements of interactivity: –problem-solving environment with tutorial guidance: –yes/no feedback on steps (can handle multiple student strategies!) –Specific feedback messages for common errors –Next-step hints Technology and development are very different –Example-Tracing Tutors: built through “programming by demonstration” - authors creates examples of how the students should solve problems –Cognitive Tutors: built by AI programming (much harder) - author creates rule-based cognitive model

18. What can CTAT do for you? - 18 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Movie Showing How an Example- Tracing Tutor is built

19. What can CTAT do for you? - 19 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Authoring with CTAT and Flash Behavior Recorder Flash authoring environment with tutor interface shown Web browser with Example-Tracing tutor embedded in HTML

20. What can CTAT do for you? - 20 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Authoring an Example-Tracing Tutor Step 1: Create a User Interface –Create the graphical user interface (GUI) used by the student Step 2: Demonstrate Behavior – Demonstrate correct, alternative correct, and incorrect solutions Step 3: Generalize – Specify how demonstrated behavior could vary within given problem allowed order of steps allowed variants for a given step Step 4: Annotate the Graph –Annotate solutions steps in the resulting “behavior graph” with: hint messages, error messages, labels for concepts or skills associated with actions Test and Iterate on Steps 1-4 … Step 5: Publish Tutor to Web

21. Behavior Recorder Data Shop CTAT’s Modular Architecture Student Interface (or external problem-solving environment) Cool Modes CyclePad Java Swing Flash Learner Management System LMS Tutor Shop Cognitive Model Development Tools TDK (Lisp-based) Jess tools (Java- based) Editor Eclipse Tutor Engine Tertle (Lisp-based) + Model Tracer Jess (Java-based) + Model Tracer Example Tracer (Java-based) Example Tracer (Flash-based) GUI Builder IntelliJ Dreamweaver Code Warrior Netbeans Flash MX 2004

22. What can CTAT do for you? - 22 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 DataShop - Inspection & Evaluation of CTAT Data Learning Curve Report

23. What can CTAT do for you? - 23 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 DataShop - Inspection & Evaluation of CTAT Data Error Report

24. What can CTAT do for you? - 24 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 CTAT’s Track Record CTAT-based tutors have been used in (approx. 15) experiments in Geometry, Chemistry, Chinese, and French Over 300 users (you not included) Four releases in past year (1.6, 2.0, 2.1, 2.2) CTAT Web site –35,000 unique visitors in 2006, 580,000 hits CTAT Downloads –2,200 in 2006 (14% CMU & Pitt)

25. What can CTAT do for you? - 25 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 http://ctat.pact.cs.cmu.edu

26. What can CTAT do for you? - 26 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Development time comparison Past estimates of ITS development 200:1 for robust beta versions used in real classrooms Example-Tracing Tutor Development 25:1 for initial alpha versions: (Koedinger et al, 2004) If creating a robust beta doubles dev time, we would have ~50:1 ratio -- a reduction of 4 times! Koedinger, K., Aleven, V., Heffernan, N., McLaren, B. M., and Hockenberry, M (2004). Opening the Door to Non-Programmers: Authoring Intelligent Tutor Behavior by Demonstration;. In the Proceedings of the Seventh International Conference on Intelligent Tutoring Systems (ITS-2004).

27. What can CTAT do for you? - 27 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Where does CTAT fit into the Realm of Authoring Tools? Murray survey (1999) discusses 7 categories of authoring tools for intelligent tutoring systems (ITS), including: Device Simulation - RIDES (Munro et al 1997) Tutoring Strategies - REDEEM (Ainsworth et al 2003; Major et al 1997) Domain Expert System - Demonstr8 (Blessing 1997) Cognitive Tutor Authoring Tools: “Domain Expert System” category of authoring systems No commercially successful tools for authoring model-tracing, Cognitive Tutors First to provide functionality to simplify & speed cog. tutor development

28. What can CTAT do for you? - 28 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Other Example-Tracing Tutors built with CTAT Chinese - tone study Chinese - listen and read French culture French Chemistry (stoichiometry) Geometry (self-assessment) Geometry (on-line testing)

29. What can CTAT do for you? - 29 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Chinese: tone study Wang, Perfetti, Liu, et al.

30. What can CTAT do for you? - 30 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Stoichiometry study McLaren et al.

31. What can CTAT do for you? - 31 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Geometry self-assessment tutor Roll et al.

32. What can CTAT do for you? - 32 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 French culture Ogan et al.

33. What can CTAT do for you? - 33 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Help seeking Aleven, McLaren, Roll, et al.

34. What can CTAT do for you? - 34 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 French Jones et al.

35. What can CTAT do for you? - 35 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Chinese: listen and read Mei, Haney, et al.

36. What can CTAT do for you? - 36 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Division Tutor (Stefan King)

37. What can CTAT do for you? - 37 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Mass Production: Create template behavior graph and spreadsheet with data for each problem Each row contains values for one variable in the Behavior Graph template Each column contains data for 1 problem A Behavior Graph is created automatically for each problem by merging the spreadsheet data into the template

38. What can CTAT do for you? - 38 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 What is Mass Production? Many learning situations require repetition of similar problems, but authoring many similar Example-Tracing Tutors is hard –It is time-consuming to demonstrate individual problems –Inconsistency between problems is often introduced Examples: –Practice fraction addition problems in a math class –Drill on proper use of articles in a language class CTAT mass production allows us to use a single demonstration as the basis for many tutored problems of the same type

39. What can CTAT do for you? - 39 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 The Mass Production process

40. What can CTAT do for you? - 40 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Self-explanation of incorrect examples Julie Booth et al.

41. What can CTAT do for you? - 41 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Another example - Thermodynamics Simulator hooked up with CTAT CTAT provides tutoring for analyzing power plant efficiency

42. What can CTAT do for you? - 42 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 What type of tutor shall I build? Cognitive Tutors v. Example-Tracing Tutors Cognitive Tutor (e.g., Algebra Cognitive Tutor, Logic Tutor) –Technology: uses rule-based cognitive model to monitor students (implemented in Jess) –Development: Difficult to build; requires AI programming –Advantages: General for a class of problems; can deal with subtle dependencies of steps, many solution paths, etc. –Sometimes an advantage: SimStudents module writes rules for you (experimental use of machine learning; tried on a small scale) Example-Tracing Tutors (the other examples in this talk) –Technology: uses examples of problem solutions to monitor students –Development: programming by demonstration, “Mass Production” facilitates authoring of many isomorphic problems (in Excel) –Advantages: Much easier to build; for non-programmers

43. What can CTAT do for you? - 43 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 How to make the choice? Often, example-tracing tutors are a good choice –Easier to build –At this point in time, their web delivery is better supported Cognitive Tutors only if –You have (AI) programming experience (possibly gained during this summer school), and –You deal with problems that have subtle dependencies among steps, or many different solutions paths Best to build an example-tracing tutor first –Rapid prototype! And supports cognitive task analysis –Then decide whether to move on to Cognitive Tutor –Example-Tracing Tutors re-usable when developing a Cognitive Tutor: Examples guide planning, serve as semi-automated test cases http://ctat.pact.cs.cmu.edu/index.php?id=tutortypes

44. What can CTAT do for you? - 44 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 During the summer school Learning to create Example-Tracing Tutors is easy, learning to create Cognitive Tutors is hard The CTAT track will cover development of Cognitive Tutors and Example-Tracing Tutors –In your project you could decide to focus mainly on example- tracing tutors –(Then again, this is your chance to get some mentoring as you build a Cognitive Tutor) If you are not in the CTAT track, but interested in learning to build tutors in limited time, it is best to focus on example-tracing tutors

45. What can CTAT do for you? - 45 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Options for developing a user interface for your tutor (no programming!) Java with Netbeans GUI Builder –Freely available –Web delivery via WebStart (we are still gaining experience with this) Flash (from Adobe) –Easy to deploy on the web: With plug-in, compatible with many browsers –Not free

46. What can CTAT do for you? - 46 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Choices, choices, … $$$ may be a factor … Whether you know Flash or Java need not be a factor –Unless you need to program custom behaviors Web delivery of Cognitive Tutors only supported with Java, but web delivery of Example-Tracing tutors is easier in Flash Sets of widgets/components are different http://ctat.pact.cs.cmu.edu/index.php?id=reference-widgets http://ctat.pact.cs.cmu.edu/index.php?id=reference-components See also : http://ctat.pact.cs.cmu.edu/index.php?id=java-flash_differences http://ctat.pact.cs.cmu.edu/index.php?id=tutortypes Our recommendation: use Flash if you can

47. What can CTAT do for you? - 47 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 During the summer school Examples shown in demos and lectures will come mainly from Java / Netbeans For your projects, you are free to choose – If the long-term prospects for your project are uncertain, then the available widgets/components should be decisive –Otherwise, all factors mentioned on the previously slide should probably be considered

48. What can CTAT do for you? - 48 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Summary of choices to be made when using CTAT for real Cognitive Tutor or Example-Tracing Tutors? –If Cognitive Tutors, use SimStudents? –If Example-Tracing Tutors, use Mass Production? Tutor interface in Flash or Java? –Also possible to program your own

49. What can CTAT do for you? - 49 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Overview of CTAT track activities

50. What can CTAT do for you? - 50 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Demo of CTAT

51. What can CTAT do for you? - 51 3rd LearnLab Summer School, Pittsburgh, PA, June 18-22, 2007© Vincent Aleven, Bruce McLaren, & the CTAT Team, 2007 Thank you!