1. 1 LearnLab: Bridging the Gap Between Learning Science and Educational Practice Ken Koedinger Human-Computer Interaction & Psychology, CMU PI & CMU Director of LearnLab

2. 2 Real World Impact of Cognitive Science Algebra Cognitive Tutor Based on ACT-R theory & cognitive models of student learning Used in 3000 schools 600,000 students Spin-off: Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

3. 3 Personalized instruction Challenging questions … individualization Progress… Authentic problems Feedback within complex solutions Cognitive Tutors: Interactive Support for Learning by Doing

4. 4 Success ingredients AI technology Cognitive Task Analysis Principles of instruction & experimental methods Fast development & use-driven iteration

5. Cognitive Task Analysis: What is hard for Algebra students? Story Problem As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work? Word Problem Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with? Equation x * 6 + 66 = 81.90

6. 6 0 10 20 30 40 50 60 70 80 90 100 Elementary Teachers Middle School Teachers High School Teachers % Correctly ranking equations as hardest Nathan & Koedinger (2000). An investigation of teachers’ beliefs of students’ algebra development. Cognition and Instruction. Expert Blind Spot! Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences. Data contradicts common beliefs of researchers and teachers

7. 7 Cognitive Tutor Algebra course yields significantly better learning Course includes text, tutor, teacher professional development ~11 of 14 full-year controlled studies demonstrate significantly better student learning Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

8. 8 Success? Yes Done? No! Why not? Student achievement still not ideal Field study results are imperfect Many design decisions with no research base Use deployed technology to collect data, make discoveries, & continually improve

9. 9 PSLC Vision Why? Chasm between science & ed practice Purpose: Identify the conditions that cause robust student learning –Educational technology as instrument –Science-practice collaboration structure Core Funding: 2004-2014

10. 10 What we know about our own learning What we do not know You can’t design for what you don’t know! Do you know what you know?

11. 11 Chemistry Virtual Lab Algebra Cognitive Tutor Ed tech + wide use = “Basic research at scale” = Transforming Education R&D Fundamentally transform –Applied research in education –Generation of practice- relevant learning theory + English Grammar Tutor Educational Games

12. Ed Tech => Data => Better learning LearnLab Thrusts LearnLab Course Committees

13. 13 How you can benefit from LearnLab Research –General principles to improve learning Methods –Cognitive task analysis, in vivo studies Technology tools People –Masters students & projects

14. 14 What instructional strategies work best? More assistance vs. more challenge –Basics vs. understanding –Education wars in reading, math, science… Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Ed Psych Review. Research on many dimensions –Massed vs. distributed (Pashler) –Study vs. test (Roediger) –Examples vs. problem solving (Sweller,Renkl) –Direct instruction vs. discovery learning (Klahr) –Re-explain vs. ask for explanation (Chi, Renkl) –Immediate vs. delayed (Anderson vs. Bjork) –Concrete vs. abstract (Pavio vs. Kaminski) –…

15. 15 Knowledge-Learning-Instruction (KLI) Framework: What conditions cause robust learning LearnLab research thrusts address KLI elements Cognitive Factors –Charles Perfetti, David Klahr Metacognition & Motivation –Vincent Aleven, Tim Nokes-Malach Social Communication –Lauren Resnick, Carolyn Rose Computational Modeling & Data Mining –Geoff Gordon, Ken Koedinger Koedinger et al. (2012). The Knowledge-Learning- Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

16. 16 Results of ~200 in vivo experiments => Optimal instruction depends on knowledge goals

17. 17 Cognitive Task Analysis using DataShop’s learning curve tools Without decomposition, using just a single “Geometry” KC, Upshot: Can automate analysis & produce better student models But with decomposition, 12 KCs for area concepts, a smoother learning curve. no smooth learning curve.

18. 18 How you can benefit from LearnLab Research –General principles to improve learning Methods –Cognitive task analysis, in vivo studies Technologies –Tutor authoring –Language processing –Educational Data Mining People: Masters students & projects

19. 19 Questions?

20. 20 Question for you What do you need in a learning science professional?

21. 21

22. 22 Extra slides

23. 23 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

24. 24 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction Hint message: “Distribute a across the parentheses.” Bug message: “You need to multiply c by a also.” Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing Known? = 85% chanceKnown? = 45%

25. 25 Cognitive Task Analysis Improves Instruction Studies: Traditional instruction vs. CTA-based –Med school catheter insertion (Velmahos et al., 2004) –Radar system troubleshooting (Schaafstal et al., 2000) –Spreadsheet use (Merrill, 2002) Lee (2004) meta-analysis: 1.7 effect size!

26. 26 Learning Curves

27. 27 Inspect curves for individual knowledge components (KCs) Some do not => Opportunity to improve model! Many curves show a reasonable decline

28. 28 DataShop’s “leaderboard” ranks alternative models 100s of datasets from ed tech in math, science, & language Best model finds 18 components of knowledge (KCs) that best predict transfer 28

29. Data from a variety of educational technologies & domains 29 Numberline Game Statistics Online Course English Article Tutor Algebra Cognitive Tutor

30. Model discovery across domains 30 11 of 11 improved models Variety of domains & technologies Koedinger, McLaughlin, & Stamper (2012). Automated student model improvement. In Proceedings of Educational Data Mining. [Conference best paper.]

31. 31 Data reveals students’ achievement & motivations We have used it to Predict future state test scores as well or better than the tests themselves Assess dispositions like work ethic Assess motivation & engagement Assess & improve learning skills like help seeking …

32. 32 LearnLab courses at K12 & College Sites 6 + cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English Data collection –Students do home/lab work on tutors, vlab, OLI, … –Log data, questionnaires, tests  DataShop Researchers Schools Learn Lab Chemistry virtual lab Physics intelligent tutor REAP vocabulary tutor

33. 33 Lab experiment In Vivo Experiment Design Research Randomzd Field Trial SettingLabSchoolSchoolSchool Control conditionYesYesNoYes Focus on principle vs. on solution (Change N things) Scientific Principle Instr. Solution Cost/Duration$/Short$$/Medium $$/Long $$$$/Long Bridging methodology: in vivo experiments

34. 34 Knowledge Components Definition: An acquired unit of cognitive function or structure that can be inferred from performance on a set of related tasks Includes: –skills, concepts, schemas, metacognitive strategies, malleable habits of mind, thinking & learning skills May also include: –malleable motivational beliefs & dispositions Does not include: –fixed cognitive architecture, transient states of cognition or affect Components of “intellectual plasticity” Koedinger et al. (2012). The Knowledge-Learning- Instruction (KLI) framework: Bridging the science- practice chasm to enhance robust student learning. Cognitive Science.

35. 35 General knowledge components, sense-making, motivation, social intelligence Possible domain-general KCs Metacognitive strategy –Novice KC: If I’m studying an example, try to remember each step –Desired KC: If I’m studying an example, try to explain how each step follows from the previous Motivational belief –Novice: I am no good at math –Desired: I can get better at math by studying & practicing Social communicative strategy –Novice: If an authority makes a claim, it is true –Desired: If considering a claim, look for evidence for & against it

36. 36 What is Robust Learning? Achieved through: –Conceptual understanding & sense-making skills –Refinement of initial understanding –Development of procedural fluency with basic skills Measured by: –Transfer to novel tasks –Retention over the long term, and/or –Acceleration of future learning

37. 37 KLI summary Learning occurs in components (KCs) KCs vary in kind/cmplxty –Require different kinds of learning mechanisms Optimal instructional choices are dependent on KC complexity Intelligence does not improve generically Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

38. 38 Conclusions Learning & education are complex systems Lots of work for learning science! Use ed tech for “basic research at scale” => Bridge science-practice chasm