1 Cognitive Principles in Tutor & e-Learning Design Ken Koedinger Human-Computer Interaction & Psychology Carnegie Mellon University CMU Director of the Pittsburgh Science of Learning Center

2 Lots of Lists of Principles 1 Cognitive Tutor Principles –Koedinger, K. R. & Corbett, A. T. (2006). Cognitive Tutors: Technology bringing learning science to the classroom. Handbook of the Learning Sciences. –Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4 (2), Multimedia & eLearning Principles –Mayer, R. E. (2001). Multimedia Learning. Cambridge University Press. –Clark, R. C., & Mayer, R. E. (2003). e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. San Francisco: Jossey-Bass. How People Learn Principles –Donovan, M. S., Bransford, J. D., & Pellegrino, J.W. (1999). How people learn: Bridging research and practice. Washington, D.C.: National Academy Press. Progressive Abstraction or “Bridging” Principles –Koedinger, K. R. (2002). Toward evidence for instructional design principles: Examples from Cognitive Tutor Math 6. Invited paper in Proceedings of PME-NA. Other lists on the web … –See learnlab.org/research/wiki

Principles on web: See learnlab.org/research/wiki

4 Overview Cognitive Tutor Principles Multimedia Principles –Theoretical & Experimental evidence Instructional Bridging Principles –Need empirical methods to apply PSLC Principles

5 Cognitive Tutor Principles 1.Represent student competence as a production set 2.Communicate the goal structure underlying the problem solving 3.Provide instruction in the problem-solving context 4.Promote an abstract understanding of the problem-solving knowledge 5.Minimize working memory load 6.Provide immediate feedback on errors

6 1. Represent student competence as a production set Accurate model of target skill to: –Inform design of Curriculum scope & sequence, interface, error feedback & hints, problem selection & promotion –Interpret student actions in tutor Knowledge decomposition! –Identify the components of learning

7 6. Provide immediate feedback on errors Productions are learned from the examples that are the product of problem solving Benefits: –Cuts down time students spend in error states –Eases interpretation of student problem solving steps Evidence: LISP Tutor Smart delayed feedback can be helpful –Excel Tutor

8 Feedback Studies in LISP Tutor (Corbett & Anderson, 1991) Time to Complete Programming Problems in LISP Tutor Immediate Feedback Vs Student-Controlled Feedback

9 Tutoring Self-Correction of Errors Recast delayed vs. immediate feedback debate as contrasting “model of desired performance” Expert Model –Goal: students should not make errors Intelligent Novice Model –Goal: students can make some errors, but recognize them & take action to self-correct Both provide immediate feedback –Relative to different models of desired performance Mathan, S. & Koedinger, K. R. (2003). Recasting the feedback debate: Benefits of tutoring error detection and correction skills. In Hoppe, Verdejo, & Kay (Eds.), Proceedings of Artificial Intelligence in Education (pp ). Amsterdam, IOS Press. [Best Student Paper.]

10 Intelligent Novice Condition Learns More F = 4.23, p <.05

11 Learning Curves: Difference Between Conditions Emerges Early Number of attempts at a step by opportunities to apply a production rule

12 Overview Cognitive Tutor Principles Multimedia Principles –Theoretical & Experimental evidence Instructional Bridging Principles –Need empirical methods to apply PSLC Principles

13 Media Element Principles of E-Learning 1. Multimedia 2. Contiguity 3. Coherence 4. Modality 5. Redundancy 6. Personalization

14 NarrationAuditory WM AnimationVisual WM Build Referential Connections OnScreen Text Long Term Memory Cognitive Processing of Instructional Materials Instructional material is: –Processed by our eyes or ears –Stored in corresponding working memory (WM) Must be integrated to develop an understanding Stored in long term memory

15 Multimedia Principle Which is better for student learning? A. Learning from words and pictures B. Learning from words alone Example: Description of how lightning works with or without a graphic A. Words & pictures Why? Students can mentally build both a verbal & pictorial model & then make connections between them

16 Coherence Principle Which is better for student learning? A. When extraneous, entertaining material is included B. When extraneous, entertaining material is excluded Example: Including a picture of an airplane being struck by lightning B. Excluded Why? Extraneous material competes for cognitive resources in working memory and diverts attention from the important material

17 Modality Principle Which is better for student learning? A. Spoken narration & animation B. On-screen text & animation Example: Verbal description of lightning process is presented either in audio or text A. Spoken narration & animation Why? Presenting text & animation at the same time can overload visual working memory & leaves auditory working memory unused.

18 Working Memory Explanation of Modality When visual information is being explained, better to present words as audio narration than onscreen text

19 Scientific Evidence That Principles Really Work PrinciplePercent Gain Effect SizeNumber of Tests Multimedia of 9 Contiguity of 5 Coherence of 11 Modality of 4 Redundancy of 2 Personalization of 5 Summary of Research Results from the Six Media Elements Principles. (From Mayer, 2001) * Used similar instructional materials in the same lab.

20 Summary of Media Element Principles of E-Learning 1.Multimedia: Present both words & pictures 2.Contiguity: Present words within picture near relevant objects 3.Coherence: Exclude extraneous material 4.Modality: Use spoken narration rather than written text along with pictures 5.Redundancy: Do not include text & spoken narration along with pictures 6.Personalization: Use a conversational rather than a formal style of instruction

21 Overview Cognitive Tutor Principles Multimedia Principles –Theoretical & Experimental evidence Instructional Bridging Principles –Need empirical methods to apply PSLC Principles

22 How People Learn Principles How People Learn book 1.Build on prior knowledge 2.Connect facts & procedures with concepts 3.Support meta-cognition Bransford, Brown, & Cocking (1999). How people learn: Brain, mind, experience and school. D.C.: National Academy Press.

23 But: What prior knowledge do students have? How can instruction best build on this knowledge?

24 Instructional Bridging Principles 1. Situation-Abstraction Concrete situational abstract symbolic reps 2. Action-Generalization Doing with instances explaining with generalizations 3. Visual-Verbal Visual/pictorial verbal/symbolic reps 4. Conceptual-Procedural Conceptual procedural Koedinger, K. R. (2002). Toward evidence for instructional design principles: Examples from Cognitive Tutor Math 6. Invited paper in Proceedings of PME-NA.

25 Dimensions of Building on Prior Knowledge Situation (candy bar) vs. Abstraction (content-free) Visual (pictures) vs. Verbal (words, numbers) Conceptual (fraction equiv) vs. Procedural (fraction add) SituationAbstraction

26 Which is easier, situation or analogous abstract problem?

27 Which is easier, situation or analogous abstract problem?

28 Key Point: Design principles require empirical methods to successfully implement

29 Overview Cognitive Tutor Principles Multimedia Principles –Theoretical & Experimental evidence Instructional Bridging Principles –Need empirical methods to apply PSLC Principles

30 PSLC Theory Wiki: Instructional principle & study pages … demo

31

32

33

34

35

36

37 … more in demo …

38 External validity of example-rule coordination Same principle tested across different background contexts (b’s) different courses

39 Cross-Cluster Theoretical Integration: Assistance Dilemma “How should learning environments balance information or assistance giving and withholding to achieve optimal student learning?” –Koedinger & Aleven, 2007 Instructional support Poor Poor learning outcome Good Good learning outcome High assistance (less demanding) scaffold Low assistance (more demanding) extraneous load Row 1 illustrates how higher levels of instructional assistance can sometimes be a “crutch” that harms learning, but other times be a “scaffold” that bootstraps learning. Row 2 illustrates how lower levels of assistance (or inversely greater imposed demands on students) can sometimes lead to poorer learning and other times lead to better learning. A long line of research on “cognitive load theory” (e.g., Sweller, Van Merriënboer, & Paas, 1998) suggests how some typical forms of instruction, like homework practice problems, put “extraneous” processing demands (or “extraneous load”) on students that may detract from learning. Another line of research on “desirable difficulties” suggests ways in which making task performance harder during instruction (reducing assistance), for instance, by delaying feedback, enhances learning (Schmidt and Bjork, 1992). And even within cognitive load theory, some task demands (e.g., increased problem variability) elicit “germane” rather than “extraneous” cognitive load and lead to better learning. Long-standing notions like zone of proximal development (Vygotsky, 1978), aptitude-treatment interactions (Cronbach & Snow, 1977), or model-scaffold-fade (Collins, Brown, & Newman, 1990) suggest that instructional assistance should be greater for beginning learners and be reduced as student competence increases. So, what’s the dilemma? Why not just give novices high assistance and fade it away as they become more expert. The theoretical problem, the dilemma, is that current theory does not predict how much assistance to initially provide nor when and how fast to fade it. It does not provide predictive guidance as to when an instructional demand is “germane” or “extraneous”, “desirable” or “undesirable.” The Assistance Dilemma remains unresolved because we do not have adequate cognitive theory to make a priori predictions about what forms and levels of assistance yield robust learning under what conditions. In Koedinger et al. (2008), we outlined the following steps toward resolving the dilemma: 1.Decompose: Identify and distinguish relevant dimensions of assistance, like giving lots of example solutions vs. withholding them (problems), giving vs. withholding immediate feedback, giving low vs. high variability examples. 2.Integrate: For each dimension: Collect, summarize, and integrate the relevant empirical and theoretical results from the research literature. 3.Mathematize: For each dimension: Characterize a set of conditions and parameters that can be used as part of a precise theoretical model that makes computable predictions about robust learning efficiency. 4.Test: Use the model to make a priori predictions about what level(s) of assistance under what conditions yield the greatest robust learning efficiency. Test those predictions in laboratory and in vivo experiments. A key goal of PSLC’s theory wiki is to get researchers working together, both within PSLC and within the broader learning science and education research community, to perform the gargantuan effort implied by steps 1 and 2. illustrates how we have carried out steps 3 and 4 with respect to the “practice spacing” dimension of assistance (Koedinger et al., 2008; Pavlik & Anderson, 2005). Figure. In the lower left is the assistance curve for the practice-spacing dimension. The top-level equation that generates the curve is shown above where eff m is the y-axis and m is the y-axis. Other equations, not shown, map from m to the variables that have m as subscript: p m, g m, and t m. An output of step 3 is a mathematical function (or set of functions) that can produce an “assistance curve”. As shown on the left in, this curve has an inverted-U shape for most reasonable values of the parameters in the equation (shown on the right). Consistent with notions like zone of proximal development described above, we suspect this inverted-U form will characterize most assistance dimensions. But, the key to resolving the assistance dilemma is creating the mathematical equations and parameters that generate the inverted-U in way that is consistent with cognitive theory and with available empirical data. Drawing on a number of PSLC projects and data from many domains, we have made considerable progress on a second dimension of assistance, the example-problem dimension -- see the Coordinative Learning section. The generation of the mathematical equations for this dimension (step 3) are being driven in part by our SimStudent model (Matsuda et al., 2008) -- see the Data Mining, Knowledge Representation, and Learning section. More generally, use of the Assistance Dilemma has driven analysis and interpretation of many other PSLC projects, some of which are described below (e.g., in the Interactive Communication section). Need predictive theory: when does assisting performance during instruction aid vs. harm learning Need more in vivo experiments to produce better theory & resolve dilemma desirable difficulty; germane load crutch

40 Home- work Exploring dimensions of instructional assistance Problem Solving Examples Added Untutored Tutored Low Feedback assistance Higher Example assistance Better learning Prior studies Best? New studies Better Learning Prior studies

A step toward resolving assistance dilemma … w ith very broad impact implications! GiveElicit Explicit rules Implicit: Example solutions Lecture Homework Current instruction: Gives rules & elicits solutions Better instruction: Gives solutions & elicits rules/concepts –Demonstrated in many lab & in vivo studies –Machine learning simulated students Worked example Self- explanation Instructor options

42 Summary of Learning Principles Lots of lists of principles … –6 Cognitive Tutor Principles –6 Multimedia Principles –See PSLC wiki for others … Should be Based on both: –Cognitive theory –Experimental studies Need Cognitive Task Analysis to apply –Domain general principles are not enough –Need to study details of how students think & learn in the domain you are teaching

43 EXTRAS

44 2. Communicate the goal structure underlying the problem solving Successful problem solving involves breaking a problem down into subgoals “Reification” = “making thinking visible” –Make goals explicit in interface

ANGLE: Tutor for geometry proof Working forward Working backward Extending physical metaphor: search “space” & “paths” Scaffolding diagrammatic reasoning

46 Common Error in ITS Design New notation = new learning burden –Examples: Goal trees, visual programming languages Alternative: Use existing interfaces as scaffolds –Adding columns in a spreadsheet Used in Geometry Cognitive Tutor –Writing an outline for a final report Used in Statistics OLI course

47 Principle 2a: Create “transfer appropriate” goal scaffolding interfaces Avoid creating new interfaces to scaffold goals –Worth it when: Alternative: Find an existing interface to use for goal scaffolding –“Transfer appropriate” because interface is useful outside of tutor –Cost of learning interface is low or pays off Cost of learning new interface Savings in domain learning due to goal scaffolding <

48 3. Provide instruction in the problem-solving context Research: context-specificity of learning –This is how students learn the critical “if-part” of the production rule! Does not address exactly when to provide instruction –Before class, b/f each problem, during? LISP tutor: –Before each tutor section when a new production is introduced –And as-needed during problem solving Cognitive Tutor Algebra course: –Instruction via guided discovery & as demanded by students’ needs or when they do not discover on their own

49 3a. Use “transfer appropriate” problem solving contexts Use authentic problems that “make sense” to kids –Intrinsically interesting, like puzzles –Relevant to employment, probs about $ –Relevant to citizens, rate of deforestation Why? –Motivation: Inspire interest –Cognitive: So students learn the goal structures & planning that will transfer outside of the classroom

50 4. Promote an abstract understanding of the problem-solving knowledge To maximize transfer, want those variables in if-parts of productions to be as general as possible! Reinforce generalization through the language of hint & feedback messages Cannot be simply & directly applied –Trade-off between concrete specifics vs. abstract terms students may not understand

51 5. Minimize working memory load ACT-R analogy requires info to be active in working memory to learn new production rules –Minimize info presentation to only what is relevant to current problem-solving step Impacts curriculum design as well as declarative instruction –Sequence problems so prior productions are mastered before introducing new productions Supported by research on “Cognitive Load” Theory –Eliminate extraneous load, leave intrinsic load, optimize germane load

52 Contiguity Principle Which is better for student learning? A. When corresponding words & pictures are presented far from each other on the page or screen B. When corresponding words & pictures are presented near each other on the page or screen Example: “Ice crystals” label in text off to the side of the picture or next to cloud image in the picture B. Near Why? Students do not have to use limited mental resources to visually search the page. They are more likely to hold both corresponding words & pictures in working memory & process them at the same time to make connections.

53 Redundancy Principle Which is better for student learning? A. Animation & narration B. Animation, narration, & text Example: The description of lightning occurs in pictures, is spoken. Text with the same words is present or not. A. Animation & narration Why? Text & animation are both processed in visual working memory and may overload it. Further, students eyes may be looking at the text when they should be looking at the animation. So, leave out the text which is redundant.

54 Personalization Principle Which is better for student learning? A. Formal style of instruction. B. Conversational style of instruction Example: “Exercise caution when opening containers that contain pyrotechnics” vs. “You should be very careful if you open any containers with pyrotechnics” B. Conversational Why? Humans strive to make sense of presented material by applying appropriate processes. Conversational instruction better primes appropriate processes because when people feel they are in a conversation they work harder to understand material. Note: Recent in vivo learning experiment in Chemistry LearnLab did not replicate this principle.