1. Cognitive Measurements to Design Effective Learning Environments 1 Open University of the Netherlands 2 New York University, USA Fred Paas 1 & Slava Kalyuga 2 I C L E P S WORKSHOP 2005 August 30 3

2. Overview COGNITIVE MEASUREMENT: GROUP-BASED INSTRUCTION Measurement of cognitive load Interpretation of performance, mental effort, and combined scores COGNITIVE MEASUREMENT: PERSONALIZED INSTRUCTION Diagnostic assessment of organized knowledge structures Applying combined efficiency measures in adaptive training

3. Basic Assumption If individuals are to learn effectively in a learning environment, the architecture of their cognitive system, the learning environment, and interactions between both must be understood, accommodated, and aligned Cognitive Measurements

4. Working Memory Sensory Memory Long Term Memory Cognitive architecture  Perceive incoming information Attend to information Limited capacity Limited duration Separate processors for visual and auditory information Permanently store all knowledge and skills in a hierarchical network (schemas) Unlimited capacity

5. Schema: categorizes elements of information according to the manner in which they will be used consists of a multi-dimensional web of interconnected nodes of information can be treated by WM as a single entity, and if the learning process has occurred over a long period of time, it may incorporate a huge amount of information can be processed consciously or automatically

6. Intrinsic represents the load that performing a particular task imposes on our cognitive system The Concept Cognitive Load Extraneous n not relevant for learning Germane n relevant for learning Determined by the number of information elements and their interactivity Determined by the manner in which the information is presented to learners Intrinsic Extraneous Germane Capacity max. Capacity min.

7. Cognitive Load Theory Instructional Techniques Decreasing Extraneous Load  Goal-free effect  Worked-example effect  Completion effect  Split-attention effect  Modality effect  Redundancy effect Minimizing Extraneous Load Increasing Germane Load  Variability effect  Self-explanation effect  Imagination effect  Interactivity effect Maximizing Germane Load  Sequencing effect  Fading support effect Managing Intrinsic Load

8. Measurement of Cognitive Load Objective measures  Task and performance  Secondary task  Psychophysiological Subjective measures  Rating scales

9. Goal-Specific Problem Solving Start Equations with goal as only unknown Equation with subgoal(s) and unknown(s) Equation with subgoal as only unknown Equation with goal and unknown(s) Problem solved Problem statement and equations in working memory Subgoal(s) added to working memory yes no Solve equations and add new known to working memory Given: A car that starts from rest and accelerates uniformly at 2 meters/s 2 in a straight line has an average velocity of 17 meters/s. Goal: How far has it traveled? Operators: s = v * t, v =.5V and V = a * t (V=final velocity, v=average velocity, a=accelaration, t=time, s=distance)

10. Goal-Free Problem Solving Start Stop search Equation with only one unknown Problem statement and equations in working memory yes no Solve equations and add new known to working memory Given:A car that starts from rest and accelerates uniformly at 2 meters/s 2 in a straight line has an average velocity of 17 meters/s. Calculate the value of as many variables as you can.

11. Measurement of Cognitive Load Objective measures  Task and performance  Secondary task  Psychophysiological Subjective measures  Rating scales Rapid RT Slow RT Cognitive resources to simple primary task Cognitive resources to complex primary task Fixed cognitive capacity Resources to secondary task

12. Objective measures  Task and performance  Secondary task  Psychophysiological Subjective measures  Rating scales Measurement of Cognitive Load

13. Heart rate variability Eye movements Pupil dilation

14. very, very low mental effort very, very high mental effort neither low nor high mental effort In solving or studying the preceding problem I invested: Measurement of Cognitive Load Objective measures  Task and performance  Secondary task  Psychophysiological Subjective measures  Rating scales

15. Subjective measures: Rating scales (NASA-TLX)

16. How to interprete performance and mental effort scores? Instructional Condition Performance (1-10) Mental Effort (1-10) ABCDABCD 22882288 92399239

17. M ental Effort P erformance M = P Efficiency = 0 high efficiency low efficiency Efficiency of Instructional Conditions

18. Motivation in Instructional Conditions M ental Effort P erformance Motivation = 0 high motivation low motivation

19. Instructional Efficiency/Motivation M ental Effort P erformance high motivation low motivation high efficiency low efficiency

20.  Find usable objective technique Intrinsic items 1- how easy or difficult do you consider probability theory at this moment? Extraneous items 2- how easy or difficult is it for you to work with learning environment? 3- how easy or difficult is it for you to distinguish important and unimportant information in the learning environment? 4- how easy or difficult is it for you to collect all the information that you need in the learning environment? Germane items 5- how easy or difficult was it to understand the solution in the last animation?  Use measures to personalize instruction  Distinguish between different types of load Measurement of Cognitive Load: Challenges

21. COGNITIVE MEASUREMENT: PERSONALIZED INSTRUCTION

22. Expertise reversal effect: Cognitive load effects depend on levels of learner expertise: instructional designs or procedures that are effective for novices may be ineffective for more proficient (expert) learners. Instructional implications: - instructional techniques need to change with alterations in expertise; - it is critical to have simple rapid measures of learner proficiency (performance and mental effort).

23. Real time (rapid online) diagnostic assessment of organized knowledge structures. Why organized knowledge structures (schemas)? Cognitive studies of expertise: organized knowledge base in LTM is central to cognitive processing (De Groot, 1946/1965, Chase & Simon, 1973); they affects the way we process information in WM and solve problems (Novices vs Experts).

24. Reading comprehension (linguistic schemas) This was the course that the student whom the school that was criticized by the newspaper expelled failed.

25. This was the course that the student whom the school that was criticized by the newspaper expelled failed

26. Solve for x: 5x = - 4

27. x = - 4/5 Solve for x: 5x = - 4

28. 5x/5 = - 4/5 Solve for x: 5x = - 4

30. Diagnostic cognitive assessment should be sensitive to different cognitive attributes sensitive to different levels of proficiency practically usable Typical time scale of cognitive processes: up to several seconds.

31. Rapid diagnostic approach: general: What is the highest level of organised knowledge structures (if any) a person is capable of retrieving and applying to the briefly presented material? first-step method: Presenting learners with a task for a limited time and asking them to indicate their first step towards solution.

33. A: multiply both sides by 3 2x + 1 = 3 B: subtract 1 from both sides 2x = 2 C: divide both sides by 2 x = 1

34. (2x + 1)/3 = 1 2x + 1 = 3 3x + 1 = 2 4x = 1 2x = 2 x = 1 3x = 1 x = 1/3 x = 1/4 A C BB CC

35. Scoring method Skipping steps reflects a higher level of proficiency: the learner has corresponding operations automated or is able to perform these operations mentally without writing them down. If a learner omitted some intermediate steps, she/he should be allocated an additional score for each skipped step. For 2x +1 = 3 scores 0 to 4: 4: x = 1 3: 2x/2= 2/2 2: 2x = 2 1: 2x + 1 - 1 = 3 - 1

36. Assessment design procedure Student model Schematic knowledge structures to be assessed Task model Patterns of tasks to elicit evidence about student schemas Evidence model Scoring procedure Measurement model

37. EXAMPLES. Arithmetic word problems, Marshall (1995): Change Schema: After 6 passengers had left the bus, 9 passengers remained. How many passengers were on the bus initially? Group Schema: Peter's book contains 50 pages. Peter read 15 pages in the morning. In the afternoon, he read the remaining pages and finished the book. How many pages did Peter read in the afternoon? Vary Schema: A car travelled 60 kilometres in an hour. If the car continued to travel at the same speed, then how far would it travel in 5 hours? Restate Schema: Water is mixed with juice concentrate in the ratio 2 : 1. How many litres of water need to be mixed with 10 litres of concentrate?

38. Rapid verification diagnostic technique: Presenting learners with a series of possible task solutions for a limited time and asking them to rapidly verify the suggested solution steps. Physics (kinematics) vector addition motion problems A ship is traveling at 7 m/s. A dog runs across the deck at the same speed in a direction of 60° relative to the direction of motion of the ship. What is the velocity of the dog relative to the sea?

40. E = P/R Combined efficiency measures

42. Using rapid diagnostic techniques in adaptive training for dynamic learning task selection (tailoring levels of task complexity and learner support).

43. Selection algorithm governing the selection of learning tasks with different levels of difficulty (stages 1-4) and support (worked examples, completion tasks, and conventional tasks/problem solving exercises). Adapted from Kalyuga and Sweller (2004)

46. Adaptive training based on combined efficiency measures 0-2-3-45 +10-2-34 +2+10-23 +3+2+102 +4+3+2+101 54321 Mental effort Performance