1. Uncovering the Problem-Solving Process: Tamara van Gog, Fred Paas, & Jeroen J. G. van Merriënboer I 3 CLEPS Workshop/Mini-conference, August 29, 2005 Cued Retrospective Reporting, Eye Tracking, and Expertise Differences

2. Overview Experiment: -Theory -Design ­Comparison of 3 verbal methods ­The 3 methods & expertise differences ­Uncovering expertise-related performance differences through eye movement data ­Present limitations and future research ­Discussion

3. Theory Use of process-tracing techniques to uncover problem-solving processes in order to advance / inform: -Psychological theory -Expert systems -User-system interaction, But also -Instructional design e.g., design of process-oriented worked examples

4. Theory From the literature (Kuusela & Paul, 2000; Taylor & Dionne, 2000) : + of concurrent reporting (“think aloud”): more information on actions taken + of retrospective reporting: more information on rationale for actions taken and strategies that control the process Needed: A method that combines + & + : Cued retrospective reporting based on a record of eye movements & mouse/keyboard operations?

6. Design Within-subjects, 26 participants, electrical circuits troubleshooting tasks: Seq.Condition + Tasks 1CR 1+2CRE 3+4RR 5+6CRR 7+8 2CRE 3+4CRR 7+8CR 1+2RR 5+6 3RR 5+6CR 1+2CRR 7+8CRE 3+4 4CRR 7+8RR 5+6CRE 3+4CR 1+2 CR = concurrent reporting; CRE = concurrent reporting with eye tracking; RR = retrospective reporting; CRR = cued retrospective reporting.

7. Comparison of 3 Methods: Hypotheses 1. Concurrent reporting (CR): more ‘action’ info than RR 2. Retrospective reporting (RR): more ‘why’, ‘how’, & ‘metacognitive’ info than CR 3. Cued retrospective reporting (CRR): -> more ‘action’ than RR -> more ‘why’, ‘how’, & ‘metacognitive’ than CR

8. Comparison of 3 Methods: Analyses Segmentation based on speech  sentences / utterances (preceded & followed by a pause) Coding scheme task-oriented main categories: ‘action’ ‘why’ ‘how’ ‘metacognitive’ 20% of protocols scored by 2 raters: kappa =.79  good; proceeded with 1 rater Analyses on nr. of codes on main categories, obtained by summing codes on subcategories

9. Comparison of 3 Methods: Results Friedman Tests with Conover (1999) comparisons CR vs RR: as hypothesized: ‘action’  CR >RR however: ‘why’ and ‘how’  CR > RR, and ‘metacognitive’ CR = RR CRR vs RR: as hypothesized: ‘action’  CRR >RR ‘why’: CRR = RR ‘how’ and ‘metacognitive’: CRR > RR

10. Expertise Differences: Explorative 5 “highest” and 5 “lowest” expertise participants (from 26). Determined by performance efficiency: “highest”: higher performance, lower mental effort, lower time-on-task “lowest”: lower performance, higher mental effort, higher time-on-task -Differences in elicited information? -Differences in preferences/experiences? (open-ended debriefing questions)

11. Expertise Differences: Elicited Information Differences in elicited information? (Mann-Whitney U Tests) CR: ‘how’ and ‘metacognitive’ info: “lowest” > “highest” RR: ‘why’ info: “highest”> “lowest” ‘how’ info: “lowest” > “highest” CRR: ‘action’ and ‘metacognitive’ info: “lowest” > “highest”

12. Expertise Differences: Experience Differences in preferences/experiences? “lowest”: experience: CR  (4/5) preference: CRR > CR & RR(4/5) “highest”: no differential experiences/preferences Mediating factors mentioned re. experience / preference, by both “lowest” and “highest”: -Time-on-task -Cue

13. Studying Expertise-Related Performance Differences: Eye Movement Data 1 Eye fixation data provide insight in the allocation of attention, and hence differ with expertise  Research use: provide information about the problem-solving process at a finer grained level than verbal protocols?  (Ultimate) educational use: guiding novices’ attention? 1 Data from Van Gog, Paas, & Van Merriënboer (2005), Applied Cognitive Psychology

14. Eye Movement Data: Participants & Procedure Same 5 “lowest” and 5 “highest” expertise participants Data collected in first 3 phases of the process: 1.Problem orientation (until pushing switch to observe circuit behavior) 2.Problem formulation and action decision 3.Action evaluation and next action decision % time spent on phase, mean fixation duration (MFD), and in 1 st phase fix. related to faults

15. Only 3 Volt Short-circuit Task

16. Eye Movement Data: Results Phase 1: problem orientation (Mann-Whitney U Tests, 2-tailed, α =.10) % of time: “highest” > “lowest” MFD:“lowest” > “highest” % fixations on battery:“highest” > “lowest” Gaze switches short-circuit:“highest” > “lowest” (NB: only trend)

17. Eye Movement Data: Results Phase 2: problem formulation & action decision (Mann-Whitney U Tests) % of time: “highest” = “lowest” MFD:“highest” = “lowest” MFD First ½: “highest” > “lowest” MFD Second ½: “highest” = “lowest”

18. Eye Movement Data: Results Phase 3: action evaluation & next action decision (Mann-Whitney U Tests) % of time: “highest” > “lowest” MFD:“highest” = “lowest” MFD First ½: “highest” = “lowest” MFD Second ½: “highest” = “lowest”

19. Eye Movement Data: Results MFD over phases (Friedman + Nemenyi post-hoc): n.s. for “lowest”; “highest” 1 3.1

20. Limitations -CRR and fabrication? -Cue: combination of eye movements AND mouse/keyboard operations -Only quantitative analyses of protocols -Eye movement data: distinction of phases -Performance efficiency measure: very relative distinction (lowest and highest within this group of participants) -Small nr of participants in analyses related to expertise differences

21. Future Research -Qualitative differences between CRR and RR? -Cue: different effects with only eye movements OR mouse/keyboard operations? -Cue: technical optimization? -(RR/)CRR: effects of other prompts? -Further study of performance efficiency measure to distinguish expertise levels -Replications with larger N

22. Thank you for your attention! tamara.vangog@ou.nl