An Examination of Learning Processes During Critical Incident Training: Implications for the Development of Adaptable Trainees Andrew Neal, Stuart T. Godley, Terry Kirkpatrick and Graham Dewsnap, Wendy Joung, and Beryl Hesketh 2006

Overview Purpose: Examine the process that trainees learn decision making skills during a critical incident training program. Discuss that rule and exemplar learning may be responsible for the acquisition of decision making skills. Results suggest trainees use both types of learning, and they can also be affected by different aspects of training structure and design.

Introduction Based off previous work by Ford and Kraiger (1995) Aim: Address gaps in research at a theoretical, methodological, and practical level.

Category Learning Historical viewpoint vs. Dual Process Model Issue: Majority of research supporting dual process models were conducted in university lab settings. o First goal – determine whether trainees use the same learning strategies as participants in a controlled lab setting.

Training Design and Structure If trainees do use both rules and examples, its important for trainers to understand how the design and structure of a training influences the use of these strategies. o Context and Practice

Practice Traditional research suggests: o Practice makes perfect! Dual Process Model o Practice allows fine tuning of decision making skills. o Common issue: Individuals will continue to use examples even if they have access to a well learned rule. Suggests context may be more effective.

Training for Adaptability Need for new techniques addressed by Feltovich et al. (1997). Exceptions are more likely to be recognized if individuals adopt a belief structure that emphasizes the complexity of task/domain. o Can be achieved through explicit instruction and learning from examples.

Overview of Experiments Experiment 1Experiment 2 Assess trainees reliance on rule and exemplar learning. Examine how aspects of training design and structure affect the use of rule and exemplar learning. Experiment 3 Determine whether explicit instruction and task practice can reduce the incidence of inappropriate generalization.

Experiment 1 Goal – Determine the extent to which trainees use rules and examples when learning to make tactical decisions. Lecture, training examples (feedback), then followed by test examples. o Used inclusion and exclusion items

Hypotheses Hypothesis 1: Performance on novel items will be significantly better than chance. Hypothesis 2: Performance on the inclusion items will be better than performance on novel items. Hypothesis 3: Performance on exclusion items will be worse than performance on novel items.

Method Setting – introductory training program Participants were 35 new recruits to New South Wales Fire Brigades. o 33- Male, 2 – Female o Average age of 30.9 o Average experience of 3.1 years Materials included a lecture (15 minutes), eight training examples, and 12 test examples which used a picture of a fire and along with an auditory script.

Analysis and Results Repeated measures ANOVA was completed along with Planned contrasts on confidence, accuracy, and reaction time. All hypotheses were supported. Results are consistent with the prediction that trainees would use both when learning strategies to make tactical decisions.

Experiment 2 Aim: Examine the effects of practice and context on the use of rules and examples. Hypothesis 1: Performance on novel items will improve from phase 1 to phase 2. Hypothesis 2: The differences in performance on the inclusion items and exclusion items will remain stable over practice. Hypothesis 3: The differences on the inclusion and exclusion items will be larger when the contextual features of the training and test items are matched.

Method Design – split item pool in half and counterbalance the order of the presentation of the two pools. Participants were 36 new recruits to the NSW Fire Brigades Materials used were the same as Experiment 1. Procedure – Lecture, then phase 1 was given test items, then phase 2 was given the test items.

Analysis Repeated measures ANOVA Planned contrasts were run, which assessed the effects of practice on the novel items and the effects of practice and context on the inclusion and exclusion items. Classification accuracy, reaction time, and confidence were the dependent variables.

Results Results supported all 3 hypotheses. Practice by itself doesn’t appear to be enough to prevent trainees from generalizing inappropriately from prior examples. Retrieval and use of prior examples were dependent on contextual overlap in training the test items.

What to take away… Trainers can enhance trainees ability to use general rules by increasing the number of practice examples. Trainers can enhance trainees ability to use examples by ensuring context from training matched transfer environment.

Experiment 3 Aim: Examine the effectiveness of training strategies designed to prevent trainees from generalizing a simple rule to cases in which it doesn’t apply. Used explicit instruction regarding exceptions to the rule. o Near and Far Exceptions

Hypotheses Hypothesis 1: Providing trainees with explicit instruction regarding exceptions to a simple rule will enhance performance on near and far exceptions. Hypothesis 2: Providing trainees with examples that illustrate exceptions to a simple rule will enhance their performance on near and far exceptions.

Methods 2x2x(3) Between group factors: o Instruction – Routine and exceptions instruction o Practice – Routine and exceptions practice Within group factor o Item type 16 items (8 – routine, 4-near exceptions, and 4 – far exceptions) Half of the participants (exceptions instruction group) received additional information regarding exceptions in the lecture.

Methods Participants included 91 recruits to the NSW Fire Brigades. o Had 16 weeks of training o All Male Procedure was the same as Experiment 2, except there was only one training and test phase.

Analyses Series of repeated measures ANOVA. Planned contrasts were run, assessing the effects of instruction and practice on the routine items, near and far exceptions. o Classification accuracy, reaction time, and confidence were dependent variables.

Results Providing general instruction regarding exceptions to the rule was more harmful than helpful. Trainees became fixated on exceptions. Possible Solution: Presenting exceptions in the set of practice examples might be more effective than explicitly instructing trainees about these exceptions.

General Discussion Trainees use both rules and examples. Results from lab setting do generalize to training contexts. Results supported that the learning processes would respond differentially to design variables, such as practice and context. o Can help trainers make design choices accordingly Such learning processes could be used in training needs analysis, and training design and evaluation.

Strengths Limitations Unsure of lasting effects Limited information in lecture, practice, and time to study material. Tasks used Simulated scenarios Effects could reflect how lectures and examples were constructed. Study Design which allowed us to be able to draw inferences about the results.

Discussion Questions Do you think such findings can be generalized to other areas in which critical incident training is used? Do you agree that we should integrate more learning strategies such as the dual process model into the training research? Would it increase trainee effectiveness? Study suggests results from lab setting can be generalized to a training context, do you feel this is just reflective of this study?