© 2012 Aptima, Inc. The Science of Game-based Training Effectiveness 29 March 2012 Krista Langkamer Ratwani Kara L. Orvis

© 2012 Aptima, Inc.2 “It is fair to say that we currently ask young infantry [personnel] to function reliably across more domains than we ever have in the past, and the defense community has responded by inundating our forces with new technology and training to meet this challenge. Although this focus stems from the admirable desire to ensure that our forces receive the latest and greatest training and technology prior to combat, it has resulted in a patchwork of training efforts and technologies that have been implemented without demonstrating substantial proof of effectiveness.” - Marine Corps Gazette (2012) Background

© 2012 Aptima, Inc.3 Current State of Game-Based Training  Deployment of games making leaps and bounds –2006: $125M spent on game-based training (Blunt, 2007)  Research making baby steps –Little evidence that games lead to learning (Blunt, 2007)  Research focused on whether serious games were motivating –Training effectiveness studies are slowly leaking in but literature still disparate with mixed results (e.g. O’Neil and Perez, 2008)  Studying game-based training “in the wild” is hard  Results are published across many domains and sources –There continue to be many research questions regarding computer games and instruction (Tobias et al, 2011)  Little understanding of “why” serious games work

© 2012 Aptima, Inc.4 Contributors to Training Effectiveness Campbell & Kuncel, 2001; Colquitt, et al., 2000; Mathieu, et al., 1992; Salas & Cannon-Bowers, 2001 The Training Program The Trainee The Situational Context

© 2012 Aptima, Inc.5 The Training Program: Things to Consider  What game features are most useful for targeting specific learning objectives? (Wilson et al., 2009)  Feedback –How often and what type of feedback should be provided?  Scaffolding –How can scaffolding approaches be incorporated into game- based training? Learning Objectives… Should Drive… Design Decisions!

© 2012 Aptima, Inc.6 The Training Program – Empirical Evidence Example  Participants: 157 students playing a game that manipulated the level of Game Fantasy (low vs. high fantasy conditions)  Fantasy significantly impacted Declarative Knowledge and Application F(1, 64) = 4.58, p =.04 F(1, 65) = 6.51, p =.013 Orvis et al., 2009

© 2012 Aptima, Inc.7 The Trainee: Things to Consider  General training rule: Training should be difficult enough to increase motivation and performance, but not beyond the trainee’s capability –Attribute by treatment (ATI) interactions have been a focus within the general training research (e.g., Gully et al., 2002)  Examples of Trainee Variables to Consider –Personality  Prior research in military settings has shown a positive relationship between personality variables and training effectiveness (e.g., Driskell et al., 1994) –Prior videogame experience  Research has shown that less than 50% of college age students play video games on a regular basis (e.g. Orvis, Orvis, Belanich, and Mullin, 2005)  Both general and genre specific game experience impacts how students react to the game (e.g. Orvis, et al., 2005)

© 2012 Aptima, Inc.8 The Trainee – Empirical Evidence Example  Participants: 139 students playing versions of the game G-DIS which manipulated task difficulty across 6 missions Bauer et al., in press β14 = -.15, t(826) = -2.15, p =.032

© 2012 Aptima, Inc.9 The Situational Context: Things to Consider  Training environment impacts effectiveness of training program (e.g., the game)  Organizational analysis is a critical component to effective training – even for serious games –Organizational or unit commitment to the training –Resources (e.g., time) –Involvement of the leader –Whether units are training together –KSA transfer environment –Available pre-training

© 2012 Aptima, Inc.10 The Situational Context – Empirical Evidence Example  Participants: 165 Soldiers (representing 9 platoons) playing VBS2  Amount of Preparation prior to playing game and Amount of Leader Involvement during the game significantly impacted Unit Performance variables F(1, 126) = 10.23, p <.01 F(2, 137) = 4.02, p <.05 Ratwani et al., 2010

© 2012 Aptima, Inc.11 The Importance of Evaluation  Evaluation is needed to actually assess learning  Common methods of assessing training effectiveness (Marine Corps Gazette, 2012) –Performance on the battlefield –Subjective assessments by trainees  Best practices –Longitudinal –Pre- and post-evaluation (assess skill level before and after training) –Comparisons to other training methods (e.g., live training; classroom training) –Good measurement  Include measures of the 3 factors just discussed  Identify and develop standards of performance

© 2012 Aptima, Inc.12  Continue to test the effectiveness of serious games  Understand how and why games lead to learning –Design characteristics –Learner characteristics –Context  Use scientific principles to build games that teach Implications and Recommendations Optimal Learning with Motivated Trainees Engaged Trainee Specific and Timely Feedback Adaptive Scenarios Ideal Learning Environment and Measurement Approach Necessary Fidelity Requirements Targeted Performance Measurement Clear Training Objectives Subject Matter Expert Input Task and Organization Analysis

© 2012 Aptima, Inc.