1. Measuring Learning Leah J Rowe Air Force Research Laboratory
711 Human Performance Wing
Measuring Learning

2. Overview Continuous learning About us The end game
Instructional design theory
Where we are going
Approach for continuous learning
Research and training on all levels
Competency based approach
Training research testbeds
Family of complimentary trainers
Live, virtual, and constructive training
Best Practices: Putting it all together

3. Continuous Learning (711HPW/RHAS)
Air Force Research Laboratory (AFMC)
711 Human Performance Wing
Human Effectiveness (RH)
Warfighter Readiness Research Division (RHA)
Develop mission relevant content for learning
Measure training outcomes in complex mission environments
Accelerate development of mission critical competencies
Optimize training value of live, virtual, and constructive environments

4. 4/12/2017
4/12/2017
About us
CHALLENGE: Training and Readiness requirements are outpacing capacity and existing opportunities to train
MISSION: Research human performance methods and technologies that provide the Warfighter the necessary knowledge and skill to dominate their operating environment 4

5. Instructional Design Theory
Instructional Theory
Instructional Event Theory
Instructional Analysis Theory
Instructional Planning Theory
Instructional Building Theory
Instructional Implementation Theory
Instructional Evaluation Theroy
Instructional Design Theory
Reigeluth & Carr-Chellman, 2009 p.9
Design theory is aimed at facilitation generative outcomes… It assists in the creation of something.1
Learning Sciences: A hybrid discipline that includes learning theory and instructional-event theory.1
Purposely multidisciplinary and not as interested in goals as in the use of certain kinds of instructional methods to shed light on certain kinds of learning processes.1 5
1 Reigeluth, C. M., & Carr-Chellman, A. A. (2009). Instructional-Design Theories and Models: Building a Common Knowledge Base (Vol. III). New York: Routledge.

6. Learning Management Systems
LMS Video 6

7. Where We Are Going: Game Changing Learning S&T
Metrics and scenarios
Tailored needs assessment/gap analyses
Cross-domain readiness assessment
Proficiency tracking and feedback
Near
Proficiency-based training
Evidence-based fidelity assessments
Cross domain after-action reviews
Team and team-of-teams assessment
Family of complementary trainers
Mid
Adaptive, distributed content and assessment
Scenario authoring, scripting, and metrics definition standards
Common data exchange, control, management
Managed learning/feedback across LVC contexts
This chart style is optional for briefers who are doing narrow, deep dives.
All briefers should convey where their research is going over the duration of their work. Where does it feed? Is it transitioned? The information in this chart (whatever form) should link to one of the last charts in the briefing --- “What’s next”
Near term = 1-3 years
Mid term = 4-9 years
Far Term = 10 years and beyond
Globally distributed learning
Seamless integration of LVC environments and players
Adaptive training and operational environments
Portable, personal learning and occupational records
Predictive performance models anticipate gaps/refreshers
Far

8. Training Research: The Big Picture
The simulators are a means to an end
Each systems is in place to execute training research
Capitalized on a training research model introduced by Schreiber and Bennett during the Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study2.
76 teams of F-16 pilots and Air Battle Managers participated
Represents the largest DMO effectiveness study known to exist today
Measures included: objective data from the simulators, multiple participant surveys, subject matter expert ratings of performance, and knowledge structure tests
DMO training lead to significant improvements in warfighter performance between the pre-training and post-training assessments
2 Schreiber, B. T., Bennett, W. Jr. (2006a). Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Volume I: Summary Report (AFRL-HE-AZ-TR Vol I). Mesa AZ: Air Force Research Laboratory, Warfighter Training Research Division.
Distribution A: Approved for public release

9. Approach for Continuous Learning
OBJECTIVES
Warehousing
What was learned
Where
How much
How often
Adapt Conditions
Based on Performance
& Diagnosis
Readiness/Proficiency Monitoring
Competencies
Knowledge
Skills
Experiences
Gaps …
Adaptation
Diagnosis
Learning
Opportunities
AAR
FEEDBACK
CONDITIONS
Technical Approach (How we do what we do)
Controlled studies with laboratory, operational, industry, and academic partners
Three phased development, demonstration and evaluation
Component and integrated evaluations
Controlled laboratory and testbed evaluations
Mission relevant testbeds and tasks
Evaluation trials at stakeholder and operational units
Integrated operational mission training venues
Demonstration and evaluation trials on instrumented ranges
Large force training, readiness, and exercise venues
Continuous involvement of ops personnel as research participants promotes transition and stakeholding in results
Record
Measure
Match Opportunities to Objectives
Scenarios Instructional Events  Structure & Triggers  Strategies & Principles Support resources
Data
ASSESSMENT
Match Measures to Objectives & Conditions
Individual/Team/Team of Teams
Objective and Subjective
What/How to Measure

10. Research & Training on All Levels
Individual
Team Of
Teams
Facilitate the development of strategies, methods, and competency based tools to promote learning
Research can be executed by assessing training at a micro level or on a grand scale
Environment that provides unique immersive training and research opportunities
Provides the capability to demonstrate robust performance assessment within an integrated theater combat environment

11. Training Research: Competency Based Approach
Use a building block or competency based approach. Training curricula, scenarios, and systems are mapped to the knowledge, skills, and experiences that they are able to provide to trainees.
Mission Essential CompentenciesSM 3,4: Higher-order individual, team, and inter-team competencies that a fully prepared pilot, crew or flight requires for successful mission completion under adverse conditions and in a non-permissive environment
Supporting Competencies: High-level skills that are demonstrated in the context of operational conditions and constraints. Some support a specific MEC, while others support multiple MECs.
Knowledge & Skills: Knowledge is defined as information or fact that can be accessed quickly under stress. Skill is defined as a compiled sequence of actions that can be carried out error-free under stress.
Experiences: Developmental event during training and/or career necessary to gain knowledge or skills, or practice a MEC under operational conditions
3Alliger, G. M., Beard, R., Bennett, W., & Colegrove, C. M. (2012). Understanding Mission Essential Competencies as a Job Analysis Method. In M. A. Wilson, W. Bennett, S. Gibson, & G. M. Alliger, The Handbook of Work Analysis Methods, Systems, Applications, and Science of Work Measurement in Organizations (pp ). New York: Routledge.
4 Colegrove, C M and W Bennett. Competency-Based Training: Adapting to Warfighter Needs. Technical Report: AFRL-HE-AZ-TR Mesa: Air Force Research Laboratory Human Effectiveness Directorate Warfighter Readiness Research Division, 2006.

12. Training Research Testbeds
Provide innovative and highly immersive training research to develop efficient and effective training warfighter training solutions
What are the supporting competencies, knowledge, and skills, for effective team and team of teams training?
What are the training needs, gaps, and opportunities in a networked training environment for the trainees?
What is the most effective training structure to effectively train a team-of-teams in a testbed that will provide enhanced training capabilities to the warfighter?
Testbeds built to assess Continuous Learning
Distribution A: Approved for public release

13. Family of Complimentary Trainers
Assess alternative training strategies and methods for enhancing RPA training by offering solutions that match the level of fidelity required for specific types of training
Effectively and appropriately using training resources
Leverages both commercial and government off the shelf solutions to build cost effective training gap interventions that allow us to research and develop innovative strategies for continuous learning
Continuum of Training

14. Distributed Mission Operations
Virtual Constructive
“Train the way we intend to fight”, enabling Air Force warfighters to maintain combat readiness and conduct mission rehearsal in an environment as operationally realistic as possible4
4 Ales, R. & Buhrow, S, (2006) “The Theater Air Ground System Synthetic Battlespace”, Proceedings of the I/ITSEC 2006 Conference, Orlando, Florida, USA, November 2006. 14

15. LVC Definitions Virtual Constructive Live
Live – human beings operating actual operational weapons system, in a typical combat environment or training scenario
Virtual – real people operating simulators of any kind (i.e., a tank driver in a sim, a JTAC in a dome, or simulated AOC
Constructive – computer generated entities, weapons, systems used to enhance the training environment
Live – real aircraft in space, burning real jet fuel.
Virtual – simulators
Constructive – computer generated
The state of LVC to date is one of integration .
Many DOD and Industry entities are “doing LVC”. They are doing the easy parts first, which is taking live entities and injecting them into the virtual environment.
They are also creating their own stovepipes, which will bring back the same problem we have with DMO – we all did DMO, and now we have to spend 10 years and billions of dollars to connect everyone together.
The next slide shows some examples of LVC integration in the Air Force…

16. Vision: Continuous Learning in LVC Ops
A capability where aiding and learning activities are interwoven seamlessly with live operations on-demand
The end state optimizes individual and team performance through the design, monitoring, support, management, and enhancement of learning and developmental/recurring experiences for current and future operations
The foundation is in science and technology related to:
Modeling learning and action
Enabling proficiency on demand
Managing the learning and experiences that occur across the career of warfighters as individuals and as members of teams in complex decision-centric environments
The Future State is Performance-Enabled Learning and Readiness

17. Best Practices: Putting it all together
Utilize the methods in the “Within-Simulator Training Effectiveness” study as best practices to develop a robust training research program.
Competency-Based Approach
Mission Essential Competencies
Individual, team, and inter-team under adverse conditions
Objectives
Scenario-Based Training
Entire Kill-Chain Exercises
Joint and Coalition Training
Conditions
Readiness
Proficiency
Lessons Learned
After Action Review
Performance Feedback
Assessment