1. Research and Best Practice for the Design and Delivery of Training
Richard Clark
ICT IMMERSIVE TRAINING WORKSHOP

2. Important Problems and Questions?

3. Questions for Today’s Workshop
I will discuss the five questions:
What have we learned recently that could help us solve pressing problems in training?
Do some technologies or media influence learning or motivation more than others?
What kind of training leads to adaptable skills and knowledge?
How can instructors and developers help trainees learn study skills?
How can we decide whether training should be delivered in the classroom or by multi media?

4. 1. Training Impact Studies
First , the Bad News
In past two decades the best training resulted on only a 20% increase in learning (meta analysis results).
In task analysis, top experts only provide 30% of information about how they perform tasks.
50% of trainees are wrong when asked how much they learned from training on reaction forms.
Providing learner control over the sequencing of lessons or content harms learning.

5. Training Impact Studies
Technology-based, distance learning is no more effective than classroom-based, instructor-led.
Requiring students to discover what they need to learn (e.g. by solving problems to learn the solution) results in about 40% less learning than demonstrating how to solve problems.
The most vulnerable students are harmed the most.
Multi media programs often distract students and reduce learning for the most vulnerable students.

6. Training Impact Studies
Evidence that the way we most often design Serious Games makes them significantly less effective for learning and more expensive than more direct ways to train.
Adjusting training for different learning styles does NOT increase learning (e.g. Multiple Intelligences, Myer’s Briggs, Visual or Verbal Ability).
30% of feedback strategies damage performance, another 30% have no impact.
WHY IS THE EVIDENCE SO COUNTER-INTUITIVE AND DISMAL?

7. 2. Our Mental Architecture Limits Learning
Thinking is limited to 3-4 ideas at one time
Evolution prevented rapid, self destructive learning.
Emotionality and stress take up thinking space.
When we exceed mental capacity we “zone out”
“Daydreaming is pleasurable and unconscious.
We have two different knowledge systems
One is conscious and takes up space in thinking
One is unconscious and does not require thinking space

8. We have two knowledge systems
Knowledge about What and Why (Declarative)
Conscious, easier to learn and forget, can be wrong.
Helps us imagine and handle novelty.
About 10%- 30 % of adult knowledge.
Knowledge about When and How (Procedural)
Unconscious, hard to learn and forget, mostly right.
Helps us automate strategies that work without taking up any of the 3 to 4 thinking spaces.
About 70% - 90% of all adult knowledge.

9. Problems caused by two knowledge systems
We are unaware of procedural knowledge and so we focus training on declarative knowledge.
People do not learn how to think or decide.
They learn 20% & get the rest by trial and error.
Experts (as SMEs or trainers) are 70% UN-aware of their own mental strategies
SMEs only provide 30% of “how” but believe they’ve given 100%
SOLUTION?

10. Possible Solutions to Problems
Three New Training Strategies:
Use Cognitive Task Analysis to capture unconscious expertise.
Use New “Direct Instruction” to design training.
Avoid Cognitive Overload with new Screen Design
Lets look at all three in the context of ADDIE

11. Effective Training We currently use the ADDIE model:
Analysis (What and who should be trained?)
Design (What plan is best for this training?)
Development (What TSPs, media and materials?)
Implementation (How should we deliver training?)
Evaluation (How will we measure success?)
Lets look at what we’ve learned about each stage:

12. I. Effective Training: Analysis
SMEs Role in Analysis? Two issues to consider –
SMEs must have successfully and recently performed the tasks being analyzed.
They most often have obsolete skills because the operational environment evolves quickly.
Even the best SMEs are only 30% aware of how they perform the mental part of even routine tasks.
Convincing evidence from many different fields – they can only describe what they can watch themselves do.
Incomplete or wrong information in training puts trainees at risk in operational environments.

13. I. Effective Training: Analysis
The new cognitive task analysis
Interview captures unconscious expertise
Increase to 80% accuracy (Chao, 94; Clark, 2006)
When used in training, performance impact doubles (from 20% to 45% - Lee, 2004)
Recent evidence that DL dropout decreases.
Dr. Ken Yates will describe and provide examples this afternoon

14. II. Effective Training: Design
Design (What plan is best for this training?)
We have a number of challenges in design:
ADDIE does not dictate training methods and the one’s we select are often ineffective.
Asking trainees to “construct” or “discover” how to do something in order to solve a problem is least effective method.
Reviews of training studies for the past 50 years indicate that “direct instruction” is best.

15. II. Effective Training: Design
David Merrill has reviewed all effective direct instruction training systems (military and civilian) and suggests that they have five key elements:
1) Authentic Problems (from mission environment)
2) Connections to trainees prior knowledge (analogies, examples)
3) Demonstration of “when and how to act and decide” captured, from highly successful experts with recent experience in current operational environment.
4) Part and whole-task practice with immediate corrective feedback
5) Transfer and automation help (field exercises, immersive simulations, serious games)

16. II. What training leads to adaptability?
Trainees must first learn why and how to perform in an authentic, routine setting.
Providing analogies that relate the how and why to previous experience increases adaptability.
“How to” knowledge must be applied in increasingly novel environments (varied practice).
All practice must be “hands on” with supportive and corrective feedback.
When trainees are asked to explain why the strategy they are using did not work, flexibility increases.

17. II. What training leads to Study Skills?
TEACH STUDENTS SIX L2L STRATEGIES:
Take notes and outline key points by translating course information into their own words.
Connect what they are learning to the imagining of previous experiences and to familiar examples and analogies.
Focus their attention on key points.
Ask higher order questions (Why? What Evidence?) to check their deeper understanding of course content.
Write summaries of main ideas in their own ideas and check.
Think about alternative points of view (What if? How does the enemy or that other person think about it?).

18. III. Effective Training: Development
Development (What media and materials?)
Best practice requires that we begin with a plan (design) and then develop media and materials based on the training plan.
Now we most often begin with development of materials without a plan.
Conclusive proof that media influence cost and access to training but NOT learning.
SMEs influence accuracy of task description
Design dictates training methods.
Development influences access, time and cost

19. III. Effective Training: Development
Development (What media and materials?)
3. Design and Development must be compatible – all five direct instruction elements must be in all training exercises – Yet:
If we don’t capture when and how to make decisions during analysis, we can’t teach essential knowledge
Our DL software makes practice and immediate corrective feedback mostly impossible
Serious games are most often based on discovery methods but could be based on direct instruction
Multi-media training often hoses students with too much information that is not relevant to objectives

20. Multimedia Design Give information with narration not print
Prevent novice or intermediate trainee control over:
Sequence of lessons, modules or information
Learning activities, for example, extra information or additional problems
Allow limited control of pacing
When presenting process or procedures, start with overview and use pointer words such as “first,” “second,” or “as a result.”

21. Development: Multi Media Development
10 Principles (Students learn better when …) % increase Tests
Eliminate extraneous visuals and sound 31% 5
Do not read text on screen – narrate without text 23% 5
Place labels next to part of graphic it describes 38% 5
Simultaneous narration and video – do not separate 42% 5
Student pacing better than system pacing 31% 3
Conceptual information just before “how to” 27% 5
Graphics + Narration not text + animation 32% 7
Add pictures to Narration - not narration alone 42% 11
Conversational style “you or we” - not formal 36% 11
Human voice better than machine voice 25% 3
Mayer, R. E. (2009). Multimedia learning. 2nd Ed. New York: Cambridge University Press.
Mayer, 2009

22. IV. Effective Training: Implementation
Implementation (How should we deliver training?)
How to decide between classroom and electronic media?:
Can sensory modes and application context be simulated by available delivery media? (video, computer, instructor)
Can practice be observed and feedback provided by computer?
If “no” to any part of 1) or 2), offer classroom or synchronous live instruction for those parts.
If any part of 1) and 2) are “yes” Use ingredients method of cost/utility analysis – if computer is cost beneficial, use it.

23. V. Effective Training: Evaluation
Evaluation (How will we measure success?) –
Kirkpatrick 4 Level Model:
Reactions only tell us about student confidence and values – NOT learning or performance.
Learning assessment must tap both conceptual knowledge and application skill
Reactions can be negative and learning positive
Learning may not transfer
Transfer assessment must capture performance not opinions about performance
Reactions, learning and transfer may be positive but the problem we were trying to solve may not be solved. .

24. Questions? Comments?

25. THANK YOU! CLARK@USC.EDU