Instructional Design, Transfer of Training & Program Evaluation Barbara A. Bichelmeyer, Ph.D Associate Professor Department of Instructional Systems Technology Indiana University Middle East Technical University Ankara, Turkey 21 June 2005

A sampling of my time in Turkey  Dolmabahce Palace  Taksim Square  Galata Tower  Bogazici University tour  Basilica Cisterns  Haghia Sophia  Blue Mosque  Topkapi Palace  Grand Bazaar & Spice Bazaar  Whirling Dervishes  First National Assembly  Anatolian Civilization Museum  Old Castle view  METU tour

What I have learned so far…  Turkey is 6,500 years old  Crossroad of East and West  36 sultans during the Ottoman Empire  Attaturk revolutionized everything  Turkey has changed constantly to survive and grow  Technology is changing everything here and everywhere  People from different countries are more alike than they are different  Just like countries, people must change in order to survive and grow

My Assumptions About You  A wide-range of experiences as educators  Working in educational institutions  Expertise in IDT and related disciplines of education  Various levels of academic experience  You could be giving a lecture to me…

What Experience Do I Have to Offer?  20 years experience: - Teacher - Instructional designer - Performance consultant - Professor - Evaluator  Work Settings: - Secondary school - Higher Education - Corporate - Government - Military - Non-profit - Health Care

What Can I Possibly Tell You?  Technology has changed the world so that the amount of information is now so great that people can never again “know all that is known”  People must develop skills that facilitate intentional change and intentional learning  Educators must teach not only their topic, they must also teach students how to learn  Educators need to know about how people experience the processes of change and growth

Purpose of this Session  Identify basic assumptions behind initiatives for human change and transfer of learning,  Identify transfer as the goal of instruction,  Describe elements that impact transfer of learning,  Explore relationship between instruction and change initiatives,  Explain implications for instruction/program evaluation,  Provide IDT professionals with a more complete skill set for conducting and evaluating instructional and change initiatives.

Influences (IC, S/K, M/I, E) Organization Goals Accomplish- ments Actions Influences = Programmatic support for organization initiatives Actions = Activities that leads to accomplishments Accomplishments = What individuals produce to support org goals Organization Goals = The value-added of the organization Assumption #1: Basic Progression of Human Performance

Assumption #2: The Performance Equation P = IC * M/I * S/K * * Env Inherent Capability = individuals’ inherent abilities and dispositions Skills/Knowledge = facilitate individuals’ knowledge of what/how to do Motivation/Incentive = support individuals’ desires to do Environmental supports = provide individuals with resources to do

Assumption #3: Basic Progression of Instructional Design Influences (IC, S/K, M/I, E) Organization Goals Accomplish- ments Actions Systematic instructional design involves reverse engineering (beginning with the end in mind): 1. What is the Goal we are trying to achieve? 2. What accomplishments must be produced in order to achieve the goal? 3. What behaviors must performers engage in? 4. What influences are required to support performers (IC, S/K, M/I, Env)?

 Instruction is an appropriate intervention if (and only if) a Skill / Knowledge deficit is present in performers  The purpose of instruction is to increase overt and covert capability in learners so they can do things they were never before able to do (as opposed to information, the purpose of which is to reduce uncertainty)  The goal is instruction is to ensure that performers’ new capabilities transfer to the “real world” Goal and Purpose of Instruction

Why does transfer of learning matter? Influences (S/K, M/I, E) Business Goals Accomplish- ments Actions We are interested in transfer of learning because: - it impacts human performance - it impacts effectiveness of organizations

1) Transfer to long-term memory 2) Transfer to “real world” performance  Can’t have #2 if you don’t have #1  This is why pre-test and post-test are so important Two Types of Learning Transfer

Impact of Instructional Design on Transfer Elements that systematically produce transfer of learning:  Goals - what do you want to transfer?  Performance Aids - tools that support transfer  Tests - how you will measure transfer  Application & Feedback - how you will facilitate transfer  Examples - how you will model for transfer  Instruction - how you will prepare learner for transfer  Leader material - how to set up instruction that transfers

Impact of Application on Transfer  Transfer to LTM = Quantity of opportunities for application  Transfer to “Real World” = Quality of opportunities for application

Major Blocks to Transfer  Lack of consideration of Performance Equation  Instruction that is not designed systematically  Not enough opportunities for application and feedback  Low levels of simulation in application  Minimal variance of problems in application  Forgetting (time between learning and performance)  Lack of E and M/I support in real world

Systemic Support for Transfer  Back to Performance equation: IC + S/K + Env + M/I  Plan and Follow up: Before - During - After  Lack of systematic support minimizes possibility of transfer  Instructional designers and program managers must work together for integrated support

Kirkpatrick & Evaluation of Training  Level 4: Long-Term Results / ROI  Level 3: OTJ Behavior  Level 2: Participant Learning  Level 1: Participant Reaction

Kirkpatrick’s 4 Levels of Evaluation  A sequence of ways to evaluate instructional programs  Each level is important  None of the levels should be bypassed to get to a level that is considered more important  As you move from one level to the next, the process becomes more difficult and time- consuming  Each level provides more valuable information

Links between Program Design, Instruction & Program Evaluation Business Goal Influences Behaviors Accomplishments Results / ROI Reaction Learning Real World Behavior Program Design: Program Evaluation: Instruction

Where Does Transfer Begin?  Level 4: Long-Term Results / ROI  Level 3: Real World Behavior  Level 2: Participant Learning  Level 1: Participant Reaction (Kirkpatrick’s four levels of training evaluation)

What To Know about Instruction 1)Purpose of instruction is to increase human capability that helps to achieve goals of program/organization 2)Instruction is but one influence on human performance 3)Instruction is an investment in human capital – most expensive investment 4)Instruction is only valuable if it transfers to performance context 5)Transfer is dependent on well-designed instruction 6)Transfer requires performance supports after instruction 7)Instructional staff must work with program management to provide appropriate supports for transfer 8)Evaluators should study relationship between instruction and program management

Implications for Evaluation  The better the instructional design, the easier it is to do instructional evaluation and program evaluation  Success of instruction is measured in Level 2  Instruction can never take complete credit for “real world” performance or ROI  This doesn’t mean that we can not evaluate whether our goals have been achieved

Seminal Sources  Joseph H. Harless (1993), An Ounce of Analysis  Donald L. Kirkpatrick (1994), Evaluating Training Programs  Robert Mager (1997), The New Mager Six-Pack  Thomas Gilbert (1968), Human Competence

Contact Information Barbara A. Bichelmeyer, Ph.D. Associate Professor Department of Instructional Systems Technology Indiana University 201 North Rose Ave., #2226 Bloomington, IN