Phillips Associates 1 Boost Training Transfer Using Predictive Learning Analytics™ Presented by: Ken Phillips Phillips Associates May 22, 2016 Session: SU 217 1:30-2:30 PM

Phillips Associates 2 Agenda 1. Discover the impact “scrap learning” has on wasted organization resources & lost credibility with business executives 2. Analyze how to build a PLA model that identifies those learners who are most & least likely to apply what they learned 3. Examine the 7 benefits of using PLA in your organization & the 3 phases & 9 steps for getting started

Phillips Associates 3 Scrap learning – What is it?

Phillips Associates 4 Term coined by KnowledgeAdvisors, a CEB company, that describes the difference between learning that’s delivered and learning that’s applied back on the job Scrap Learning

Phillips Associates 5 How big is the problem?

Phillips Associates 6 Benchmark Study 1 45% Source: Confronting Scrap Learning CEB White Paper, 2014

Phillips Associates 7 Benchmark Study 2 < 20% Applied new skills back on the job > 15% Didn’t try to apply new skills back on the job 65% Tried applying new skills back on the job but reverted back Source: Robert Brinkerhoff, 2004

Phillips Associates 8 View from individual organization level

Phillips Associates 9 According to ATD 2015 “State of the Industry Report” Average per employee training expenditure Average number of training hours consumed per employee $ = =

Phillips Associates 10 Calculating Scrap Learning at individual organization level $1229 X 45% = 32.4 hours X 45% = $1229 X 80% = 32.4 hours X 80% = $ $

Phillips Associates 11 Houston, we have a problem! Source: James Lovell, Apollo 13 flight

Phillips Associates 12 The solution: Predictive Learning Analytics ™

Phillips Associates 13 is a methodology for peering into the future, at the conclusion of a learning program, and forecasting learner outcomes and actions, with the intent of changing those outcomes and actions for the better. Predictive Learning Analytics™(PLA) Source: Ken Phillips

Phillips Associates 14 PLA vs. Traditional Measurement & Evaluation

Phillips Associates 15 Kirkpatrick / Phillips Evaluation Model Levels of EvaluationMeasurement FocusTime Frame Level 1: Reaction Participant favorable reaction to a learning program Conclusion of learning program Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes Conclusion of learning program or within 6 to 8 weeks after Level 3: Behavior Degree to which participants applied back-on-the-job what was learned 2 to 12 months Level 4: Results Degree to which targeted business outcomes were achieved 9 to 18 months Level 5: ROI Degree to which monetary program benefits exceed program costs 9 to 18 months

Phillips Associates 16 PLA vs. Traditional Learning M&E High Low PastFuture Value Time Traditional Learning M&E Predictive Learning Analytics Traditional Learning M&E Focuses on programs or cohorts Describes what has happened Predictive Learning Analytics Focuses on individual learners Predicts future likelihood of certain behaviors and actions

Phillips Associates 17 Algorithm consisting of 10 factors that are known to contribute to training transfer & research shows have a strong positive correlation with either Level 2 learning, Level 3 behavior or Level 4 results Heart of PLA:

Phillips Associates 18 Instructions 1. Form a group of 3, 4 or 5 persons 2. Brainstorm a list of things we know to be “truths” about training transfer 3. Be prepared to share your ideas with the whole group Example: Training transfer increases when learners have an immediate opportunity to apply what they learned back on the job

Phillips Associates 19 Level 2 Training Transfer Factors 1. Acquire new information – knowledge, skills or attitudes 2. See a program as relevant to themselves and their job 3. See a program as an important investment in their career development Learners need to:

Phillips Associates 20 Level 3 Training Transfer Factors 4. Be personally motivated to apply what was learned 5. Have confidence in their ability to apply what was learned 6. Reflect on key lessons learned & how they can help improve their performance Learners need to:

Phillips Associates 21 Level 3 Training Transfer Factors (Cont.) 7. Be actively engaged by their manager post-program regarding what was learned 8. Be supported by work colleagues, post- program, to apply what was learned 9. Have an immediate opportunity to apply what was learned Learners need to:

Phillips Associates 22 Level 4 Training Transfer Factor 10. See a likely improvement in a key business metric tracked by their department if new information learned is applied Learners need to:

Phillips Associates 23 Using the Algorithm

Phillips Associates 24 Using The Algorithm Step 1: Convert factors into survey items and include as part of a Level 1 evaluation or administer as a separate survey Step 2: Collect data from Calibration Cohort – first participants to attend PLA target training program

Phillips Associates 25 Using The Algorithm Step 4: Target participants who are at risk & least likely to apply what they learned for follow-up & reinforcement activities in order to increase training transfer Step 3: Calculate Learner Application Index™ score for each Calibration Cohort program participant

Phillips Associates 26 LAI Individual Scores

Phillips Associates 27 LAI Factor Scores

Phillips Associates 28 Additional Calculations

Phillips Associates 29 Additional Calculations Manager Training Support Index Manager Training Support Index™ Sort learners into groups arranged by manager Compute Manager Training Support Index™ score for each manager & identify managers who do a good & poor job of supporting learning Work with those managers who need help doing a better job

Phillips Associates 30 Manager Training Support Index™ Score

Phillips Associates 31 Additional Calculations Compute Overall Program Quality score & compare the quality of one program with another Target programs not delivering value for either revision or elimination Overall Program Quality

Phillips Associates 32 Benefits of Using Predictive Learning Analytics

Phillips Associates Less money & time wasted on learning that is delivered but not applied back on the job – scrap learning 2. Benefits Of Using PLA 3. Increased personal credibility in eyes of business executive stakeholders More effective & efficient use of follow-up activities by targeting participants who are at risk & least likely to apply what they learned in a program back on the job

Phillips Associates Benefits Of Using PLA (cont.) Objective way to identify managers who do a poor job of supporting learning so that their approach can be improved Objective way to evaluate the effectiveness of a learning program design

Phillips Associates Objective way to compare the overall quality of one learning program with another using a single number 7. Benefits Of Using PLA (cont.) Enhanced reputation among L&D colleagues

Phillips Associates 36 3 Phases & 9 Steps for Implementing PLA

Phillips Associates 37 PLA Phases & Steps

Phillips Associates 38 Summary The issue of scrap learning has been around forever. But, what’s different today is that with Predictive Learning Analytics™ there now is a way to manage it. Source: Ken Phillips

Phillips Associates 39

Phillips Associates 40 Phillips Associates (847) N. Wooded Glen Drive Grayslake, Illinois Ken Phillips

Phillips Associates 41 Ken Phillips, PhD, CPLP He regularly speaks to Association for Talent Development (ATD) groups, university classes and corporate learning and development departments. Since 2008, he has spoken at the ATD International Conference on topics related to measurement and evaluation of learning. In addition, in 2015 he was invited to present at the ATD Middle East North Africa (MENA) Conference and the ATD China Summit on topics related to measurement and evaluation of learning. Prior to pursuing a Ph.D. in the combined fields of organization behavior and educational administration at Northwestern University, Ken held management positions with two colleges and two national corporations. In addition, he has written articles that have appeared in td magazine, Training Today and HR.com, and is a contributing author to four books in the L&D field. Ken earned the Certified Professional in Learning and Performance (CPLP ® ) credential from ATD in 2006 as a pilot pioneer and was recertified in 2009, 2012 and again in Ken is founder and CEO of Phillips Associates, a consulting and publishing company with expertise in measurement and evaluation of learning and performance management. He has more than 30 years experience designing learning instruments and assessments and has authored more than a dozen published learning instruments. LinkedIn:

Phillips Associates 42 Select a single, high profile, and costly learning program for your PLA project and initially “stay under the radar” Implementation Guidelines 2. Phase 1: Setting the stage 1. Build your Predictive Learning Analytics™ algorithm

Phillips Associates 43 Collect data & calculate individual Learner Application Index™ score for each program participant Implementation Guidelines 3. Phase 1: Setting the stage (cont.) Calculate “scrap learning” percentage baseline score associated with program 4.

Phillips Associates 44 Implementation Guidelines Phase 2: Implementing the methodology 5. Identify learners at risk & least likely to apply what was learned back on job & target for follow-up activities 6. Conduct Level 2 & Level 3 evaluations in order to validate accuracy of PLA algorithm 7. Recalculate scrap learning percentage following implementation of follow-up activities

Phillips Associates 45 Implementation Guidelines Phase 3: Sharing your success Report results to business executive stakeholders Enhance accuracy of PLA algorithm by including additional data from company LMS or HRIS, if available

Phillips Associates 46 The goal of training: “Helping learners achieve great results, not providing great training!” Source: Robert Brinkerhoff