1. Phillips Associates 1 Business Results Made Visible: Design Proof Positive Level 4 Evaluations Presented by: Ken Phillips Phillips Associates May 24, 2016 Session: TU 112 10:00-11:15 AM

2. Phillips Associates 2 Agenda 1.Examine Level 4 evaluation facts 2.Discover which learning programs are ideally suited for conducting a Level 4 evaluation 3.Examine a two-phase eight-guideline process for conducting Level 4 evaluations 4.Analyze three methods for connecting learning programs to business results

3. Phillips Associates 3 Kirkpatrick / Phillips Evaluation Model Levels of EvaluationMeasurement FocusTime Frame Level 1: Reaction Participant favorable reaction to a learning program Conclusion of learning program Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes Conclusion of learning program or within 6 to 8 weeks after Level 3: Behavior Degree to which participants applied back-on-the-job what was learned 2 to 12 months Level 4: Results Degree to which targeted business outcomes were achieved 9 to 18 months Level 5: ROI Degree to which monetary program benefits exceed program costs 9 to 18 months

4. Phillips Associates 4 Level 4 Evaluation Facts Source: ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009 37% Organizations evaluate some programs at Level 4 of all programs evaluated 15%75% Organizations view data collected as having high or very high value

5. Phillips Associates 5 CEOs Say The #1 thing they would most like to see from their learning and performance investments is evidence of business impact (Level 4 evaluation)… Source: ROI Institute Research Study, 2009 96% receive this type of information …but only 8%

6. Phillips Associates 6 Why the Disconnect? Level 4 business results perceived as too difficult to measure Belief that high probability of uncontrollable variables affecting business outcomes makes measuring business results meaningless Not currently measuring Level 2 & 3 so can’t be expected to measure Level 4

7. Phillips Associates 7 What’s the Solution? Find ways to provide CEOs with the information they want!

8. Phillips Associates 8 Source: Donald & James Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 2006. Data Collection Methods

9. Phillips Associates 9 Programs to Evaluate at Level 4 Ones that are strategically important Ones with high development and/or implementation costs Ones that a large number of participants will attend Ones the business executive stakeholder wants evaluated at Level 4

10. Phillips Associates 10 Most Common Level 4 Metrics Adapted from ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009 MetricPercentage Customer satisfaction39% Employee satisfaction/engagement37% Proficiency/competency levels33% Productivity indicators26% Turnover/promotions25% Actual business outcomes22%

11. Phillips Associates 11 Metrics that Correlate Strongest with Evaluation Success Source: ASTD Research Study, 2009 MetricPercentage Customer satisfaction39% Employee satisfaction/engagement37% Proficiency/competency levels33% Productivity indicators26% Turnover/promotions25% Actual business outcomes22%

12. Phillips Associates 12 Conduct Level 4 Evaluations in Two Phases Phase 1 Phase 2 Identify business results that have a strong relationship with learning program content Connect the learning program to the business results

13. Phillips Associates 13 Phase 1: Guiding Principles 1. Plan for Level 4 evaluations from start of training project 2. Identify existing Level 4 business metrics – think operational, financial & HR 3. Search first at stakeholder, next at department and last at organizational level to identify business metrics 4. Select business metric(s) with a strong relationship to program content & ones you can access

14. Phillips Associates 14 Instructions 1. Form a group of 3, 4 or 5 persons 2. Identify what business results you might track to conduct a Level 4 evaluation, given the training topic selected by your group 3. Use your own organization as a reference to identify actual business results you might track – think operational, financial & HR data 4. Be prepared to share your ideas with the whole group

15. Phillips Associates 15 Select One Training Program First-level supervisor Selling skills Performance management New hire orientation Employee engagement

16. Phillips Associates 16 Business Metric Examples Financial Sales Net income Revenue Expenses Cost of workforce Operational Quality Productivity Supply chain Innovation Customer satisfaction

17. Phillips Associates 17 Business Metric Examples HR Turnover Grievances Safety incidents/accidents Absenteeism Employee satisfaction/engagement Retention Bench strength

18. Phillips Associates 18 Source: Jim Kirkpatrick Level 4 business results are the easiest to measure, but the hardest to correlate with training.

19. Phillips Associates 19 Phase 2: Guiding Principles 5.Three ways to connect a learning program to business results: Trend-line analysis Expert estimation Control group 6.Use data collection frequency & data analysis level to determine best approach 7.When using Expert Estimation, discount results by potential error of estimate 8.When conducting a Level 4 evaluation, also conduct a Level 1, 2 & 3 evaluation

20. Phillips Associates 20Instructions Note: All managers & supervisors attended a leadership program during highlighted period. 1. Form a group of 3, 4 or 5 persons 2. Review the 2 Business Results Data charts 3. Develop a plan for how to connect the leadership program with each of the business results 4. Be prepared to share your ideas with the whole group

21. Phillips Associates 21 Business Results Data HR data collected over a 12 month period MonthTurnoverGrievances Jan178 Feb157 Mar187 Apr166 May177 June156 July127 Aug126 Sept106 Oct75 Nov75 Dec54 Jan53

22. Phillips Associates 22 Business Results Data Percent of favorable responses to annual Employee Engagement Survey Year Employee Engagement Score 200765 200850 200952 201055 201152 201270

23. Phillips Associates 23 Business Results Data MonthTurnoverGrievances Jan178 Feb157 Mar187 Apr166 May177 June156 July127 Aug126 Sept106 Oct75 Nov75 Dec54 Jan53 HR data collected over a 12 month period

24. Phillips Associates 24 Don’t Do This! Report the data chronologically and note where the learning program occurred Compute the average of the data before and after implementing the learning program

25. Phillips Associates 25 Use trend-line analysis to connect the leadership program to the business results data Do This!

26. Phillips Associates 26 Trend-Line Analysis Use preprogram performance as base and extend trend into future. Following program implementation, actual performance is compared to projected trend line performance

27. Phillips Associates 27 Trend-Line Example Compute Difference

28. Phillips Associates 28 Trend-Line Analysis Five conditions must be present to use: 1. At least 6 data points of pre-program performance available 2. Pre-program data is relatively stable 3. Pre-program trend is expected to continue regardless of whether learning program is implemented 4. No new variables enter process after learning program is implemented 5. Frequent collection points for targeted business results data

29. Phillips Associates 29 Business Results Data Year Employee Engagement Score 200765 200850 200952 201055 201152 201270 Percent of favorable responses to annual Employee Engagement Survey

30. Phillips Associates 30 Don’t Do This! Report comparative data noting previous and current business results data

31. Phillips Associates 31 Do This! Use expert estimation or a control group design to connect the leadership program to the business results data

32. Phillips Associates 32 Expert Estimation Obtain estimates from people who have first-hand knowledge or credible insight into cause-and-effect relationship between implementing a learning program and a change in a business metric Technique developed by Jack Phillips in 1983

33. Phillips Associates 33 Calculating Expert Estimation Participant % Leadership Program Contributed to Improved Results Confidence Level of Estimate Adjusted Connection 1 907063 2 808568 3 60 36 4 859077 5 608048 6 75 56 7 805040 8 908072 9 856555 10 75 56 Fact: Employee Engagement score improved from 52 to 70% Favorable Responses

34. Phillips Associates 34 Expert Estimation Formula Sum Participant Adjusted Estimates Number of Participants Average Adjusted Percentage Calculation 1: Post-Program Business Metric Value Pre-Program Business Metric Value Average Adjusted Contribution Percentage Calculation 2: (571)(10)(57.1) (70)(52)(57.1) (10.3)

35. Phillips Associates 35 Expert Estimation Three conditions must be present to use: 1. Learning program conducted 2. Business metrics identified prior to program and monitored following program reveal an improvement 3. Experts are able to provide input connecting learning program and business results – discounted by potential error of their estimate

36. Phillips Associates 36 Control Group Experimental design in which two groups of learners are compared where one group attends a learning program and the other doesn’t. Comparisons are made either following the learning program or both before and after the program

37. Phillips Associates 37 Post-Test Only Control Group Design Experimental Group Learning Program Business Result Control Group Business Result Note: To obtain valid results, Experimental and Control Groups must be similar! Compute Difference

38. Phillips Associates 38 Pre/Post-Test Control Group Design Learning Program BR M2 BR M1 Control Group BR M2 BR M1 Compute Difference Compute Difference Experimental Group

39. Phillips Associates 39 Control Group Four conditions must be present to use: 1. Natural control group exists – large number of participants to attend same learning program staggered over time 2. Business results can be broken down to individual participant level 3. Stakeholder agrees with method 4. Credibility of results is of highest importance

40. Phillips Associates 40 Summary Plan for Level 4 evaluations from the start of the training project Seek first to identify existing business metrics before creating new ones Only select business metrics that have a strong relationship with the program content

41. Phillips Associates 41 Summary (cont.) To connect a learning program to business results, use either: Control group Trend-line analysis Expert estimation Use data collection frequency & data analysis level as guide in determining best approach to connect learning to business results

42. Phillips Associates 42 Everything that can be counted doesn’t necessarily count and everything that counts cannot necessarily be counted. Source: Albert Einstein Keep in Mind:

43. Phillips Associates 43 Phillips, Ken, “Eight Tips on Developing Valid Level 1 Evaluation Forms”, Training Today, Fall 2007, pps. 8 & 14. Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8. Phillips, Ken, “Capturing Elusive Level 3 Data: The Secrets of Survey Design”, Unpublished article, 2013. Phillips, Ken, “Level 1 Evaluations: Do They Have a Role in Organizational Learning Strategy?”, Unpublished article, 2013. Phillips, Ken, “Business Results Made Visible: Designing Proof Positive Level 4 Evaluations”, Unpublished article, 2013. Free Articles

44. Phillips Associates 44 Phillips Associates ken@phillipsassociates.com (847) 231-6068 www.phillipsassociates.com 34137 N. Wooded Glen Drive Grayslake, Illinois 60030 Ken Phillips