1. How to Demonstrate the Value of Contract Education Jack J. Phillips, Ph.D. Chairman ROI Institute, Inc. May 11, 2016 10:00 AM- 12:00 PM 5 Levels of Evaluation to Bring Value to Your Client and Your Training Programs

2. 2 Objectives After completing this session, participants should be able to: 1.Define the five levels of outcomes possible from contract education, including ROI. 2.Identify the categories that show value to clients. 3.Explain how these measures are developed. 4.Communicate results to clients and other stakeholders.

3. Which of These are True? 1.Most of contract education is wasted (not used). 2.The education outcome desired by executives in client organizations is rarely measured. 3.Most contract education providers do not have data showing that they make a difference in the organization. 3

4. The Top Executive View 4

5. Measure We currently measure this We should measure this in the future My ranking of the importance of this measure 1. Inputs: “Last year, 78,000 employees received formal learning.” 94%85% 2. Efficiency: “Formal learning costs $2.15 per hour of learning consumed.” 78%82% 3. Reaction: “Employees rated our training very high, averaging 4.2 out of 5.” 53%22% 4. Learning: “92% of participants increased knowledge and skills” 32%28% *CEO Survey—Fortune 500 and Large Private Companies, ROI Institute N=96 Respondents The Executive View of Metrics* 5

6. Measure We currently measure this We should measure this in the future My ranking of the importance of this measure 5. Application: “At least 78% of employees are using the skills on the job” 11%61% 6. Impact: “Our programs are driving our top 5 business measures in the organization.” 8%96% 7. ROI: “Five ROI studies were conducted on major programs yielding an average of 68% ROI.” 4%74% 8. Awards: “Our learning and development program won an award from American Society for Training and Development 40%44% *CEO Survey—Fortune 500 and Large Private Companies, ROI Institute N=96 Respondents The Executive View of Metrics* 6

7. 7 2015 Measurement and Metrics Study by CLO Magazine 35.6% Use business data to demonstrate the impact of the training organization on the broader enterprise. 21.6%Use ROI for that purpose. 22.6 % Plan to implement ROI within twelve months. 9.7% Plan implementation in twelve to twenty-four months. 17.3%Plan implementation with no time frame. (N=335) 71.2%

8. Ten Steps to Deliver Business Value Start with why Connect the right solution Define expectations Design for results Create or acquire powerful content Deliver for efficiency and effectiveness Ensure the transfer of learning to the job Measure results Communicate the results to the key stakeholders Use the results to optimize and allocate 8

9. 9 Alignment is the key Is it a problem or opportunity? Specific business measure(s) 1. Start with why

10. 10 A B C Start Here End Here 5 ROI Objectives 5 4 Impact Objectives 4 3 Application Objectives 3 2 Learning Objectives 2 1 Reaction Objectives 1 Project Business Needs Performance Needs Learning Needs Preference Needs Impact Learning Reaction Initial Analysis Measurement and Evaluation Application Business Alignment and ForecastingThe ROI Process Model Alignment Model Payoff Needs ROI

11. 11 Ask:  What are we doing (or not doing that’s influencing the business measure?  How can we achieve this performance? 2. Connect to the Right Solution

12. 12 Set objectives at multiple levels Redefine success of contract education Expand responsibilities 3. Define Expectations

13. 13 Matching Evaluation Levels with Objectives Exercise 1.Reaction and Planned Action 2.Learning 3.Application 4.Business Impact 5.Return on Investment

14. 14 What’s Your Business? Your choiceThe Possible Value Descriptions “Serve the largest number of people with the least amount of disruption and cost.” “Participants are engaged, enjoy the programs, and see their experience as valuable.” “Participants are learning the latest information and skills to help make this a great organization.” “Participants leave the center, take action, use the content, and make important changes.” “Participants are driving important business measures and having an impact in their work units and the organization.” “Participants and the organization have a positive return on the investment of their time and the resources of the center.”

15. 15 Focus on application and impact objectives Build on application and impact tools Design for data collection Remove/minimize barriers 4. Design the Results

16. 16 Focus on the objectives Make or buy Make it relevant Make it new Make it action-oriented 5. Create or Acquire Powerful Content

17. 17 Low cost doesn’t mean more impact Focus on numerator and denominator of ROI equation 6. Deliver for Efficiency and Effectiveness

18. 18 Benefits-Cost Ratio (BCR) Program Benefits Program Costs = ROI (%) Net Program Benefits Program Costs = X 100 ROI is Reported in Two Ways

19. 19 Before learning program During learning program After learning program 7. Ensure Transfer of Learning to the Job

20. 20 Level% of Programs 0Input100% 1Reaction100% 2Learning80-90% 3Application30% 4Impact10% 5ROI5% 8. Measure Results

21. 21 Define audience Identify why they need it Select method Move quickly 9. Communicate Results to Key Stakeholders

22. 22 10. Use Results to Optimize and Allocate EVALUATIONOPTIMIZATIONALLOCATION Make it Better Improve Results Get More Funding

23. Status of Measurement 23 LevelMeasurement Category Current Status* Best Practice Comments About Status O Inputs/Indicators Measures the number of programs, participants, audience, costs, and efficiencies 100% This is being accomplished now 1 Reaction and Planned Action Measure reaction to, and satisfaction with, the experience, content, and value of program 100% Need more focus on content and perceived value 79% †

24. 24 LevelMeasurement Category Current Status* Best Practice Comments About Status 2 Learning Measures what participants learned in the program– information, knowledge, skills, and contacts (takeaways from the program) 30-40%80-90% Must use simple learning measures 54% † 3 Application Measures progress after the program– the use of information, knowledge, skills, and contacts 10%30% Need more follow-up 31% † Status of Measurement

25. 25 LevelMeasurement Category Current Status* Best Practice Comments About Status 4 Impact Measures changes in business impact variables such as output, quality, time, and cost-linked to the program 5%10% This is the connection to business impact 14.4% † 5 ROI Compares the monetary benefits of the business impact measures to the costs of the program 1%5% The ultimate level of evaluation 4.3% †

26. 26 Reasons to Measure Impact and ROI 7 1.The boss wants it. 2.ROI provides evidence your programs make an impact. 3.ROI can help improve programs. 4.ROI can help keep and expand programs, not eliminate them. 5.ROI can protect your budget. 6.ROI helps build key relationships. 7.ROI can build the support that you need.

27. The ROI Methodology Develop Objectives of Solution Develop Evaluation Plans and Baseline Data Collect Data During Solution Implementation Collect Data After Solution Implementation Level 2: Learning Level 4: Business Impact Level 1: Reaction, Satisfaction, and Planned Actions Level 3: Application/ Implementation Evaluation Planning Data Collection 27

28. Isolate the Effects Convert Data to Monetary Value Calculate the Return on Investment Capture Costs of Solution Identify Intangible Measures Generate Impact Study Level 5: ROI Intangible Measures Data Analysis Reporting 28 The ROI Methodology

29. 29Questions? Visit us online: www.roiinstitute.net Email: jack@roiintitute.net Connect with us on: