1. Beyond Measuring Usage - Tools & Techniques for Demonstrating Program Value SkillSoft and Knowledge Advisors Learning: Re-Imagined

2. Workshop Agenda Welcome and Introduction Metrics in Times of Economic Uncertainty Jeffrey Berk - Knowledge Advisors Metrics That Matter ™ Case Study –CNA Tom Simon and Mike Castillo - CNA SkillSoft Tools & Techniques Heather McCain, Laura Rexford, Lori Turley - SkillSoft Wrap up

3. Economic Uncertainty Emphasizes Human Capital Measurement Jeffrey Berk KnowledgeAdvisors

4. © 2008, KnowledgeAdvisors 49-Oct-15 Measurement in Economic Uncertainty Agenda Measurement for Performance Management Business Case for Measurement

5. © 2008, KnowledgeAdvisors Bumpy Economic Times  S&P 500 Down Double Digits  $700 Billion Bailout  DJIA Record Daily Losses  Unemployment on the Rise  Well Known Brands in Trouble © 2008, KnowledgeAdvisors59-Oct-15

6. © 2008, KnowledgeAdvisors What are Business Leaders Saying? © 2008, KnowledgeAdvisors69-Oct-15 Source: UK Commission for Employment and Skills, November 10, 2008 Firms that don’t train are 2.5 times more likely to fail than those that do train. “Investing now in building new skills will put us in the strongest position as the economy recovers.” “The skills of our people are our best guarantee of future prosperity - - and the best investment a business can make in challenging times.”

7. © 2008, KnowledgeAdvisors 79-Oct-15 The Economy Changed in 16 Years 1992 0.6:1 2008 5:1 Tangible Assets Intangible Assets Market Value Intangible Assets Tangible Assets Intangible/Tangible = HCI (Human Capital Index) 66% Productivity Gain in 16 Years Gary Becker Wins Nobel Prize in Human Capital S&P 500 Grows at Record Levels Organizations Invest More in L&D, Leadership, HRIS, LMS, Analytics, Performance Management, Engagement & Recruiting

8. © 2008, KnowledgeAdvisors 89-Oct-15 Measuring Talent Maximizes Economic Value Shareholder Returns of companies investing in L&D measurement have 15.18% higher returns than the S&P 500 Source: Human Capital Measurement and Its Impact on Stock Performance, KnowledgeAdvisors, 2008

9. © 2008, KnowledgeAdvisors 99-Oct-15 Identify Gaps in Human Capital Processes PERFORMANCE RECRUITING L&D TALENT LEADERSHIP ENGAGEMENT PERFORMANCE Forecast Workforce Requirements Recruit, Select, and Hire International Assignment Mobile Workforce Employee Turnover On-boarding L&D Coaching Knowledge Mgmt Competencies Assessment Succession Planning Leadership Development Compensation and Benefits Employee Satisfaction Employee Engagement Work/Life Balance Workforce Diversity Performance Appraisal

10. © 2008, KnowledgeAdvisors Case: Talent Gap 10/9/2015© 2008, KnowledgeAdvisors10 Background: Printing and publishing company Business Issue: Gaps identified in skilled resources for changing business strategy Measurement Solution: Provided general skills competency set, collected and reported data w/analysis Results: Data used for L&D strategy and for recruiting job descriptions.

11. © 2008, KnowledgeAdvisors Case: Leadership Evaluation 10/9/2015© 2008, KnowledgeAdvisors11 Background: Major health benefits provider Business Issue: Turnover in first 90 days of employment was 25 to 30% Measurement Solution: Built a leadership program for the Blue Chip hires and used smarter evaluations to measure it and improve it Results: 93% of Blue Chip new hires were retained and no longer flight risks. 73% reduction in admin burden in the measurement process itself “Metrics that Matter has provided objective data that supports “gut feelings” about where improvements can be made and where things are going well.” -Director of Leadership & Associate Development

12. © 2008, KnowledgeAdvisors Case: Template Links L&D to Sales 10/9/2015© 2008, KnowledgeAdvisors12 Background: Consumer Packaged Goods firm Business Issue: Needed reasonable evidence of sales impact from training Measurement Solution: Used actual results and reasonable assumptions via impact template Results: Proved that $74,256/person of sales dollars were tied to a $5,000/person training investment. Example of template used to showcase ROI from Sales training Source: KnowledgeAdvisors

13. © 2008, KnowledgeAdvisors Case: ERP Technology Company 9-Oct-15© 2008, KnowledgeAdvisors13 Background: ERP company performing customer education Business Issue: Show value of training to customers in terms of increased productivity and quality Measurement Solution: Collected 12,770 end of program evaluations within 6 months asking participants to forecast impact Analytics Results: 18.2% estimated increase in productivity from the training (adjusted down for self-reported bias)

14. © 2008, KnowledgeAdvisors 149-Oct-15 Measurement in Economic Uncertainty Agenda Measurement for Performance Management Business Case for Measurement

15. © 2008, KnowledgeAdvisors 159-Oct-15 Fact 1.Most organizations invest 1% to 4% of revenue in talent development source ASTD stated L&D is 2 to 3% of revenue 2.Most organizations waste 50% or more of their talent development expenses source Metrics that Matter 600,000 follow up data points is 55%, IDC sites 50% waste, Brinkerhoff cites 80% or more waste 3.Most organizations waste.5% to 2% of revenue on ineffective talent development calculation from item 1 x 2 above 4.Effective investments in predictive analytics increases performance by 2.4% source IDC Talent Measurement Results: Increases Performance * Reduces Waste * Slashes Admin Burden

16. © 2008, KnowledgeAdvisors 169-Oct-15 Fact According to IDC, programs that use predictive analytics solutions, such as Metrics that Matter, generate 60% better returns than those that don’t. Example: If one invests $1,000,000 in an L&D program that generates a $1,000,000 profit, without using a predictive measurement solution, effective use of one, could have generated an additional $600,000 in profit.

17. © 2008, KnowledgeAdvisors 179-Oct-15 Measurement Impact on Increased Performance 1.Major services firm realized $200,000 increased annual profit by using Metrics that Matter predictive analytics capabilities. 2.Through measurement a leading federal agency significantly increased output and business impact without increasing budget over a 3 year period. 3.Major telecommunications firm using Metrics that Matter was able to increase sales by $1.25 million dollars through better supply chain management. 4.15,000 hours in improved productivity by a floral company in identifying appropriate training investments.

18. © 2008, KnowledgeAdvisors 189-Oct-15 Measurement Impact on Reduced Waste 1.25 retail stores avoided $100,000 of wasted training. Retail store training measured after first of six weeks saw 1 set of ineffective trainers and discontinued using them. 2.Instructor compared herself to other instructors and changed approach. 6 months of additional sessions performed better to avoid at least $100,000 in scrap learning. 3.$500,000 in funding gained by measuring scrap learning. Services firm measured training results and found high quality but low impact. The program was retained and additional funding rewarded to consult with managers on increasing impact. 4.88% of learners not applying training is NOT due to bad training. Waste is due to unsupportive employee environment. A billion dollar company can save at least $1,000,000 if it can reduce waste by just 20% by performance managing the employee environment.

19. © 2008, KnowledgeAdvisors 199-Oct-15 Measurement Impact on Reduced Admin 1.75% reduction in administration costs from automating measurement process vs. manual measurement. 2.15 seconds to retrieve actionable data for decision-making vs. 15 hours prior to automating the measurement process. This automation of 1 report alone saved over $36,000 per year. 3.$400,000 savings by major accounting firm that used smarter measurement in a practical way to yield similar results compared to complex and burdensome alternative measurement techniques

20. © 2008, KnowledgeAdvisors 209-Oct-15 Calculate Performance Gain $1,000,000,000Revenue 25%Labor Percentage $250,000,000Labor Cost 1% Increased Productivity from Measurement % $2,500,000 Increased Performance from Measurement

21. © 2008, KnowledgeAdvisors 219-Oct-15 Calculate Waste Reduction Description $1,000,000,000Revenue 1%L&D as % of Revenue $10,000,000L&D Investment 50%Waste or Unrealized Value % $5,000,000Waste or Unrealized Value 5%Waste Reduction from Measurement % $250,000 Waste Reduction from Measurement

22. © 2008, KnowledgeAdvisors 229-Oct-15 Calculate Admin Savings Description $55,000Salary of 1 FTE admin for metrics $50,000Cost avoided from unnecessary analysis $50,000Cost to replication fraction of MTM benchmarks 75% Admin savings from MTM measurement and benchmarks $116,250 Administration Savings from Measurement

23. © 2008, KnowledgeAdvisors 239-Oct-15 Total Value Value Summary $2,500,000Performance Gain $250,000Waste or Scrap Avoided $116,250Administration Savings $2,866,250Total Quantified Value

24. © 2008, KnowledgeAdvisors 249-Oct-15 Measurement in Economic Uncertainty Agenda Measurement for Performance Management Business Case for Measurement

25. © 2008, KnowledgeAdvisors Talent and L&D Measurement Tips  Identify and Audit Gaps in Human Capital Processes  Build a Robust Evaluation Process  Track and Link to Actual Business Results  Reduce Errors and Scrap  Leverage Technology  Use Metrics for Performance Management © 2008, KnowledgeAdvisors259-Oct-15

26. © 2008, KnowledgeAdvisors 269-Oct-15 Benchmarks for Performance Management Benchmarks: Internal or External Provide a point of reference Motivate by example Available thru Metrics that Matter

27. © 2008, KnowledgeAdvisors 279-Oct-15 Goals for Performance Management Goals: Actual to Goal Set in Metrics that Matter Move in right direction Challenging yet attainable

28. © 2008, KnowledgeAdvisors 289-Oct-15 Trends for Performance Management Trends: Time based analysis Available in Metrics that Matter Shows improvement over time

29. © 2008, KnowledgeAdvisors 299-Oct-15 Dashboards for Performance Management Dashboards: Concise Data Summary Data Visual View Available in Metrics that Matter for course managers, instructors, impact and operations

30. © 2008, KnowledgeAdvisors 309-Oct-15 A template for the learning team to conduct more comprehensive business impact and ROI analysis. Templates bring stewardship and collaborative guidance to the process of linking training to results. Track & Link to Actual Business Results

31. © 2008, KnowledgeAdvisors Build a Robust Evaluation Process

32. © 2008, KnowledgeAdvisors 329-Oct-15 Measurement in Economic Uncertainty Agenda Measurement for Performance Management Business Case for Measurement

33. © 2008, KnowledgeAdvisors 339-Oct-15 Experts in metrics that matter KnowledgeAdvisors is a human capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions. KnowledgeAdvisors

34. © 2008, KnowledgeAdvisors KnowledgeAdvisors Clients 10/9/2015© 2008, KnowledgeAdvisors34

35. © 2008, KnowledgeAdvisors9-Oct-15Page 35 www.knowledgeadvisors.com For further information on the information contact: Jeffrey Berk Chief Operating Officer +1 312 676 4411 (phone) +1 312 676 4401 (fax) jberk@knowledgeadvisors.com

36. Measurement at CNA Tom Simon & Michael Castillo Area of Presenter

37. Measurement CNA At A Glance 1 million business and professionals insured 7th largest commercial lines insurer U.S. and international operations 9,000 employees $7.8 billion in revenues Assets of $52 billion “A” rated for financial strength by A.M. Best

38. Measurement

43. Why ROI? Benefits of ROI: Business cases benefit strongly from ROI Measuring financial impact helps illustrate importance of learning Value-based measurements support Learning Growth Model progression Challenges of ROI: Limited time and resources Hard to evaluate ROI for intangibles (learning) Securing internal data difficult

44. SkillSoft Measurement Tools EZ ROI Calculator - “soft measure” Business Impact Analysis – Validated by internal survey Value Measurement Toolkit (VMT) – ROI, Phillips Level 5 Business Impact Analysis EZ ROI Calculator Value Measurement Toolkit time, resources, data involved estimate vs. rigorous measure

45. EZ ROI Calculator Business Impact Analysis EZ ROI Calculator Value Measurement Toolkit time, resources, data involved estimate vs. rigorous measure

46. Objectives and Attributes Objectives: Deliver more value-based metrics Help strengthen business cases Prospective use (Should we train?) and retrospective use (How did we do?) Attributes: Easy to use Requires little time and very little data Defensible Immediate analysis Cost displacement and performance improvement measures

47. EZ ROI CALCULATOR Components: Introduction and assumptions Cost displacement calculator Cost displacement sample Productivity impact calculator Productivity impact sample

48. Application Ideas for EZ ROI Calculator SkillSoft versus another learning option Monetary value of productivity gains Limited licenses? Select wisely

49. Business Impact Analysis EZ ROI Calculator Value Measurement Toolkit time, resources, data involved estimate vs. rigorous measure

50. Objectives and Attributes Objectives: Measure satisfaction, determine motivation and determine job impact by the following: Determine individual skill requirements and needs Evaluate program satisfaction Gauge skill improvement (based on individual responses/group roll-up) to determine productivity gains Attributes: Easy to use Company-specific Requires limited time and data Longer timeline Stronger measurement story Potential testimonials

51. Business Impact Analysis Components Custom survey Communications to launch the survey Summary of feedback collected Impact/Results Summary

52. Sample Survey Questions 1.Job Category Drop Down of Job Families 2.Salary Range Drop Down of Salary Ranges 3.Would you recommend the SkillSoft resources to a colleague? Select One (Yes, Somewhat, No) 4.What was the main reason why you accessed SkillSoft training? Drop down with list of likely options plus an “other” option that has a blank field 5.Primarily, what skill were you seeking to develop? Open short text field 6.What percent of your job requires you to use this skill? Drop down with a range of percentages (0%, 1-10%, 11-20%, 21-30%, 31-40%, 41-50%, 51-60%, 61-70%, 71-80%, 81-90%, 91-100%) 7.As a result of the training, how much of a skill increase did you experience? Drop down with a range of percentages (0%, 5-10%, 11-15%, 15-20%, 21-25%, 26-30%, More than 30%) 8.If you did not have access to SkillSoft resources, what alternatives would you have considered? Drop down of common training types including “no alternatives” and “other” options 9.In approximate terms, how much do you think the alternate option would have cost? Open numerical field, includes the options “no cost” and “not sure” 10.When are you planning to use SkillSoft resources again? Select One (Not sure, within 2 weeks, within 30 days, within 90 days, more than 90 days)

53. Output of Survey Survey Results document including: Background Survey Responses (graphical representation) Comment Highlights ROI Impact

54. Application Ideas for Business Impact Analysis Translate productivity gains into monetary terms specific to your company Utilize survey results for ongoing marketing and increased usage Create targeted programs based on skill needs identified

55. Value Measurement Toolkit Business Impact Analysis EZ ROI Calculator Value Measurement Toolkit time, resources, data involved estimate vs. rigorous measure

56. Objectives and Attributes Objectives: Measure the value derived from the implementation and ongoing support of learning initiatives. A method offered to aid clients to establish a process for measurement. Deliver value in terms your stakeholders will believe in – cot savings, productivity impact, increased revenue, etc. Attributes: Longer term study based on a defined process Requires internal, validated data Defensible and credible due to stakeholder input Return on investment

57. Notice that the 5 levels are not each a step in the process. The process can be used to any level. The Value Measurement Process

58. White Paper Readiness Worksheet Planner Calculator Case Studies Value Measurement Tools

59. Application Ideas for Value Measurement Validate the investment in technology enabled learning Meet stakeholder expectations/shape expectations Measuring key strategic programs with high visibility

60. Hands-On Exercise Divide into teams Review your scenario Agree on appropriate tool Be prepared to present considerations for tool selection

61. Thank you for participating!