1. 1 CENTER FOR LEARNING & DEVELOPMENT Extracting Value From Post-course Evaluations Using Advanced Statistical Techniques November 12, 2009 4:30 – 6:00P Extracting Value From Post-course Evaluations Using Advanced Statistical Techniques November 12, 2009 4:30 – 6:00P

2. 2 Agenda KPMG LLP and Center for Learning & Development Headlines / Industry Trends Purpose Measurement Tools and Data Set Analytic techniques applied ResultsBenefits © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

3. 3 KPMG KPMG LLP Audit, Tax & Advisory services US Member Firm = 23,000 partners and staff Center for Learning & Development Responsible for creating national training programs Awards 1.4M to 1.6M CPE credits per year 110 partners and staff responsible for Learning & Development (L&D) Recognized by Training Magazine’s Top 125 #7 in 2007 #5 in 2008 #2 in 2009 © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

4. 4 BI Headlines / Industry Trends All information provided is of a general nature and is not intended to address the circumstances of any particular individual or entity. Although we endeavor to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. No one should act upon such information without appropriate professional advice after a thorough examination of the particular situation. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

5. 5 Logic Chains and Business Intelligence Training AB Profit AB BI Training AB Profit All information provided is of a general nature and is not intended to address the circumstances of any particular individual or entity. Although we endeavor to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. No one should act upon such information without appropriate professional advice after a thorough examination of the particular situation. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

6. 6 Purpose To describe the benefits of extracting value from training data in this ever changing economic environment. Here is what we know: Learning organizations only spend 3 – 5% of their training budget on evaluation Training data contain information about: Course effectiveness (e.g., K&S gain, use of skills, ROI) Business impact Areas for improvement © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

7. 7 CLD Data Trained 28,000 + people Awarded more than 1.5 M Continuing Professional Education (CPE) Credits Gathered almost 100,000 post-course training evaluations © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

8. 8 Training Evaluation Data We use a Web-based evaluation tool to gather & analyze evaluation results. Post-course evaluations are Smart Sheets. Questions are standardized (identical) across learning modalities (ILT, WBT, OLF); one or two questions are unique per modality. Data are comparable. Reports are easy to generate. Data are easy to download to Excel and SPSS. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

9. 9 Analytic Techniques Applied Correlation Factor Analysis Regression Structural Equation Modeling Survival Analysis © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

10. 10 Correlation Do Level 1 (Kirkpatrick / Phillips) results predict higher levels of evaluation? These are bogus data for illustrative purposes only. Value: If there is strong relationship between Level 1 and higher levels, then extensive follow-up studies may not be necessary. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

11. 11 Factor Analysis Can the evaluation forms be shortened? Which items are so highly correlated they can be eliminated? Only two items were found to be highly correlated. Value: reducing the evaluation form can increase response rates and yield more reliable data. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

12. 12 Regression Key Driver Analysis was conducted to predict two separate criteria: Overall satisfaction with the course Training will improve my performance Value: L&D course managers were able to focus on a few variables (questions) to determine areas for improvement rather than the entire evaluation form. The regression equation also provided priorities in the event multiple aspects of the course needed improvement. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

13. 13 Structural Equation Modeling Predicted “Training will improve job performance” Value: Helped L&D managers identify areas for improvement; included all relevant questions; structure provided a framework for understanding the interaction of variables. It became easier for L&D to communicate areas for improvement. It also provided new and more accurate benchmarks to replace the arbitrary 4.00 on a 5-point scale. Ralph Grubb (Performance Improvement Associates) assisted with analysis. Nick Bontis (McMaster University) is also doing work in this area. Only a portion of the model is shown here. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

14. 14 Survival Analysis Linked training histories to turnover results to show that increased training leads to employee retention. Value: Describes the relationship mathematically and graphically between training and turnover. Non-KPMG DataExample from Mattox & Jinkerson (2005) © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

15. 15 Business Intelligence Value BI is only valuable when it: Demonstrates a relationship / an effect Is easy to understand Is easy to communicate to leaders Connects logically to actions. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.

16. 16 Contact information Michele Graham, KPMG LLP 201 307 8122 John Mattox, KPMG LLP 615 591 1032 Heather Maitre, KPMG LLP 201 307 8177 Pete Sanacore, KPMG LLP 201 307 7495 All information provided is of a general nature and is not intended to address the circumstances of any particular individual or entity. Although we endeavor to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. No one should act upon such information without appropriate professional advice after a thorough examination of the particular situation. © 2009 KPMG LLP, a U.S. limited liability partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International, a Swiss cooperative. All rights reserved.