Introduction to Executive Reporting for CLOs Tamar Elkeles Vice President of QUALCOMMs Corporate Learning Center Version: 26-Feb-11

Learning measurement: Key performance indicators Source: KnowledgeAdvisors Learning Analytics Best Practices Research Study, Does your organization have a standard set of key performance indicators that you measure and monitor regularly? © 2011 KnowledgeAdvisors 2

The Measurement Gap for CEOs 2009 ROI Institute Study based on responses from 96 CEOs © 2011 KnowledgeAdvisors 3

The Idea is Born Conversation between Kent and Tamar in Spring 2010 Executive Council created to provide guidance Dave Vance asked to provide leadership Industry thought leaders have provided advice, insight, and further guidance © 2011 KnowledgeAdvisors 4

Executive Council Members Tamar Elkeles, Qualcomm, Chair Terry Bickham, Deloitte Cedric Coco, Lowes Marilyn Figlar, Lockheed Martin Karen Kochler, CIGNA Sundar Nagarathnam, Net App Sandy Shaw, Sodexo Don Shoultz, BP Tom Simon, CNA Insurance Carrie Beckstrom, ADP © 2011 KnowledgeAdvisors5

Industry Thought Leaders Providing Advice and Counsel to the Project Laurie Bassi Josh Bersin Rob Brinkerhoff Jack Phillips Frank Anderson Jac Fitz-enz © 2011 KnowledgeAdvisors 6

The Reporting Effort Goals Practical, relevant guidance for reporting in the talent development field Standard measures and definitions Standard reports Adopted by many organizations and becomes the standard for the industry Greater clarity and efficiency for each organization Ability to compare across the industry © 2011 KnowledgeAdvisors 7

The Three Phase Plan Page: 8 © 2011 KnowledgeAdvisors Phase IPhase IIPhase III Basic Reporting for L&D Reporting for Talent Development Expanded Reporting for L&D Focus of todays presentation: progress on Phase I

Executive Reporting for CLOs: What They Want and How to Get It Dave Vance President, Manage Learning LLC

The Need Clear guidance for L&D Reporting Data measures and definitions Similar to what GAAP provides for accounting Generally Accepted Accounting Principles Reports: Income Statement, Balance Sheet and Cash Flow Statement Measures and definitions: Income, expenses How to calculate and report Examples: Depreciation, value of an asset © 2011 KnowledgeAdvisors 10

Benefits Clear, practical reporting guidance for CLOs to use within their own organization Manage the L&D function to produce desired outcomes effectively and efficiently Provide appropriate, meaningful reports to the CEO © 2011 KnowledgeAdvisors 11 Standard measures and definitions to benchmark and learn from other organizations

Phase 1 Deliverable Reflects Efforts of Many Kents vision and commitment Executive Council input and support Thought leaders guidance and counsel Key leaders at Knowledge Advisors Peggy Parskey Jeffrey Berk Kendall Kerekes Page: 12 © 2011 KnowledgeAdvisors

Overview Two primary audiences for reporting CLO, including managers and staff CEO, including governing boards and senior leaders Four recommendations Adopt a set of guiding principles Adopt standard measures and definitions Adopt three standard statements Adopt three standard reports 13 © 2011 KnowledgeAdvisors

Focus of the Executive Reporting Process Data Statements Reports Standard Measures Summary Conclusions, Actionable Recommendations, Issues for Further Analysis Data Sources and Systems System Wide Analytics Program/ Initiative Analytics Systems and processes for organizing data, calculating measures Specific methodology (e.g. Phillips, Brinkerhoff) Scope of White Paper

Data Aggregation and Reporting Page: 15© 2011 KnowledgeAdvisors Standard Measures Low Aggregation of Data Level of Customization Low High Summary Statements Summary Reports Detailed Statements Detailed Reports Scope of White Paper High

The Assumptions L&D broadly defined: All programs and initiatives designed to increase human capital to enable greater organizational and personal capacity, effectiveness, and efficiency Page: 16 © 2011 KnowledgeAdvisors Primary purpose of L&D: Enable organization to achieve its goals Learning is strategically aligned to goals Recommended reports used appropriately

The Eight Principles 1.Employ concise and balanced measures reported in a consistent and clearly defined format 2.Produce and communicate Executive Reporting with a frequency and thoroughness to enable appropriate management of the function 3.Include actionable recommendations in all Executive Reporting 4.Maintain data integrity and completeness Page: 17 © 2011 KnowledgeAdvisors

The Eight Principles 5.Employ appropriate analytical methods 6.Provide the impact and value or benefit of programs and initiatives whenever appropriate 7.Capture and report the full costs of L&D 8.Support continuous improvement of executive reporting and the underlying data bases Page: 18 © 2011 KnowledgeAdvisors

Executive Reporting Process Page: 19© 2011 KnowledgeAdvisors Learning Executive Reports Learning Executive Reports Business Executive Summary Report Business Outcomes Statement Business Outcomes Statement Learning Effectiveness Statement Learning Efficiency Statement Learning Efficiency Statement Executive Reports Statements Data Sources Learning Management System Other Sources (e.g HRIS, ERP, CRM) Financial Data Evaluation System Extract, convert and calculate Standard Measures Extract, convert and calculate Standard Measures L&D Summary Report L&D Program Report L&D Operational Report Business Outcomes Efficiency Effectiveness Data Sets Guiding Principles

Standard Measures Business Outcomes Organizational goals L&Ds impact on those goals Effectiveness Levels 1-5, timeliness Efficiency Costs Volume (participants, programs, classes, hours) Ratios Utilization rates Program and vendor management 20 © 2011 KnowledgeAdvisors

The Statements Three standard statements Business Outcome Effectiveness Efficiency Each statement shows Last years actual Plan (or goal) for this year Year-to-date results Standard measures are used but choice of measures depends on organization 21 © 2011 KnowledgeAdvisors

Business Outcome Statement 22© 2011 KnowledgeAdvisors

Effectiveness Statement November 10, © 2011 KnowledgeAdvisors

Efficiency Statement November 10, © 2011 KnowledgeAdvisors

The Reports Highly customized, pulling measures from the statements Three levels of reports L&D Summary Report L&D Program Report L&D Operations Report Each report shows Last years actual Plan (or goal) for this year Year-to-date results Forecast for this year 25© 2011 KnowledgeAdvisors

Data Aggregation and Reporting Page: 26© 2011 KnowledgeAdvisors Standard Measures Low Aggregation of Data Level of Customization Low High Summary Statements Summary Reports Detailed Statements Detailed Reports Scope of White Paper High

L&D Summary Report November 10, © 2011 KnowledgeAdvisors

L&D Program Report November 10, © 2011 KnowledgeAdvisors

L&D Operations Report November 10, © 2011 KnowledgeAdvisors

Summary Executive Reporting effort provides Guiding principles for reporting (GARP) Standard measures and definitions Standard statements Executive reports Benefits Provide industry guidance on reporting Support better management of the function Support benchmarking and cross company comparisons 30 © 2011 KnowledgeAdvisors

Next Steps Continue to share and socialize the work Webinars Workshops Industry begins to implement Refine and expand the effort for L&D (Phase II) Extend the effort to the rest of talent development (Phase III) This is an open, collaborative effort; no copyrights 31 © 2011 KnowledgeAdvisors

Conclusion Please share your thoughts with us Other suggested measures Difficulty in applying standard measures Need for better definitions Write to Dave Vance: At This PowerPoint deck White Paper (70-page) Overview Paper (25-page) 32 © 2011 KnowledgeAdvisors