Metrics That Matter Learning Analytics The standard in Learning Measurement There has been lot’s of talk about measuring learning for years now, but it’s only really in the last 12-18 months that we’ve seen real progress in being able to make assessments on how a learning programme has affected the performance of a company. I was recently at a client meeting with BT and the Director of Learning said he needed to define, in the same way as his fellow managers – the marketing Director, Commercial Director for example, the value that his operation was providing to BT. A story that I’m sure that is close to heart – we are now expected to justify in financial terms what the company ‘gets back’ when we spend their money.

Metrics That Matter Learning Analytics Web-enabled automated learning evaluation process Post-class, follow-up and bespoke surveys Detail, summary and management reporting Internal and external benchmark comparisons Analysis tools Gives a better insight into training’s Key Performance Indicators Demonstrate the business value of training Improve performance of the enterprise Better use of training resources and budgets Increase support from across the enterprise for training

Method Kirkpatrick methodology and Phillips ROI layer … model shows an IIP ‘Level 0’ layer 1 Did they like it? Course content, instructors, course materials venue, LMS experience, etc Level 2 Did they learn? Test knowledge, intentions, relation to own situation, application Level Level 3 Do they use it? Time to job impact, level of job impact, barriers to use provided to managers > > > Learning objectives Manager completes with delegate Level 4 What were the results? Sales, services, quality, cycle time, errors, timescale, etc Level 5 What’s the ROI? Financial return, timescale, value, human capital value Needed by managers > > >

Automated process Data Collection Data Storage Data Processing Data Reporting Post Event Survey Follow Up Survey Manager Survey Instructor Survey Scored Testing Needs Assessments Phillips ROI Input Templates Paper, web, email LMS Integration ASP solution SQL Server Centralized storage Secure environment Raw data access Sophisticated queries OLAP technology Multiple aggregations Multiple cuts / views Web-enabled reports Push email alerts Phillips ROI report Custom query Custom reports Analytical tools Forecast calculator Automated Standard Level 1 (Post Event) Survey Developed by learning measurement experts Administered at the end of the learning Will provide internal and external comparisons Captures data on the Instructors, Physical environment, Training content, Learning effectiveness, Job impact, Business results, Perceived Value Standard Level 3 (Follow Up) Survey Utilized by many KnowledgeAdvisors customers Automatically sent to participants sixty days after the training Captures data on the following Learning effectiveness, Job impact, Business results, Realized Value Data collection Access to an Administration Module to coordinate surveying and manage the account (it takes seconds to administer measurement of a class) Data collected in 3 ways: Send a URL link to participants via email after the class Direct students to a specific website address Collect via hard copy/paper and key or scan data into system Unlimited data collection at no additional charge Data processing and storage Automated processing of raw data into summarized and aggregated reports Centralized, electronic storage of all evaluation data Completely outsourced, ASP solution in a secure, reliable environment Free access to raw data downloads available upon request (72 hours notice) Standard reports Evaluation retrieval tool Learner comments tool Class evaluation summary report Operational reports Activity analysis Comparative performance/standard reports Variance reports Hierarchy reports Performance to Goals Reports Alarms Learning Levels Score Card Human Capital ROI Score Card ROI Calculators Report Cards Top/Bottom Report Report Generator (custom query tool) Valued time Because data collection, processing and reporting is automated, you have more time to analyze the data and turn it into actionable business intelligence

Post event learner survey - corporate providers

Follow up survey - learner

Follow up survey - manager

Comparative performance/standard reports Compare the average (mean) survey performance results for the overall survey, question category, and question level detail at various form type (post event, follow up) and report type (class, course, curriculum, program, learning provider, client, business unit, location, methodology, and instructor) levels. Client A Client B Client C Client D Client E

Performance to goals reports Allows you to establish pre-set goals for each question category on the Post event and Follow up surveys and compare them to actual performance.

Variance reports Calculate and display the percentage of responses that fall within user-preset bounds for the overall survey, question category, and question level detail at various form type (post event, follow up) and report type (class, course, curriculum, program, learning provider, client, business unit, location, methodology, and instructor) levels.

Learning levels score card – K1-5 The Learning Levels Score Card allows you to generate a report based on questions mapped to the Five Levels of Learning. You can then compare actual performance to internal or external benchmarks and preset goals.

Top/bottom report Wade Media Tonnell Grp JKT North Paris Textile Qamoli Dent Morcam Customer Nottingham Aberdeen Chester London Norwich Liverpool Displays the top and bottom performers as reported by the Overall Post Event (class scores).

What percent of skills were applied to the job What percent of skills were applied to the job? This indicator provides insight into the amount of new knowledge/skills picked up in the training for each course (or curriculum or program) that the participant directly applied when they went back on the job as the data is collected several months after the training when the participant is back on the job. When is the time to job impact? This lets you know if and when training was applied to the job. It is collected months after the training, automatically by Metrics that Matter. It helps the organization understand when training effects the job, if at all. How much more productive are employees? You can track specific business results, such as productivity. Asking a manager or a participant to estimate productivity once time has passed on the job is a good indicator of learning’s impact to a specific business objective.

How is training aligned with business objectives?

ROI Score card The scorecard is a great management tool to showcase value to key stakeholders. This report shows you each level of learning and the Key Performance Indicators within each level. Level five shows quantitative ROI metrics including: Benefit to Cost Ratio Payback Period ROI%

Push Email Reporting Allows you to setup reports to be sent to you via email at anytime you desire so you do not have to go into MTM to retrieve your favorite reports.

Phillips ROI Toolset in MTM A summary of the ROI Methodology to use as a reading reference Examples of case studies applying the Phillips ROI Methodology A list of selected publications of Jack and Patti Phillips The authorized Phillips End of Program Data collection instrument The authorized Phillips Follow Up Data collection instrument The authorized Phillips Cost Calculation Template The authorized Phillips Follow Up Data collection instrument KnowledgeAdvisors will be working with Jack and Patti Phillips to continue adding additional tools with each quarterly update to the Metrics that Matter software!

Is training aligned with business results? Survey

Phillips ROI Toolset in MTM

Sample clients

Contact Information For further information on Metrics that Matter please contact: New Wave Learning 4a Flemming Court, Glasshoughton Castleford, West Yorkshire.WF10 5HW Tel: 01977 628980 Email: info@newwavelearning.co.uk Website: www.newwavelearning.co.uk