1. Metrics That Matter Learning Analytics
The standard in Learning Measurement
There has been lot’s of talk about measuring learning for years now, but it’s only really in the last months that we’ve seen real progress in being able to make assessments on how a learning programme has affected the performance of a company.
I was recently at a client meeting with BT and the Director of Learning said he needed to define, in the same way as his fellow managers – the marketing Director, Commercial Director for example, the value that his operation was providing to BT. A story that I’m sure that is close to heart – we are now expected to justify in financial terms what the company ‘gets back’ when we spend their money.

2. Metrics That Matter Learning Analytics
Web-enabled automated learning evaluation process
Post-class, follow-up and bespoke surveys
Detail, summary and management reporting
Internal and external benchmark comparisons
Analysis tools
Gives a better insight into training’s Key Performance Indicators
Demonstrate the business value of training
Improve performance of the enterprise
Better use of training resources and budgets
Increase support from across the enterprise for training

3. Method Kirkpatrick methodology and Phillips ROI layer … model shows an IIP ‘Level 0’ layer1
Did they like it? Course content, instructors, course materials venue, LMS experience, etc
Level 2
Did they learn?
Test knowledge, intentions, relation to own situation, application
Level
Level 3
Do they use it?
Time to job impact, level of job impact, barriers to use
provided to managers > > >
Learning objectives Manager completes with delegate
Level 4
What were the results?
Sales, services, quality, cycle time, errors, timescale, etc
Level 5
What’s the ROI?
Financial return, timescale, value, human capital value
Needed by managers > > >

4. Automated process Data Collection Data Storage Data Processing
Data Reporting
Post Event Survey
Follow Up Survey
Manager Survey
Instructor Survey
Scored Testing
Needs Assessments
Phillips ROI Input
Templates
Paper, web,
LMS Integration
ASP solution
SQL Server
Centralized storage
Secure environment
Raw data access
Sophisticated queries
OLAP technology
Multiple aggregations
Multiple cuts / views
Web-enabled reports
Push alerts
Phillips ROI report
Custom query
Custom reports
Analytical tools
Forecast calculator
Automated
Standard Level 1 (Post Event) Survey
Developed by learning measurement experts
Administered at the end of the learning
Will provide internal and external comparisons
Captures data on the Instructors, Physical environment, Training content, Learning effectiveness, Job impact, Business results, Perceived Value
Standard Level 3 (Follow Up) Survey
Utilized by many KnowledgeAdvisors customers
Automatically sent to participants sixty days after the training
Captures data on the following Learning effectiveness, Job impact, Business results, Realized Value
Data collection
Access to an Administration Module to coordinate surveying and manage the account (it takes seconds to administer measurement of a class)
Data collected in 3 ways:
Send a URL link to participants via after the class
Direct students to a specific website address
Collect via hard copy/paper and key or scan data into system
Unlimited data collection at no additional charge
Data processing and storage
Automated processing of raw data into summarized and aggregated reports
Centralized, electronic storage of all evaluation data
Completely outsourced, ASP solution in a secure, reliable environment
Free access to raw data downloads available upon request (72 hours notice)
Standard reports
Evaluation retrieval tool
Learner comments tool
Class evaluation summary report
Operational reports
Activity analysis
Comparative performance/standard reports
Variance reports
Hierarchy reports
Performance to Goals Reports
Alarms
Learning Levels Score Card
Human Capital ROI Score Card
ROI Calculators
Report Cards
Top/Bottom Report
Report Generator (custom query tool)
Valued time
Because data collection, processing and reporting is automated, you have more time to analyze the data and
turn it into actionable business intelligence

5. Post event learner survey - corporate providers

6. Follow up survey - learner

7. Follow up survey - manager

8. Comparative performance/standard reports
Compare the average (mean) survey performance results for the overall survey, question category, and question level detail at various form type (post event, follow up) and report type (class, course, curriculum, program, learning provider, client, business unit, location, methodology, and instructor) levels.
Client A
Client B
Client C
Client D
Client E

9. Performance to goals reports
Allows you to establish pre-set goals for each question category on the Post event and Follow up surveys and compare them to actual performance.

10. Variance reports
Calculate and display the percentage of responses that fall within user-preset bounds for the overall survey, question category, and question level detail at various form type (post event, follow up) and report type (class, course, curriculum, program, learning provider, client, business unit, location, methodology, and instructor) levels.

11. Learning levels score card – K1-5
The Learning Levels Score Card allows you to generate a report based on questions mapped to the Five Levels of Learning. You can then compare actual performance to internal or external benchmarks and preset goals.

12. Top/bottom report
Wade Media
Tonnell Grp
JKT North
Paris Textile
Qamoli
Dent Morcam
Customer
Nottingham
Aberdeen
Chester
London
Norwich
Liverpool
Displays the top and bottom performers as reported by the Overall Post Event (class scores).

13. What percent of skills were applied to the job
What percent of skills were applied to the job? This indicator provides insight into the amount of new knowledge/skills picked up in the training for each course (or curriculum or program) that the participant directly applied when they went back on the job as the data is collected several months after the training when the participant is back on the job.
When is the time to job impact? This lets you know if and when training was applied to the job. It is collected months after the training, automatically by Metrics that Matter. It helps the organization understand when training effects the job, if at all.
How much more productive are employees? You can track specific business results, such as productivity. Asking a manager or a participant to estimate productivity once time has passed on the job is a good indicator of learning’s impact to a specific business objective.

14. How is training aligned with business objectives?

15. ROI Score card The scorecard is a great management tool to showcase value to key stakeholders. This report shows you each level of learning and the Key Performance Indicators within each level.
Level five shows quantitative ROI metrics including:
Benefit to Cost Ratio
Payback Period
ROI%

16. Push Reporting
Allows you to setup reports to be sent to you via at anytime you desire so you do not have to go into MTM to retrieve your favorite reports.

17. Phillips ROI Toolset in MTM
A summary of the ROI Methodology to use as a reading reference
Examples of case studies applying the Phillips ROI Methodology
A list of selected publications of Jack and Patti Phillips
The authorized Phillips End of Program Data collection instrument
The authorized Phillips Follow Up Data collection instrument
The authorized Phillips Cost Calculation Template
The authorized Phillips Follow Up Data collection instrument
KnowledgeAdvisors will be working with Jack and Patti Phillips to continue adding additional tools with each quarterly update to the Metrics that Matter software!

18. Is training aligned with business results? Survey

19. Phillips ROI Toolset in MTM

20. Sample clients

21. Contact Information
For further information on Metrics that Matter please contact:
New Wave Learning
4a Flemming Court, Glasshoughton
Castleford, West Yorkshire.WF10 5HW
Tel:
Website: