1. Informal Learning Measurement John R. Mattox, II, Ph.D. Director of Research KnowledgeAdvisors

2. Our Promise to You November 13, 20132© 2011, KnowledgeAdvisors KnowledgeAdvisors will improve learning effectiveness within your organization.

3. KnowledgeAdvisors 3 Experts in metrics that matter KnowledgeAdvisors is a human capital analytics firm. We are the worlds largest provider of learning and talent measurement solutions. November 13, 2013© 2010, KnowledgeAdvisors

4. Features and Benefits November 13, 20134© 2011, KnowledgeAdvisors

5. KnowledgeAdvisors Clients 5November 13, 2013© 2011, KnowledgeAdvisors

6. Human Capital Process Model 6November 13, 2013© 2011, KnowledgeAdvisors

7. Informal Learning Measurement Approaches 7November 13, 2013© 2010, KnowledgeAdvisors What is Informal Learning Research on Informal Learning tactics Measurement in Informal Learning

8. Informal Learning 8November 13, 2013© 2010, KnowledgeAdvisors Informal Learning: Knowledge transfer that occurs without the assistance of structured curriculum. Source: KnowledgeAdvisors

9. What Informal Learning is Supported? 9November 13, 2013© 2010, KnowledgeAdvisors 73%56%54%46%13% % of respondents supporting each type of informal learning # of respondents Source: KnowledgeAdvisors Informal Learning Research, 2009

10. How Much is Spent on Informal Learning? 10November 13, 2013© 2010, KnowledgeAdvisors % supporting each method $70K$85K$170K$110K Avg spend Source: KnowledgeAdvisors Informal Learning Research, 2009 9% of L&D budget on Informal Learning

11. Informal Learning Measurement 11November 13, 2013© 2010, KnowledgeAdvisors Are your Execs asking for quantitative measures? Are you measuring business benefits? While only 21% of respondents indicated that their Executives are asking for quantitative measures, twice that number expect to be asked soon. Source: KnowledgeAdvisors Informal Learning Research, 2009

12. Measurement Correlates to Budget 12November 13, 2013© 2010, KnowledgeAdvisors Measurement discipline L&D Spend / person COPsM&CVKSEPSS Those who measure $958$168,000$84,000$250,000 Those who dont measure$1004$68,929$82,679$97,917$129,286 Overall$996$71,042$83,026$109,615$167,750 % delta ( over ) -5%143%6%155%93% The group measuring Informal Learning spends less per person on L&D annually. However, their Informal Learning spend is notably higher than the non- measurement group in most areas. We might conclude that measuring impact of Informal Learning enables organizations to demonstrate outcomes and justify greater allocation of L&D $ to these critical areas. Source: KnowledgeAdvisors Informal Learning Research, 2009

13. Informal Learning Measurement : When? 13November 13, 2013© 2010, KnowledgeAdvisors 1. Become aware / consider 3. Engage 4. Use, learn, practice, contribute 5. Complete and/or recommit At each interaction Periodically At the conclusion 2. Search, access, sign in or join Source: KnowledgeAdvisors Informal Learning Measurement White Paper, 2010

14. Informal Learning Measurement : How? 14November 13, 2013© 2010, KnowledgeAdvisors Systems Based Web analytics Supporting system statistics Performance data Tools Based Needs analysis Usability studies Feedback links Content ratings Surveys Assessment People Based Observation Interviews Focus groups Case studies High TechHigh Touch Source: KnowledgeAdvisors Informal Learning Measurement White Paper, 2010

15. Match Method to Timing 15November 13, 2013© 2010, KnowledgeAdvisors MethodAt interaction points PeriodicallyAt conclusion Web analytics X Web feedback X Supporting system data XX Surveys XX Content ratings (or single question) X Usability studies X Interviews (1-1) XX Focus groups X Source: KnowledgeAdvisors

16. Informal Learning in Metrics that Matter® 16November 13, 2013© 2009, KnowledgeAdvisors Leverage Micropolls and Pop-up surveys for point of interaction on-demand resources performance support systemsc virtual knowledge networks Poll as learners consume the content just in time Utilize thought leading standard surveys for mentoring, performance support, on-demand resource, and community of practice programs Report results by filtering by program, delivery method, and content

17. Metrics that Matter Easy wizard for administration of informal learning measurement

18. Metrics that Matter Micro Poll Experience

19. Metrics that Matter Pop Up Experience

20. Metrics that Matter Standard Report on Informal Learning

21. Metrics that Matter Standard Report on Informal Learning

22. Take-away Information Informal learning is complex Measure successfully by embracing three factors November 13, 201322© 2009, KnowledgeAdvisors Program Components Measurement Tools Appropriate Timing

23. Take-away Information Evaluating Informal Learning is not simple Multiple components Web content, CoPs, Coaching Multiple measurement tools Micropolls, pop-up surveys, end-of- program evaluations, formal survey Many points in time point of interaction, periodically, end-of- study November 13, 201323© 2009, KnowledgeAdvisors

24. Recommendations Define the scope of the informal learning that needs to be measured Select the right data collection tools Determine appropriate timing Leverage available systems (e.g., MTM) Seek guidance (e.g., from KnowledgeAdvisors) November 13, 201324© 2009, KnowledgeAdvisors

25. Contact Information For further information on the information contact: John R. Mattox, II, Ph.D. Director of Research +1 615 591 1032 jmattox@knowledgeadvisors.com www.knowledgeadvisors.com