1. A Comprehensive Framework for Evaluating Learning Effectiveness in the Workplace Presented by Dr Cyril Kirwan

2. The case for evaluating the ‘learning system’  Organisations in Europe spend between 1% and 4% of payroll costs on formal learning & development (L&D) activities each year  Estimates suggest only between 10% and 30% of this investment actually translates into changed behaviour back at work  Research has demonstrated the important influence of a variety of factors on the application of learning on the job (learning transfer)  In order to derive the most benefit from L&D resources, it is necessary to take account of these factors (the learning system)  If not, there is a risk of ignoring more than 70% of the factors that contribute to successful application of learning on the job Cyril Kirwan & Associates, Management and Organisation Development 2

3. Model of the learning system Motivation to learn Programme content and design Learning & Development PROGRAMME Peer supportMotivation to transfer Organisational climate for transfer Personal ability to transfer Manager support and coaching Learning and transfer outcomes Cyril Kirwan & Associates, Management and Organisation Development 3

4. Factors in the learning system Motivation to learn  The degree to which individuals are prepared to enter and participate in the programme Programme content and design  The degree to which the programme has been designed and delivered to make it easier for participants to transfer learning back to the job, and the degree to which participants regard the programme content as appropriate to their needs Motivation to transfer  The direction and persistence of effort in applying back at work skills and knowledge learned Personal ability to transfer  The extent to which individuals have the time, energy and mental space in their work lives to help transfer learning to the job Manager support & coaching  The extent to which supervisors/managers support and reinforce the use of learning on the job Peer support  The extent to which peers reinforce and support the use of learning on the job Organisational climate for transfer  The extent to which the work environment is conducive to the use of learning on the job Learning and transfer outcomes  The extent to which participants have changed their behaviour or gained new insights into themselves and/or their role as a result of attending the programme Cyril Kirwan & Associates, Management and Organisation Development 4

5. Evaluating learning effectiveness – two important questions  Why do I want to know this?  What will I do with the information when I have it? Cyril Kirwan & Associates, Management and Organisation Development 5

6. WORK ENVIRONMENT Peer, manager and organisation support Motivation and ability to learn and transfer OUTCOMES Reactions, Learning, Behaviour, Results, RoI INPUTS Training content and design EVALUATING LEARNING EFFECTIVENESS FRAMEWORKS TOOLS AND TECHNIQUES Cyril Kirwan & Associates, Management and Organisation Development 6

7. Areas to evaluate Three main areas of evaluation 1. Programme inputs, in particular programme content and design 2. Programme outcomes at reactions, learning, behaviour and results levels 3. Work environment characteristics Cyril Kirwan & Associates, Management and Organisation Development 7

8. Evaluation of programme outcomes – what to evaluate  Reactions level evaluation – did participants enjoy the programme, and/or find it useful?  Learning level evaluation – did their knowledge increase as a result of attending the programme?  Behaviour level evaluation – are they now doing things differently back at work?  Results level evaluation – has this had any effect on unit/team/organisational performance?  Return on Investment (ROI) level evaluation – was it worth the investment, in financial terms? Cyril Kirwan & Associates, Management and Organisation Development 8

9. A word of warning!  Generally speaking….  Outcomes at the reactions level are unrelated to those at other levels  Outcomes at the learning level are no better predictors of subsequent job performance than utility (how useful was the programme) reactions  While some correlations between outcomes at behaviour levels and organisational impact levels have been observed, they are influenced by many variables (Alliger and Janak, 1989; Alliger et al. 1997) Cyril Kirwan & Associates, Management and Organisation Development 9

10. Evaluation of programme outcomes – how to evaluate ReactionsLearningBehaviourResults QuestionnairesTestsBehavioural or competency questionnaires Performance data - internal Group reviewSimulationsFocus groupsPerformance data - external Self assessmentInterviews Trainer assessment Observation Action plans and assignments Critical incidents Cyril Kirwan & Associates, Management and Organisation Development 10

11. Evaluation of programme content and design  Based on valid and reliable learning needs analysis  Balance of theory and practice, in particular sufficient time allocated for practice  Variety of methods  Distributed vs massed learning  Adult learning principles  Learning situated in participants’ actual work  Reflection, goal setting and action planning  The skills and expertise of the trainer/facilitator Cyril Kirwan & Associates, Management and Organisation Development 11

12. Evaluation of work environment factors  Motivation to learn  Motivation to transfer  Personal ability to transfer  Peer support  Manager support  Organisational climate for transfer Cyril Kirwan & Associates, Management and Organisation Development 12

13. Tools and techniques  Choices between breadth and depth  Choices between measuring what and measuring how or why  Choices between qualitative and quantitative approaches  Choices between methods, such as questionnaires; focus groups; interviews; critical incidents; success case method; diaries; learning logs; observation; naturally-occurring organisational data; etc. Cyril Kirwan & Associates, Management and Organisation Development 13

14. Going deeper  Using a quantitative instrument such as Learning Transfer Evaluation (LTE), climate measure, satisfaction survey, engagement survey, workplace audit, etc. will give an indication where to focus questions for deeper insights  Interviews or focus groups can then be used to explore these themes further and inform recommendations Cyril Kirwan & Associates, Management and Organisation Development 14

15. In summary…  Organisations in Europe spend between 1% and 4% of payroll costs on learning and development  If only 20% of this investment makes it back to the workplace, a huge amount of time and money has been spent that hasn’t added any value  Even raising this figure to 40% will double the return you get on your investment! Cyril Kirwan & Associates, Management and Organisation Development 15

16. For more information…. Contact: Dr Cyril Kirwan Cyril Kirwan & Associates, Ltd. 6 Maywood Park, Raheny, Dublin 5, Ireland Tel: +353 1 8310469 Mobile: +353 86 8365434 Email: cyril.kirwan@gmail.com See also: Kirwan, C. (2009). Improving Learning Transfer: A Guide to Getting More Out of What You Put in to Your Training. Farnham: Gower. Kirwan, C. (2013). Making Sense of Organisational Learning: Putting Theory into Practice. Farnham: Gower. Cyril Kirwan & Associates, Management and Organisation Development 16

17. Any other questions I can help you with? Cyril Kirwan & Associates, Management and Organisation Development 17