1. Phillips Associates Learn 11 Surprising Techniques for Obtaining Powerful Data From Level 1 Evaluations Learn 11 Surprising Techniques for Obtaining Powerful Data From Level 1 Evaluations Presented by: Ken Phillips Phillips Associates April 10, 2012

2. Phillips Associates 2AGENDAAGENDA ➤ Review Kirkpatrick/Phillips five level evaluation model ➤ Examine Level 1 evaluation facts ➤ Analyze strengths and shortcomings of a sample Level 1 evaluation form ➤ Discover 11 tips to create Level 1 evaluations that produce valued data

3. Phillips Associates 3 Levels of EvaluationMeasurement FocusTime Frame Level 1: ReactionParticipant reaction to a learning program Conclusion of program Level 2: LearningDegree to which participants acquired new knowledge, skills or attitudes Conclusion of program or within 6 to 8 weeks after Level 3: BehaviorDegree to which participants applied back-on-the-job what was learned 2 to 12 months Level 4: ResultsDegree to which targeted business outcomes were achieved 9 to 18 months Level 5: ROIDegree to which monetary program benefits exceed program costs 9 to 18 months KIRKPATRICK/PHILLIPS EVALUATION MODEL

4. Phillips Associates 4 LEVEL 1 FACTS* ➤ 92% of organizations evaluate at least some learning programs at Level 1 ➤ Organizations that use Level 1s on average evaluate 78% of all learning programs ➤ 36% of organizations view data collected as having high or very high value *2009 ASTD Research Study: The Value of Evaluation: Making Training Evaluations More Effective

5. Phillips Associates 5 WHY THE DISCONNECT? ➤ Level 1 Evaluations focus on learning department not business operation data ➤ Examples Facility Course design Facilitator Food ➤ The data collected is not used

6. Phillips Associates 6 WHAT’S THE SOLUTION? ➤ Create Level 1 Evaluations that produce data with high perceived value for both you & your business stakeholders ➤ Aggregate data and conduct trend lines or comparisons at group, project, department, regional level

7. Phillips Associates 7 SAMPLEEVALUATIONFORMSAMPLEEVALUATIONFORM

8. 8 11 TIPS FOR CREATING LEVEL 1 EVALUATIONS THAT PRODUCE VALUED DATA

9. Phillips Associates 9 TIP 1 (CONTENT) Only ask questions that lead to actionable data and focus on important issues

10. Phillips Associates 10 WHAT’S WRONG WITH THESE? 3c. The workshop materials were well designed 5c. The room set-up and size were comfortable

11. Phillips Associates 11 TIP 2 (CONTENT) Write learner-centered not trainer- centered evaluation items* *Jim Kirkpatrick, “The New World Level 1 Reaction Sheets, “ unpublished article

12. Phillips Associates 12EXAMPLESEXAMPLES 3c. The workshop materials aided my understanding of the program content 5c. My learning was enhanced by the room set-up and size

13. Phillips Associates 13 TIP 3 (CONTENT) Where appropriate, match up qualitative questions with quantitative measures

14. Phillips Associates 14EXAMPLEEXAMPLE In a word, how would you describe this session? ____________________________ Using a number, how would you describe this session? No Great Value Value 1 2 3 4 5 67

15. Phillips Associates 15 TIP 4 (CONTENT) Include at least one item asking participants how relevant the learning event/material was to them and their job

16. Phillips Associates 16EXAMPLEEXAMPLE This: This: How would you rate the overall relevance of this session to you and your job? Not at all Very Relevant Relevant 1 2 3 4 5 67 Not this: Not this: 3a. Overall I was satisfied with the program

17. Phillips Associates 17 TIP 5 (CONTENT) Create predictive questions that forecast participant learning, intent to apply what was learned back-on-the-job and likely impact on business results

18. Phillips Associates 18 EXAMPLE – LEVEL 2 LEARNING How much did you know about developing individual performance objectives before attending this session? No Thorough Knowledge 1 2 3 4 5 67 How much do you know about developing individual performance objectives after attending the session? No Thorough Knowledge 1 2 3 4 5 67

19. Phillips Associates 19 EXAMPLE – LEVEL 3 BEHAVIOR How likely are you to apply the skills and behaviors learned in this seminar back-on-the-job? What obstacles, if any, are likely to prevent you from applying these skills and behaviors back on the job?____________________________ Not at all Likely Extremely Likely 1234567

20. Phillips Associates 20 EXAMPLE – LEVEL 4 RESULTS How likely are any of the key business metrics tracked by your department to improve as a result of you applying the knowledge and skills you learned in this program? Not at all Likely Extremely Likely 1234567 How confident are you in your response? Not at all Confident Extremely Confident 1234567

21. Phillips Associates 21 WHAT’S WRONG WITH THIS? SA = Strongly Agree A = Agree D = Disagree SD = Strongly Disagree SAADSD

22. Phillips Associates 22 TIP 6 (MEASUREMENT) * Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”, Harvard Business Review, 2002. When collecting quantitative data using a Likert scale, create a response scale with numbers at regularly spaced intervals and words only at each end*

23. Phillips Associates 23 Not at all True Completely True 1234567EXAMPLEEXAMPLEThis: Not This: Not at all True Rarely True Occasionally True Somewhat True Mostly True Frequently True Completely True 1234567 Never This: Not at all True Rarely True Occasionally True Somewhat True Mostly True Frequently True Completely True

24. Phillips Associates 24 TIP 7 (MEASUREMENT) * Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys” Use a response scale with an odd number of points (7, 9, & 11 point scales are best)*

25. Phillips Associates 25 ODD vs EVEN SCALE

26. Phillips Associates 26 TIP 8 (MEASUREMENT) Place small numbers at the left or low end of the scale and large numbers at the right or high end of the scale

27. Phillips Associates 27EXAMPLE Not at all True Completely True 12 3 4 56 7 Not at all True 76 5 4 32 1 This: Not This:

28. Phillips Associates 28 WHAT’S WRONG WITH THESE? 4b. Did the facilitator clearly describe the learning objectives? 4c. Did the facilitator keep the session lively and interesting?

29. Phillips Associates 29 TIP 9 (MEASUREMENT) Write items either as a continuum or as a statement

30. Phillips Associates 30EXAMPLESEXAMPLES This: This: How effectively did the AV materials used during the session help reinforce your understanding of the program material? Or This: Or This: The AV materials used during the session helped reinforce my understanding of the program material. Not at all True Completely True 12 3 4 5 6 7 Not Effectively Very Effectively 1 234567

31. Phillips Associates 31 TIP 10 (ADMINISTRATION) Only use Level 1 evaluations to improve a learning program not to prove something

32. Phillips Associates 32 TIP 11 (FORMAT) TIP 11 (FORMAT) Place questions regarding respondent demographics (e.g. name, title, department, etc.) at end of evaluation form, make completion optional and keep questions to a minimum

33. Phillips Associates 33REFERENCESREFERENCES Boehle, Sarah, “Remember that mean, crabby teacher in high school?”, Training, August 2006, pps. 17-22. Morrel-Samuels, Palmer, “Getting the Truth into Workplace Surveys”, Harvard Business Review, February 2002, pps. 111-118. Phillips, Ken, “Eight Tips on Developing Valid Level 1 Evaluation Forms”, Training Today, Fall 2007, pps. 8 & 14. Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8.

34. Phillips Associates 34 Ken Phillips Phillips Associates 34137 N. Wooded Glen Drive Grayslake, Illinois 60030 (847) 231-6068 www.phillipsassociates.com ken@phillipsassociates.com