Phillips Associates Learn 11 Surprising Techniques for Obtaining Powerful Data From Level 1 Evaluations Learn 11 Surprising Techniques for Obtaining Powerful Data From Level 1 Evaluations Presented by: Ken Phillips Phillips Associates April 10, 2012

Phillips Associates 2AGENDAAGENDA ➤ Review Kirkpatrick/Phillips five level evaluation model ➤ Examine Level 1 evaluation facts ➤ Analyze strengths and shortcomings of a sample Level 1 evaluation form ➤ Discover 11 tips to create Level 1 evaluations that produce valued data

Phillips Associates 3 Levels of EvaluationMeasurement FocusTime Frame Level 1: ReactionParticipant reaction to a learning program Conclusion of program Level 2: LearningDegree to which participants acquired new knowledge, skills or attitudes Conclusion of program or within 6 to 8 weeks after Level 3: BehaviorDegree to which participants applied back-on-the-job what was learned 2 to 12 months Level 4: ResultsDegree to which targeted business outcomes were achieved 9 to 18 months Level 5: ROIDegree to which monetary program benefits exceed program costs 9 to 18 months KIRKPATRICK/PHILLIPS EVALUATION MODEL

Phillips Associates 4 LEVEL 1 FACTS* ➤ 92% of organizations evaluate at least some learning programs at Level 1 ➤ Organizations that use Level 1s on average evaluate 78% of all learning programs ➤ 36% of organizations view data collected as having high or very high value *2009 ASTD Research Study: The Value of Evaluation: Making Training Evaluations More Effective

Phillips Associates 5 WHY THE DISCONNECT? ➤ Level 1 Evaluations focus on learning department not business operation data ➤ Examples Facility Course design Facilitator Food ➤ The data collected is not used

Phillips Associates 6 WHAT’S THE SOLUTION? ➤ Create Level 1 Evaluations that produce data with high perceived value for both you & your business stakeholders ➤ Aggregate data and conduct trend lines or comparisons at group, project, department, regional level

Phillips Associates 7 SAMPLEEVALUATIONFORMSAMPLEEVALUATIONFORM

8 11 TIPS FOR CREATING LEVEL 1 EVALUATIONS THAT PRODUCE VALUED DATA

Phillips Associates 9 TIP 1 (CONTENT) Only ask questions that lead to actionable data and focus on important issues

Phillips Associates 10 WHAT’S WRONG WITH THESE? 3c. The workshop materials were well designed 5c. The room set-up and size were comfortable

Phillips Associates 11 TIP 2 (CONTENT) Write learner-centered not trainer- centered evaluation items* *Jim Kirkpatrick, “The New World Level 1 Reaction Sheets, “ unpublished article

Phillips Associates 12EXAMPLESEXAMPLES 3c. The workshop materials aided my understanding of the program content 5c. My learning was enhanced by the room set-up and size

Phillips Associates 13 TIP 3 (CONTENT) Where appropriate, match up qualitative questions with quantitative measures

Phillips Associates 14EXAMPLEEXAMPLE In a word, how would you describe this session? ____________________________ Using a number, how would you describe this session? No Great Value Value

Phillips Associates 15 TIP 4 (CONTENT) Include at least one item asking participants how relevant the learning event/material was to them and their job

Phillips Associates 16EXAMPLEEXAMPLE This: This: How would you rate the overall relevance of this session to you and your job? Not at all Very Relevant Relevant Not this: Not this: 3a. Overall I was satisfied with the program

Phillips Associates 17 TIP 5 (CONTENT) Create predictive questions that forecast participant learning, intent to apply what was learned back-on-the-job and likely impact on business results

Phillips Associates 18 EXAMPLE – LEVEL 2 LEARNING How much did you know about developing individual performance objectives before attending this session? No Thorough Knowledge How much do you know about developing individual performance objectives after attending the session? No Thorough Knowledge

Phillips Associates 19 EXAMPLE – LEVEL 3 BEHAVIOR How likely are you to apply the skills and behaviors learned in this seminar back-on-the-job? What obstacles, if any, are likely to prevent you from applying these skills and behaviors back on the job?____________________________ Not at all Likely Extremely Likely

Phillips Associates 20 EXAMPLE – LEVEL 4 RESULTS How likely are any of the key business metrics tracked by your department to improve as a result of you applying the knowledge and skills you learned in this program? Not at all Likely Extremely Likely How confident are you in your response? Not at all Confident Extremely Confident

Phillips Associates 21 WHAT’S WRONG WITH THIS? SA = Strongly Agree A = Agree D = Disagree SD = Strongly Disagree SAADSD

Phillips Associates 22 TIP 6 (MEASUREMENT) * Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”, Harvard Business Review, When collecting quantitative data using a Likert scale, create a response scale with numbers at regularly spaced intervals and words only at each end*

Phillips Associates 23 Not at all True Completely True EXAMPLEEXAMPLEThis: Not This: Not at all True Rarely True Occasionally True Somewhat True Mostly True Frequently True Completely True Never This: Not at all True Rarely True Occasionally True Somewhat True Mostly True Frequently True Completely True

Phillips Associates 24 TIP 7 (MEASUREMENT) * Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys” Use a response scale with an odd number of points (7, 9, & 11 point scales are best)*

Phillips Associates 25 ODD vs EVEN SCALE

Phillips Associates 26 TIP 8 (MEASUREMENT) Place small numbers at the left or low end of the scale and large numbers at the right or high end of the scale

Phillips Associates 27EXAMPLE Not at all True Completely True Not at all True This: Not This:

Phillips Associates 28 WHAT’S WRONG WITH THESE? 4b. Did the facilitator clearly describe the learning objectives? 4c. Did the facilitator keep the session lively and interesting?

Phillips Associates 29 TIP 9 (MEASUREMENT) Write items either as a continuum or as a statement

Phillips Associates 30EXAMPLESEXAMPLES This: This: How effectively did the AV materials used during the session help reinforce your understanding of the program material? Or This: Or This: The AV materials used during the session helped reinforce my understanding of the program material. Not at all True Completely True Not Effectively Very Effectively

Phillips Associates 31 TIP 10 (ADMINISTRATION) Only use Level 1 evaluations to improve a learning program not to prove something

Phillips Associates 32 TIP 11 (FORMAT) TIP 11 (FORMAT) Place questions regarding respondent demographics (e.g. name, title, department, etc.) at end of evaluation form, make completion optional and keep questions to a minimum

Phillips Associates 33REFERENCESREFERENCES Boehle, Sarah, “Remember that mean, crabby teacher in high school?”, Training, August 2006, pps Morrel-Samuels, Palmer, “Getting the Truth into Workplace Surveys”, Harvard Business Review, February 2002, pps Phillips, Ken, “Eight Tips on Developing Valid Level 1 Evaluation Forms”, Training Today, Fall 2007, pps. 8 & 14. Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8.

Phillips Associates 34 Ken Phillips Phillips Associates N. Wooded Glen Drive Grayslake, Illinois (847)