SNDT Women's University Introduction to Evaluation and Assessments  Presented by Kathleen (Kat) Miller, Senior Consultant with Booz Allen Hamilton  4 August, 2007

2 SNDT Women's University Evaluation Model – Donald Kirkpatrick  In 1959, Donald Kirkpatrick developed what has become one of the most popular models for evaluating education programs.  Kirkpatrick's system has four levels of evaluation.

3 SNDT Women's University Evaluation Purpose of Evaluation “The reason for evaluating it to determine the effectiveness of a training program.” © Donald L. Kirkpatrick, “Evaluating Training Programs: The Four Levels,”

4 SNDT Women's University Main Reasons for Evaluation  Determine how to improve future programs  Determine whether a program should be continued or dropped.  Justify the existence of the training department

5 SNDT Women's University Kirkpatrick’s Level 1 - Reaction  Measure of “Customer Satisfaction”  Measures the emotional reaction of participants after a learning experience. –It attempts to answer questions regarding the participants' perceptions: –Did they like it? –Was the material relevant to their work? –Did it meet their expectations? –How should the learning experience be improved?

6 SNDT Women's University Why Measure Reaction?  It provides valuable feedback and suggestions  It makes the trainees feel that their input is important  It is easy to do effectively A positive reaction promotes motivation!

7 SNDT Women's University Guidelines for Evaluating “Reaction” 1.Determine exactly what you want to find out 2.Design a form the will quantify reaction 3.Encourage written comments and suggestions 4.Get a 100 percent immediate response 5.Get honest responses 6.Develop acceptable standards 7.Measure reactions against standards and that action 8.Communicate (as appropriate)

8 SNDT Women's University Kirkpatrick’s Level 2 - Learning  Measures whether or not a student achieved the learning objectives during the learning event or during a series of such events.  For example: we might test for written communication skills after instruction to determine if the learner is now qualified for the job. Before Training After Training

9 SNDT Women's University Why Measure “Learning?” Kirkpatrick defines learning as, “the extent to which participants change attitudes, increase knowledge and/or skill as a result of attending a program.”  Was knowledge increased?  Were skills developed?  Were attitudes changed?

10 SNDT Women's University Guidelines for Evaluating “Learning” 1.Use a control group if possible 2.Use pre/post tests –Written for knowledge measurement –Performance for skills 3.Get 100 percent response 4.Use results to take appropriate action

11 SNDT Women's University Kirkpatrick’s Level 3 - Behavior  Measures whether learners are able to apply their new knowledge and skills to their job. –Are they using those skills on the job? –Are there other issues that are stopping them from being successful on the job? –Has their behavior changed?  Information for Level 3 evaluations is generally gathered via surveys and personal interviews.

12 SNDT Women's University Four Conditions for Behavior to Change  The person must have the desire to change  The person must know what to do and how to do it  The person must work in the right climate  The person must be rewarded for changing

13 SNDT Women's University Guidelines for Evaluating “Behavior”  Use a control group if possible  Allow time for the change in behavior to take place  Evaluate before and after the program  Survey or interview the trainee, as well as someone who observes their behavior  Get 100 percent response  Repeat the evaluation at appropriate times  Consider cost versus benefits

14 SNDT Women's University Kirkpatrick’s Level 4 - Results  Increased Production  Improved Quality  Decreased Costs  Reduction in Accidents  Increased Sales  Reduced Turnovers  Higher Profits

15 SNDT Women's University Challenges to Evaluating “Results”  Some are intangible  Results may not occur for a long time  Many other factors impact results  The trainer cannot control all factors Such is hard to control all those wild cats!

16 SNDT Women's University Guidelines for Evaluating Results  Use a control group if possible  Allow time for results to be achieved  Measure both before and after  Repeat the measurement at appropriate times  Consider cost versus behavior Be satisfied with evidence, if proof is not possible

17 SNDT Women's University Assessments What is an assessment ? “Any systematic method of obtaining evidence from posing questions to draw inferences about the knowledge, skills, attitudes, and other characteristics of people for a specific purpose” © Copyright Questionmark Corporation 2004

18 SNDT Women's University Types of Assessments  Exam  Test  Quiz  Survey

19 SNDT Women's University Uses of Assessments  Diagnostic  Formative  Needs  Reaction  Summative

20 SNDT Women's University Diagnostic Assessments  Typically used in pre-learning  Identity skill gaps and needs  Identity student preferences

21 SNDT Women's University Formative Assessment  Provide feedback for the learners  Search and retrieval  May contribute to overall grades

22 SNDT Women's University Needs Assessments  Assess knowledge, skills, abilities, and attitudes  Serve as a gap analysis  Determine what should be taught based upon needs

23 SNDT Women's University Reaction Assessments  Students’ opinions are collected  Kirkpatrick's Level 1  Instructor can alter the learning experience for the future

24 SNDT Women's University Summative Assessments  Summary of knowledge/skill of the learner  Provides a quantitative grade  May result in certification

25 SNDT Women's University Stakes of an Assessment Assessment “stakes” are the consequences to the candidate  Low  Medium  High

26 SNDT Women's University Comparison Assessment TypeAssessment UseAssessment Stakes ExamSummativeMedium, High TestDiagnosticLow, Medium QuizFormativeLow SurveyNeeds, Reaction, Diagnostic Low

27 SNDT Women's University Summary  Kirkpatrick’s Evaluation Model  Types of Assessments  Uses for Assessments  Assessment Stakes

28 SNDT Women's University Thank You for Your Time! Contact me at: Goodbye from your new “virtual” friend, Kat Miller