1. Using Pre and Post Scenarios to Assess Skill Attainment in Educational Settings Allison Nichols, Ed.D. Evaluation Specialist West Virginia University Extension Service

2. Evaluation of Training Kirkpatrick’s Four-Level Model for Training Evaluation (1959a & b) – Reaction – how participants feel – Learning – knowledge, skills, attitudes – Behavior – change of behavior – Results – change in the setting in which participant works or lives

3. Effective Professional Development Assessment Evaluation of satisfaction is not enough – “Evaluations that focus narrowly on self-reported satisfaction with professional development activities offer inadequate guidance and direction to improvement efforts.” (Guskey, 2003)

4. Pre and Post Scenarios can: Measure skill attainment Assess how the learner will use newly acquired skills in real-life situations

5. Use of Scenarios in Professional Settings FBI – train new agents and assess mastery of skills (Witcomb, 1999) Medical educators – to assess ability to perform clinical skills (Davis et.al. 2000)

6. Other Reasons for Using Scenarios Scenarios help to ensure that trainees will use new skills, thus promoting the likelihood of results. – Studies show that trainees are more likely use their new skills if they have confidence that they will work. (Guskey, 1991, Lortie, 1975). Information gained from scenarios can be used to design follow-up evaluations

7. Other Reasons for Using Scenarios Scenarios can enhance other evaluation methodologies within an evaluation strategy Scenarios can be designed to collect various kinds of data Quantitative data Qualitative data Both quantitative and qualitative data

8. Examples TRANSFER Project – Alzheimer’s Disease training program Intergenerational Childcare class Focus Group Training workshop

9. Administrating Scenarios Administer pre- and post-training Send to participants ahead of training if possible Assure participants that they receive training that will allow them to complete the post-scenario After completion discuss scenarios to enhance learning and inform future evaluations

10. Preventing Measurement Bias Participants should complete scenarios before training begins or at least before instruction begins Include an additional scenario on the post- assessment

11. Analyzing Scenarios Qualitative – Example from Intergenerational Childcare class Evaluation question – “Can students visualize the intergenerational approach working in traditional childcare settings?” If this the answer is yes, students will be more likely to incorporate intergenerational activities in the settings where they work.

12. Pre Assessment – Students did not criticize the teacher for not accepting the intergenerational mission. – Students made excuses for the teacher: “Teacher lacks the social graces needed to deal with older adults.” – “Teacher is fearful of older people.” – Students said that they would not confront the parents if they did not want to enroll their children in an intergenerational program. Post Assessment – Students say, “Teacher who does not accept the intergenerational mission should be considered incompetent.” – Students said that they would defend the intergenerational program to the parents. – Students say, “If any teacher cannot, or will not, embrace the intergenerational mission, she should be encouraged to leave and find employment in a traditional child care setting.”

13. Analyzing Scenarios Quantitative – Example for the TRANSFER Project (Rankin, et.al. 1996) This was a 3-phase train the trainer program. The original group was trained by experts and subsequent groups were trained by trainees. Using statistical analysis such as T-tests and ANOVAS, we were able to associate the participants’ ability to review case material and make accurate decisions with their phase of training.

14. Analyzing Scenarios Combination of Quantitative and Qualitative – Example – Focus Group Training Workshop T-tests showed that participants had not gained the skills level expected and that more training needed to be done Content analysis of answers highlighted trainees strengths and weaknesses

15. Using Scenarios to Inform the Assessment of Results Follow-up questionnaire to examine use of skills in the workplace – Example: The TRANSFER Project Analysis looked at frequency and depth of use in different types of workplace Analysis looked at the value of the skill to practice in different workplaces

16. Using Scenarios to Inform Assessment of Results On site training – Example – Focus Group Training Information from the scenarios will help the trainer understand how to help trainees design focus group studies: – Which areas will they need help? – Which areas will they need practice? – Which areas they will they be most successful?

17. Conclusion Scenarios can be used to assess “learning” and “behavior.” (Kirkpatrick’s model) Scenarios can utilize both qualitative and quantitative data and can enhance other methodologies used in the evaluation. Scenarios can inform future assessment of results.

18. Sources Davis, P., Kvern, B., Donen, N., Andrews, E., and Nixon, O. (2002). Evaluation of a problem-based learning workshop using pre- and post-test objective structured clinical examinations and standardized patients. Journal of Continuing Education in the Health Professions, 20(3), 164-170. Guskey, T.R. (1991). Enhancing the effectiveness of professional development programs. Journal of Educational & Psychological Consultation, 2(3), 239-248.. Guskey, T.R. (2003). What makes professional development effective? Phi Delta Kappan, June, 748-750.

19. Sources Kirkpatrick, D. (1959a). Techniques for evaluating training programs. Journal for the American Society of Training Directors, 13, 3-9. Kirkpatrick, D. (1959b). Techniques for evaluating training programs. Journal for the American Society of Training Directors, 14, 21-26. Lortie, D.C. (1975). Schoolteacher: A sociology study. Chicago: University of Chicago Press. Rankin, E. D., Keefover, R. W., and Nichols, A. H. (1996). “Training rural health care and social service providers in assessment and care of dementia. Gerontology & Geriatrics Education, 17(2), 33-48. Witcomb, C. (1999). Scenario-based training at the F.B.I. Training and Development, 53(6), 42-46.