1. Evaluating simulations
Susan Prion EdD, RN, CNE
Professor
School of Nursing and Health Professions
University of San Francisco
San Francisco, CA, USA

2. “Tis better to have assessed incompletely than to never have evaluated at all”
More is better
Multiple methods
Multiple perspectives
Multiple times

3. Purpose of evaluation Assess learner/participant outcome achievement?
Purpose of evaluation
Assess learner/participant outcome achievement
Quality improvement
Evidence for additional resources

4. Evaluation, assessment & debriefing?
Evaluation, assessment & debriefing
Evaluation: making a judgment about performance
Assessment: giving feedback for the purpose of improvement
Debriefing: a purposeful, facilitated discussion and analysis of an experience by its participants

5. Evaluation or assessment?
In simulation learning experiences, assessment and evaluation are frequently used interchangeably.

6. Evaluation or assessment?
Learners/participants need to be clear if the purpose of the simulation is constructive feedback or a test (evaluation) of their performance.

7. Measurement in nursing education and research
Valid and reliable measurement tools are essential components of nursing research.
Poor or inadequate measurement tools limit the scope potential and quality of research the scope,potential and quality of research (DeVellis, 2003).
“Excellence in nursing education requires evidence-based curricula, teaching approaches, and evaluation methods” (Oermann, 2009, p. 2)

8. What to assess/evaluate?

9. What is the goal of the simulation?
What are the learner/participant learning outcomes?
What are the critical learning points for the simulation?
Is the purpose practice, feedback or testing?

10. participant
facilitator

12. Possible outcomes of simulation
Knowledge
Skills
Self-confidence
Satisfaction
Critical thinking

13. Other outcomes of simulation?
Teamwork
Delegation
Communication
Integration of theory and practice
Time management
Other?

14. HOW to assess/evaluate?

15. Student/participant learning outcomes
Direct information
Indirect information
Program outcomes
Student/participant learning outcomes
Formative
Summative ?

16. Direct: observation or demonstration of performance
Indirect: self-report or other’s report about performance ?

17. Program outcomes: did the program meet the stated goals?
Student/participant learning outcomes: did the students/participants accomplish or achieve the expected learning? ?

18. Formative: feedback is collected during the learning experience to make ongoing improvements
Summative: feedback is collected AT THE CONCLUSION of the learning experience to make improvements for the next time ?

19. LEARNING Self-report or instructor-report Pre-posttest?
LEARNING
Self-report or instructor-report
Pre-posttest
Tests in theory class
Academic paper
Demonstration (“tell me why…”)
Concept map
Observation of change in behavior
Journal or blog
Checklist of critical knowledge elements
Subsequent clinical simulation performance

20. Sample theory questions to evaluate student learning?

21. Sample clinical instructor report of increased student learning?

22. SKILLS Skills checklist Repeat demonstration
Self-assessment of increased skill proficiency
Instructor/preceptor observation of increased skill proficiency ?

23. Sample critical elements checklist?

24. Sample self-report of skill proficiency?

25. Rubrics for clinical evaluation*
Issacson, J.J. and Stacy, A.S. (2009). Rubrics for clinical evaluation: Objectifying the subjective experience. Nursing Education in Practice, 9,

26. Creighton Simulation Evaluation Instrument*
Assessment
Communication
Critical Thinking
Technical Skills
*Todd, M., Manz, J.A., Hawkins, K.S., Parsons, M.E., and Hercinger, M. (2008). The development of a quantitative evaulation tool for simulations in nursing education. International Journal of Nursing Education Scholarship, 5 (1), article 41.

27. STUDENT SATISFACTION Qualitative versus quantitative?
Social desirability of responses?
Relationship of satisfaction to learning?
Relationship of satisfaction to events?

28. 1. What was the best or most useful part of the simulation? Why?
Sample student satisfaction open-ended questions
1. What was the best or most useful part of the simulation? Why?
2. What was the least useful part of the simulation? Why?
3. What parts of the simulation experience would you change? Why?
4. Overall how satisfied are you with the simulation as a learning experience? ?

29. Sample student satisfaction items?

30. STUDENT SELF-CONFIDENCE?
Qualitative versus quantitative?
Social desirability of responses?
Relationship of confidence to learning?
Relationship of satisfaction to confidence to learning?
Relationship of confidence to simulation events?

31. Sample student self-confidence items?

32. Sample student self-confidence items
Do you feel more confident about XXX after participating in the clinical simulation? Why or why not?
What was the most influential in changing your self-confidence about performing Y procedure?
What activities could have helped increase your confidence about working with a patient with XXX? ?

33. CRITICAL THINKING Qualitative versus quantitative?
Novice characteristics?
Relationship of learning, skills, confidence and satisfaction to learning and critical thinking?
Measurement issues for critical thinking
Critical thinking or clinical judgment???

34. “…the ways in which the nurse comes to understand the problems, issues, or concerns of clients/patients, to attend to salient information and to respond in concerned and involved ways”
Benner, P., Tanner, C.A. & Chesla, C. A. (1996). Expertise in nursing practice: caring, clinical judgment and ethics. New York: Springer. ?

35. Tanner Clinical Judgment model
Noticing
Interpreting
Responding
Reflecting
Tanner, C.A. (2006). Thinking like a nurse: a research-based model of clinical judgment in nursing. Journal of Nursing Education, 45, ?

36. CRITICAL THINKING Self-report Clinical instructor/preceptor report
Demonstration
Observation
Written case study analysis with questions
Repeat simulation demonstration ?

37. Lasater Clinical Judgment Rubric
Lasater, K. (2007). High fidelity simulation and the development of clinical judgment: students’ experiences. Journal of Nursing Education, 46 (6), pp ).
Lasater, K. (2007). Clinical judgment development: using simulation to create an assessment rubric. Journal of Nursing Education, 46 (11), pp ?

38. Lasater, K. (2007). Clinical judgment development: using simulation to create an assessment rubric. Journal of Nursing Education, 46 (11), pp ?

41. Prion, S. K. , Gilbert, G. E. , Adamson, K. A. , Kardong-Edgren, S. E
Prion, S.K., Gilbert, G.E., Adamson, K.A., Kardong-Edgren, S.E. and Quint, S. (2016, accepted for publication). Development and testing of the Quint Leveled Clinical Competency Tool (QLCCT). Clinical Simulation in Nursing. ?

42. QLCCT 10 competencies 4 proficiency levels
Focused observation & info seeking
Recognizing deviations from expected patterns
Prioritizing data
Making sense of data
Response demeanor
Clear communication
Effective interventions
Nursing skills
Nursing evaluation
Self reflection & improvement
4 proficiency levels
4=Graduating nurse
3=Advancing
2=Progressing
1=Novice

44. WHEN to assess/evaluate?

45. Before the experience for baseline
During for formative improvements
Immediately after for summative changes
Later for application, transfer and retention ?

46. HOW to assess/evaluate?

47. Hard-earned advice….. Don’t do it alone.
Don’t develop your own tool unless absolutely necessary.
Consult an expert at least three times during the process-beginning, middle, and end.
Use a statistician from the very beginning.
Links not leaps!

48. Resources ?

49. International Nursing Association for Clinical Simulation and Learning (INACSL)

50. Clinical Simulation in Nursing professional journal

51. Prion, S.K. (2008). A practical framework for assessing the impact of clinical simulation experiences in the nursing curriculum. Clinical Simulation in Nursing Education, 4 (5).
Prion, S.K. (2016, under review). Updating the framework for assessing the impact of clinical simulation experiences in the nursing curriculum. Clinical Simulation in Nursing.

52. Society for Simulation in Healthcare

53. Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare

54. Simulation Innovation Resource Center (SIRC)

55. Prion, S. K. (2008, 2015). Evaluation involved in using simulations
Prion, S.K. (2008, 2015). Evaluation involved in using simulations. Available at resource/view.php?id=38. New York: National League for Nursing.

56. Kardong-Edgren,S. , Adamson, K. A. , and Fitzgerald, D. (2010)
Kardong-Edgren,S., Adamson, K.A., and Fitzgerald, D. (2010). A review of currently published evaluation instruments for human patient simulation. Clinical Simulation in Nursing, 6, e Adamson, K.A., Kardong-Edgren, S., and Willhaus, J. (2013). An updated review of published simulation evaluation instruments. Clinical Simulation in Nursing, 9, e

58. Simple linear regression
Multiple regression
Latent variable analysis
Pearson product moment correlation coefficient
MANOVA
ANOVA
Independent t-tests
Normal curve
Measures of variability
Significance
Cronbach’s alpha
Central tendency
Dependent t-tests
Frequencies
Effect size
Statistical power
The differences between parametric and non-parametric analysis
Chi-square
Post-hoc power analysis
Logistic regression
Choosing the right correlation coefficient
Generalizability theory
ANCOVA
Repeated measures studies
Multiple comparisons for ANOVA
Transformations to achieve normality
Real world research
Helpful resources
Ethics and participant recruitment
Rigor in qualitative research
Validity
Reliability
Need for rigor in simulation research
Operationalizing research variables
Formative and summative evaluation during simulation
Feedback, assessment and evaluation
Quantitative data analysis software
Qualitative data analysis softward
Role of qualitative research in simulation practice
Developing a data analysis plan for your research study
Program evaluation
Cost-effective simulation research
Developing strong research questions
Testing validity using Cain’s model
Quality improvement research and the SQUIRE Guidelines
Odds ratio
Lawshe’s content validity index
Is my data normal? Paramatric and nonparametric data analysis
Nonparametric measures of association

59. Obstacles to evaluation?

60. ? Time Resources Creativity Valid and reliable instruments
Clear student learning outcomes
Opportunity for follow-up
Recognition of importance of information ?

61. “Tis better to have assessed incompletely than to never have evaluated at all”
Multiple methods
Multiple perspectives
Multiple times

62. Questions? Comments? Great evaluation ideas?