Planning: From Objectives to Outcomes Carolyn L. Cason, RN, PhD Susan Scheid, RN, MSN Smart Hospital TM University of Texas at Arlington The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Objectives Discuss core components to a healthcare simulation Discuss the importance of outcomes evaluation and challenges to traditional assessments Discuss the importance of validity, reliability, and feasibility as it relates to assessment Discuss types of assessments and their application in healthcare educa tion The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Strategy for the Session Describe the process used to develop a near end-of-program clinical performance exam Examine the –Validity, reliability and standardization challenges –Evaluation challenges –Performance outcomes Describe the ways in which the outcomes –Guided exam revision –Informed curriculum/program effectiveness The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Impetus for Exam Faculty desire for clinical performance assessment to complement the knowledge assessment obtained via the HESI Exit exam Faculty search for a meaningful and comprehensive approach to evaluation of program outcomes Faculty search for a way to benchmark/assess clinical competencies The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

UTA College of Nursing Junior IFoundations, Assessment, Health Promotion Junior IIMedical-Surgical Nursing, Psychiatric-Mental Health Nursing, Research Senior IPediatric Nursing, OB Nursing, Critical Care Nursing Senior IICommunity Health, Leadership & Management, Nursing Trends, Gerontology, Capstone The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

OSCE Objectives Provide safe patient care Obtain focused objective & subjective assessment Plan, implement and evaluate patient care based on nursing assessment & diagnosis Communicate effectively & professionally with the patient and healthcare team Complete appropriate documentation The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Pilot: July 2010 Clinical Skills –Manual B/P; O 2 administration, telephone order, IV management –NG insertion, IV start, urinary catheter insertion, restraints, I/O etc. Patient Care Management –1 st day post-op bowel surgery –Post-op hemorrhage The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Summer 2010 OSCE Dallas SUN April 2011 Copyright CLCason

Summer 2010 OSCE Dallas SUN April 2011 Copyright CLCason

Lessons Learned Summer 2010 More preparation of students and faculty needed Skills check separate from patient care scenarios Faculty involvement needed (Crucial) The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Lessons Learned Summer 2010 Inconsistencies Faculty decision to use OSCE as diagnostic assessment Incorporate more simulation into Nursing curriculum The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Fall 2010 OSCE Changes Increased faculty involvement Increased preparation of students Skills Check: separate day in context The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Fall 2010 OSCE Changes Patient scenarios individual and private - confidential Use OSCE as diagnostic tool for Capstone Clinical Development of simulation for all levels of undergraduate nursing program The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Fall 2010 Outcomes Skills Validation Safety: Patient identification failure (15-21%) No allergy check (26-86%) Inconsistent hand hygiene (21-28%) Sterile procedure contamination (23- 32%) Not using alcohol to clean skin or IV tubing ports (21-57%) The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Fall 2010 Outcomes Skills Validation Medication Administration: Failure to draw up or give insulin correctly (23-31%) Improper use of gloves (23-42%) Recapping used needles (25-39%) Inability to hang IVPB correctly (28- 32%) The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Fall 2010 Scenarios Focused patient assessment, vital signs, recognize change in patient condition, independent nursing judgment and action Capstone faculty – sometimes present during OSCE scenarios The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Fall 2010 Scenarios: OSCE Evaluation Similar student performance concerns Safety issues: –No patient identification ( 31-40%) –Inadequate hand hygiene (33- 44%) –Improper use of gloves (14%) No medication administration required for scenarios except IV Fluid The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Fall 2010 Skills and OSCE Deficiencies in student performance Related to lack of simulation experience Inconsistency of evaluators and operators - 20 different people involved in check-offs -Standardized pre-recorded voice responses of manikins -Operators – differences in prompting with responses -Assistance given by evaluator varied The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Student Feedback Skills & OSCE Fall 2010 Most students (more than 70%) agreed or strongly agreed : Time was sufficient Helped meet learning objectives Were able to transfer to clinical setting Conducive to learning The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Student Feedback Skills & OSCE Fall 2010 Some students (30-40%) were neutral or disagreed: Helped them gain confidence in role, skills, or clinical setting Felt it was not realistic (40%) Were not satisfied with method of learning The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Student Comments Four main categories of comments: Realism - Some very real, some unrealistic Liked individual thinking and action Wanted immediate feedback (debriefing) Confidence – mostly yes The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Student Suggestions Suggestions for Improvement: Immediate debriefing More simulation throughout program Opportunities to practice & more information The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Summary Fall 2010 OSCE Objectives stayed the same –Faculty wanted end of program evaluation –Student performance needed improvement Reliability: multiple evaluators and operators Validity: use as diagnostic tool Feasibility: labor intensive, costly Positive changes More work needed The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Spring 2011 Changes Faculty Involvement: Immediate feedback and debriefing by clinical instructors at time of scenario Emphasis on OSCE as plan for Capstone Clinical Scheduling of students by faculty The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Spring 2011 Changes Reversed order of scenarios for realism Revisions for consistency and standardization –Identical pre-recorded manikin voice responses –More instructions to students, faculty, operators, and evaluators The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Spring 2011 OSCE Outcomes Preliminary results for skills validation –Similar safety and medication concerns Slight improvement of hand hygiene Faculty presence and participation –Less student anxiety –Favorable student responses Overall improved outcomes? The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Planning from Objectives to Outcomes Program Outcome Objectives Clinical Performance Assessment Comprehensive approach to evaluation of program outcomes Determine Clinical Competency before graduation The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Planning from Objectives to Outcomes Student Objectives for Skills and scenarios: Provide safe patient care Obtain a focused patient assessment Plan, implement and evaluate patient care Communicate effectively The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Quality Assessment Reliable Valid Practical Replicable Generalizable The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason

Thank You Questions? The University of Texas at Arlington UTA College of Nursing Dallas SUN April 2011 Copyright CLCason