1. Assessing Shoulder Dystocia Simulations for Quality
Relieving the stuck shoulders is a cognitive dance of hands countering the inherent and primal response to pull the head. Studies that examined the efficacy of shoulder dystocia maneuvers uniformly agree that the maneuvers must be performed calmly, but quickly, in order to decrease fetal morbidity. (Gherman, Ouzounian, & Goodwin, 1998) The maneuvers lend themselves to repetition and can be practiced via low-fidelity simulation or within the practice and drills of complex emergency events involving high-fidelity scenarios and full multi-disciplinary teams (Fahey, 2008). The potential for the variety of programs and anticipated impending global utilization, justify the need for the standardization of simulation assessment and enhanced external validity.
Elizabeth Yznaga, CNM, MSN,
sDNP University of San Francisco, Kaiser Permanente San Jose

2. Quality Shoulder Dystocia Simulations
Quality simulations balance internal and external validity.
They are designed to demonstrate replicability, integrity and generalizability.
Quality can be observed and tracked by process checklists and assessment forms.
Participants recognize quality as learner support, excellent feedback, and a coherent reflection of real life events. Prion, 2008
Over the last five years, evidence that simulation can improve patient safety has influenced accrediting agencies such as the US Joint Commission on Accreditation of Healthcare Organizations to recommend simulation of critical events such as the obstetrical emergency of shoulder dystocia. (JCAHO, 2004) In addition recent evidence that shoulder dystocia simulation decreases fetal morbidity, (Draycott, 2008) is expected to promote said simulation for accreditation and medico-legal concerns.
As hospitals and educational sites embrace simulation drills and trainings, the increasing variability of design, personnel, and outcomes make external comparisons between simulations difficult.

3. Inherent structural validity
Unique restrictions of time, language, personnel, and actions promote internal and external validity
Place
Process, People
Statistical analysis
Treatment of the process
Specificity
Internal Validity
Generalizability
Organizational design formulas and protocols for simulations create internal validity within a specified simulation series, by the replication of the scenarios, the consistency of treatments, and by the reliability of the observers’ assessments.
The external validity of a simulation is more difficult to establish without a close review of the characteristics that define comparable, operational structures in simulation design. The Jefferies Simulation Design Characteristics (Objectives, Fidelity, Problems Solving, Student Support, and Debriefing, Jeffries, 2007)
The SIRC web course Evaluating Simulations by Susan Prion, provides a theoretical basis for exploring the essential characteristics of shoulder dystocia simulations that enhance external validity.
External Validity

4. Quality characteristics
Clear objectives: Learning, practice or drill
Reliability of the content: Evidenced-based, and standardized
Reliability of the experience: Scripted scenario, consistency in treatments ( fetal manikin), clear directions, and endpoints Jefferies (2007)
Consistent trainers: Standardized cues and other participant support
Documentation: Checklists for process and assessment, audiovisual records decrease bias in assessment
Thoughtful Feedback: Respectful, standardized forms with empirical, aesthetic, personal, ethical, and reflection prompts Johns (2004), Jefferies (2007)
Evaluation: Promotes improvement and integrity
Prion suggests that, assessment feedback given to maintain and improve performance is the important feature and most important focus of simulation. “Assessment is not associated with a grade or any other type of formal measurement, as the goal is to give helpful information that will assist the learner to master the content or skill.” (Prion, 2008).

5. Tracking quality
Increase fidelity with checklists: the setting, participant roles, manikin, and equipment
Standardized forms for observation and audiovisual assessments
Pre and post testing of content and confidence
Electronic simulators are high fidelity by their limited nature
The plastic pelvis and doll are low fidelity unless consistent treatments are followed
Quality is not conferred by the manikin, but by the scenario

6. Design for the site Scheduled simulations Birth center and home birth
Best control of most factors
Education costs
Easier to use audio video technology
Simple checklists
Small teams
Multiple responsibilities
On shift simulations
Hospital sites
Flexible participants
Informal team building
Short programs
Shift acuities may degrade the simulation
Multidisciplinary teams
Complexity = participants
Multiple checklists and evaluators

7. Define the basic tasks Performance check points Identify the problem
Call for help: assistants, nurses, pediatricians, neonatal code team
Communicate with the staff and the patient and family
Call out and perform the maneuvers
Document the delivery
Feedback

8. Assessments: measuring quality
Track the study or course methods of assessment
Study
Pretest/
training
Posttest/
Written Test in Class
Student Self-Report/feedback
Clinical Instructor Feedback
Clinical Elements Checklist
Repeat Demo/ Other
Crofts 2006
Baseline test
1-3 wks
185 questions
Force data
Video recording
Key events
Actor rated com
Track the shoulder dystocia maneuvers
Identifies the problem
Call for help
Rubins
Woods
Posterior arm
MacRobert’s
Suprapubic pressure
Com/ Doc.
Time
Track the internal validity of the scenario
Clear objectives
Fidelity
Adherence to scenario
Students are supported
Reflective feedback
Evaluation
Scenario 1

9. Summative Evaluation Simulation Summative Evaluation Scorecard
Category
Element 1 2
Comments
Knowledge and understanding
Clear objectives
Appropriate level of complexity
Skill performance
High expectations of everyone
Collaboration and teamwork
Learner satisfaction
Active learners
Diverse learning
Fidelity for engagement
Critical thinking abilities
Appropriate debriefing
Appropriate level of problem solving
Self confidence
Appropriate student support
Total
Total out of 20

10. Summary Design the simulation to reflect the site, the objectives
Maintain internal validity by the use of scripts, checklists and consistent trainers
Develop external validity by sharing scripts, checklists and assessments with other entities
Design basic simulations for on-site, spontaneous practice