Challenges for Designing Simulation Research KIM LEIGHTON, PHD, RN, ANEF DEVRY MEDICAL INTERNATIONAL’S INSTITUTE FOR RESEARCH & CLINICAL STRATEGY

Disclosures The presenter has no disclosures Acknowledge Dr. Mary Ann Cantrell, Dr. Ashley Franklin, and Dr. Gregory Gilbert with whom I have previously presented and published variations of this material HPSN 2016

Learning Objectives The participants will be able to: 1.Identify the state-of-the science in simulation-based research and the required rigor of future studies to advance the science base in this area of research. 2.Describe the purpose and function of a theoretical framework when designing a research study. 3.Outline common pitfalls related to simulation-based research and strategies to overcome them. HPSN 2016

Background  National Council of State Boards of Nursing (NCSBN) study  Predict increased integration of simulation across nursing curriculum  Further development and evaluation of simulation needed  Need for simulation-based research to advance the science—all levels of learners HPSN 2016

State of the Science  Need robust, complex intervention studies  Skill development and transfer from simulation to actual patient care settings  Many academic educators and staff development educators are unprepared for rigor of design and implementation of simulation-based research  Beyond standard research textbooks HPSN 2016

Five Common Challenges  Theoretical framework throughout study design  Intervention fidelity  Reliable and valid instrument use  Sample size and power analysis  Ethical integrity of research process HPSN 2016

Theoretical Framework A set of interrelated concepts that symbolically represent and convey a mental image of a phenomenon (Alligood, 2013). HPSN 2016

Theoretical Framework or Conceptual Framework? HPSN 2016 INACSL State of the Science Project

Theoretical Frameworks in Research  Existing theory –Quantitative  Good literature review  Outside of healthcare  Look in education and psychology literature  Create theory –Qualitative  E.g. Grounded theory HPSN 2016

Common Frameworks for Simulation Research  Kolb’s Experiential Learning Theory  Concrete experience, reflective observation, abstract conceptualization, active experimentation  Piaget’s Constructivist Theory  New knowledge builds on past experiences and knowledge  Deliberate Practice/Mastery Learning  10,000 hours/times  NLN/Jeffries Simulation Theory  Learning & facilitator, Educational practices, Design characteristics/Simulation, Outcomes  Benner’s Novice-to-Expert  Cognitive Load Theory HPSN 2016

Developing the Conceptual Basis of a Study An Idea for a Study is Identified The Research Team Brainstorms about the Problem’s Dimensions A Literature Review is Conducted Concepts are Identified from Research Findings A Theoretical Framework that describes the Relationships among these Variables is Chosen Research Questions (based on the Theoretical Framework) are developed HPSN 2016

Intervention Fidelity  Degree to which an intervention is implemented as originally designed  In simulation-based research intervention fidelity is maintained by:  Have a specific, written, detailed protocol  Clear role delineation of research team members  Validation of a scenario by content experts; pilot tested with participants similar to the target population  Detailed script and cues for each person enacting a scenario  Consistent use of props in a scenario  Conduct dry run of scenarios HPSN 2016

Validity HPSN 2016

Measurement Issues: Instruments  Should use valid and reliable instruments  Tested with similar population as your study  Measuring performance in simulation  Interrater reliability > 80% —checklists, rubrics  Tool development  Psychometric analysis  Statistician consultant  Fully describe in manuscript HPSN 2016

Measurement Instruments: Experience  The Simulation Effectiveness Tool - Modified (SET-M)  Prebriefing, Confidence, Learning, Debriefing  The Creighton Competency Evaluation Instrument (C-CEI ™ )  Assessment, Communication, Critical Thinking, and Technical Skills  The Sweeney-Clark Simulation Performance Evaluation Tool  Assessment, History Taking, Patient Teaching, Lab/Dx, Nursing Interventions, Clinical Judgment, Communication & Safety  The Lasater Clinical Judgment Rubric (LCJR © )  Noticing, Interpreting, Responding, Reflecting  The Clinical Simulation Evaluation Tool (CSET)  Safety, Assessments & Critical Thinking, Problem Identification & Critical Thinking, Interventions, Evaluations & Critical Thinking, Other Critical Thinking & Processing Components HPSN 2016

Other Types of Instruments  The Clinical Learning Environment Comparison Survey (CLECS)  Teaching-Learning Dyad, Holism, Communication, Nursing Process, Self- Efficacy, Critical Thinking  Indiana University Simulation Integration Rubric (IUSIR) (interprofessional communication); DASH (debriefing) HPSN 2016

Sample Size  Determined before you start a study  “Larger is better” principle?  Statistical approach  Population size (unknown)  Margin of error (confidence interval = +/- 5%)  Confidence level (95%)  Standard deviation (0.5)  Calculation of sample size (Z score for CL) 2 x (SD)(1-SD) (margin of error) 2 HPSN 2016

Power Analysis  Determines if sample size could be too small or large  What sample size is needed to detect an effect of a given size with a given degree of confidence 1.Sample size 2.Effect size: impact of the intervention on the variable 3.Significance level: P(Type I error) = 0.05, 0.01, or Power: 1 – P(Type II error) = 0.8  Need to know three of these to determine the fourth  Engage a statistical consultant! HPSN 2016

Ethical Integrity  Must have Institutional Review Board (IRB) approval to publish research findings  Simulation specific ethical concerns  Learning or evaluation  Students need to know purpose  Video use  Confidentiality—in sim vs protecting research protocol  Research misconduct  Misuse of data  Failure to declare conflict of interest  Plagiarism, falsification, fabrication HPSN 2016

Take Aways  Theoretical foundation is the basis for all decision-making when designing research study  Valid and reliable instruments must be used  Sample size and power analysis are determined BEFORE data collection begins  IRB must approve study if it is to be published  Use a statistician! HPSN 2016

Questions? Kim Leighton HPSN 2016