Educational Research 102: Selecting the Best Study Design for your Research Question Francis S. Nuthalapaty, MD 2010 APGO Faculty Development Seminar

Disclosures No relevant financial disclosures to declare

Learning Objectives Describe types of research and study designs Understand the characteristics of a good research question Understand how to convert the research question into a hypothesis Understand how to select the most appropriate study design to test the hypothesis

What do we already know? 1. Issenberg SB, McGaghie WC, Petrusa ER, Gordon DL and Scalese RJ (2005) Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Medical Teacher 27, 2, pp Fletcher KE, Davis SQ, Underwood W, Mangrulkar RS, McMahon LF Jr, Saint S. Systematic review: effects of resident work hours on patient safety. Ann Intern Med Dec 7;141(11): TRUE FALSE

Write an educational research question in which you have interest

Types of Research Empirical vs. Non-empirical Basic vs. Applied

Empirical vs. Non-Empirical Research Empirical Involves collection of data first hand Non-Empirical No first hand data collection

Example: Empirical or Non-empirical? The Effect of House Staff Working Hours on the Quality of Obstetric and Gynecologic Care Bailit, J et al, Obstet Gynecol 2009 OBJECTIVE: To measure the effect of house staff working hours reforms on the quality of obstetric and gynecologic care.

Example: Empirical or Non-Empirical? Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Issenberg SB et al, Med Teach 2005 OBJECTIVE: Review and synthesize existing evidence in educational science that addresses the question, 'What are the features and uses of high-fidelity medical simulations that lead to most effective learning?'.

Types of Research Empirical vs. Non-empirical Basic vs. Applied

Basic vs. Applied Research Basic Results apply to a great many people and situations Results are related to general theory or to a general field of knowledge Results need not have immediate or even clear implications Applied Results are applicable only to a specific group of people in a particular situation. Results are not necessarily related to a broader field of knowledge Results must have immediate and clear implications for practice

Example: Basic or Applied? Assessing Vaginal Surgical Skills Using Video Motion Analysis Diwadkar G et al, Obstet Gynecol 2009 OBJECTIVE: To demonstrate the feasibility of using video motion analysis to quantitate a key step of vaginal hysterectomy and define measurable differences between novice and experienced surgical trainees during vaginal hysterectomy.

Example: Basic or Applied? Effects of a Depression Education Program on Residents’ Knowledge, Attitudes, and Clinical Skills Learman L et al, Obstet Gynecol 2003 OBJECTIVE: To determine whether an interactive educational program would improve obstetrics and gynecology Residents' knowledge, attitudes, confidence, and skills in caring for depressed patients.

Research Methods What is the difference between Qualitative Research and Quantitative Research?

Qualitative vs. Quantitative Research Qualitative Descriptive and exploratory focus Used to gain insight into attitudes, behaviors, values Analysis of unstructured information Narrative reporting Quantitative Hypothesis driven Used to identify association and/or causation Analysis of discrete variables Statistical reporting

Quantitative Research Experimental & Quasi-experimental Non-Experimental –Causal-comparative –Correlational

Quantitative Research Experimental & Quasi-experimental –Assess effect of an independent variable on dependent variables –Comparison of 2 or more groups –Control over ‘treatment’ & measurement –Randomization –Control group

Quantitative Research Designs Study Population Treatment Control Randomize Measurement

Quantitative Research Designs Study Population Treatment Control Measurement Randomize Measurement

Quantitative Research Designs Study Population Treatment Control Measurement Treatment Measurement Control Measurement Randomize

Quantitative Research Designs Treatment Measurement Study Population Study Population

Quantitative Research Designs Treatment Measurement Study Population Study Population

Quantitative Research Designs Treatment 1 Measurement Study Group 1 Study Group 1 Treatment 2 Measurement Study Group 2 Study Group 2

Quantitative Research Designs Treatment 1 Measurement Study Group 1 Study Group 1 Treatment 2 Measurement Study Group 2 Study Group 2 Measurement Matching

Quantitative Research Non-Experimental –Causal-comparative –Correlational

Causal-Comparative Baseline differences are pre-existing –Non-permutable: Ethnicity, Gender –Permutable: Teaching style Determine the cause or consequences of differences Associations can be identified Causality cannot be determined

Quantitative Research Designs Exposures Outcome Control Outcome

Quantitative Research Designs Exposures Outcome Exposures Control

Quantitative Research Designs Exposure Outcome Group 1 Group 2

Correlational Research Single group of subjects Describe degree to which 2 or more quantitative variables are related Help explain important human behaviors Predict likely outcomes Identifies associations Causality cannot be determined

Correlational Research

Correlation Coefficient –Calculated by regression <.35 = only a slight relationship = possible theoretical value.50 = minimum for crude predictions >.65 = reasonably accurate predictions >.85 = close relationship –Positive correlation = direct association –Negative correlation = inverse association

Threats to Internal Validity Subject characteristics Loss of subjects Maturation Repeated measures Statistical regression Investigator bias

External Validity Can the findings from the study be generalized to larger populations? –Subject characteristics –Environment (lab vs. natural setting) –Did the subjects act differently b/c they were enrolled in a study (Hawthorne effect)?

What are the steps in the research process?

The Research Process Define Research Question Define Research Question Conduct Literature Review Conduct Literature Review Refine Research Question Refine Research Question Design Study Design Study Obtain IRB Approval Obtain IRB Approval Collect Analyze Data Collect Analyze Data Disseminate Results Disseminate Results No hole in the literature Hole in the literature

FINER Research Questions F – Feasible I – Interesting N – Novel E – Ethical R - Relevant

FINER Research Questions Feasible: –Can be investigated with available: Time Money Space Other resources

FINER Research Questions Interesting: –Interesting to you –Interesting to others in the field Novel: –Will answers to the question advance knowledge in the field?

FINER Research Questions Ethical: –Will any physical or psychological harm come to anyone as a result of the research? Protect participants from harm Ensure confidentiality of data Avoid knowing deception of participants Relevant: –How might answers to this question improve educational practice?

What is a research hypothesis and how is it different from a research question?

Research Hypothesis A prediction regarding the possible outcomes of the study Wallen & Fraenkel. Educational Research: A Guide to the Process. 2 nd Ed.

Research Hypothesis A statement which: –Summarizes the elements of the study The sample The design The predictor and outcome variables Yoder E, MERC Workshop, 2009 AAMC Annual Meeting

Research Hypothesis Based upon a FINER research question Simple – one difference or relationship Specific – clearly stated, defined Stated in advance – before data collection Yoder E, MERC Workshop, 2009 AAMC Annual Meeting

Research Hypothesis May be stated: –As the alternative hypothesis Non-directional –The difference is unknown Directional hypothesis –Existing studies/data suggest direction of relationships –As the null hypothesis Yoder E, MERC Workshop, 2009 AAMC Annual Meeting

Research Hypothesis H a : There is a difference between groups or variables –Female medical students have a different IQ than male medical students –μ f ≠ μ m or μ f > μ m H 0 : There is no difference between groups or variables –Female medical students have a different IQ than male medical students –μ f = μ m Yoder E, MERC Workshop, 2009 AAMC Annual Meeting

Example: Hypothesis Improving Resident Competency in the Management of Shoulder Dystocia With Simulation Training Deering S, et al, Obstet Gynecol 2004 METHODS: Residents from 2 training programs were randomized by year-group to a training session on shoulder dystocia (SD) management that used an obstetric birthing simulator or to a control group with no specific training. Both groups were subsequently tested on a standardized SD scenario, and a physician grader rated the resident's performance with a standardized evaluation sheet.

Example: Hypothesis What is the research question? What is the null hypothesis? What is the research study design? –Basic or Applied? –Qualitative or Quantitative? –Experimental or Non-Experimental

Quantitative Research Designs Study Population Treatment Control Randomize Measurement

Small Group Activity Share your research questions Select one research question to convert to a hypothesis Select the most appropriate study design

Give Your Research Impact Investigate important questions Connect your study to prior studies and help build the body of work Uses appropriate research design Consult with a biostatistician in the design phase Dissemenate your results

Session content, including narrated MS Powerpoint slides available at: