Session 3 Overview of Research Designs Introduction to Research and Evaluation in Education

Quality in Quantitative Research 2 Validity (does it measure what it intends to measure) Internal: Are the observed differences on the dependent variable a direct result of manipulation of the independent variable, not some other variable? External (Ecological): Are the results generalizable or applicable to groups and environments outside of the experimental setting? Construct: Does the test measure an intended hypothetical construct or trait that explains behavior? Content: Does the test measure an intended content area, as determined by expert judgment? Reliability (is the measurement consistent) Objectivity

Examples of threats to validity 3 InternalExternal HistoryMultiple-treatment interference MaturationHawthorne effect TestingNovelty and disruption effects InstrumentationExperimenter effect Experimental mortalityPre-test or post-test sensitization RivalryInteraction of history and treatment effects Resentful demoralizationInteraction of time of measurement and treatment effects

Quantitative Research Designs Single Group, Experimental and Quasi-Experimental Research Causal Comparative and Correlational Research Survey Methods Single-Case Research 4

Single Group Designs 5 One-shot case study (X O) One-group pretest-posttest (O X O) Time series (O O O O X O O O O) R=Random assignment X=Experimental Treatment O=Observation of the dependent variable

Experimental Designs 6 Design TypeDesign Characteristics Pretest-posttest control groupTreatment group Control group R O X O R O O Posttest only control groupTreatment group Control group R X O R O Single-factor multiple-treatmentTreatment group 1 Treatment group 2 Control group R O X1 O R O X2 O R O O Solomon 4-GroupTreatment group 1 Control group 1 Treatment group 2 Control group 2 R O X O R O O R X O R O FactorialMultiple dependent or independent variables; study of individual variables and their interaction Cluster RandomizationRandom assignment to group; Data analysis by members of the group. R=Random assignment; X=Experimental Treatment; O=Observation of the dependent variable

Quasi-Experimental Designs Static-group comparison design two non-randomly formed groups, one receives treatment, both posttested Nonequivalent control group design two groups, pre and post, one receives treatment Regression-discontinuity (R-D) design looks for discontinuity in regression line between individuals who score lower (treatment group) and higher (control group) than some cutoff score 7

Causal comparative focuses on group comparisons Correlational focuses on magnitude of relationships between two variables Challenges: Can obscure heterogeneity of a population Can confuse biological characteristics with social constructs Assuming causality (Post hoc fallacy) Researchers’ beliefs about differences and similarities Group identification Assuming similarities within racial and ethnic groups (Myth of homogeneity) Correlation coefficient statistic (Range between 0 and +/-1) 8 Causal comparative & correlational research

Survey Methods Challenges Response rate (70%) Literacy levels Children Teenagers Disability perspectives Sampling concerns 9 ApproachFrequencySample characteristics Simple descriptiveOne-shotOne group Cross-sectionalOne-shotSeveral groups LongitudinalSeveral installmentsOne group or cohort

Single-case Research Experimental with N=1 Design options Phase change: Withdrawal, Reversal, or ABAB(+) Multiple-Baseline designs (treatment administered successively to different participants or one participant in several settings) Behavior: test treatment/intervention’s effectiveness to change behavior in a person People: test treatment/intervention’s effectiveness with different people Settings: test treatment/intervention’s effectiveness to change one person’s behavior in multiple settings. 10

Quality in Qualitative Research 11 Credibility (parallels internal validity) Prolonged, persistent engagement; peer debriefing; member checks; triangulation Transferability (parallels external validity) Thick description; multiple cases Dependability (parallels reliability) Confirmability (parallels objectivity) Chain of evidence Transformative Fairness; ontological authenticity, community, attention to voice, positionality, reciprocity, social change

Qualitative Methods Ethnographic Case study Phenomenological research Grounded theory Participatory research Clinical research Focus groups Historical-narrative research 12

Judging Quality in Mixed Methods QuantitativeQualitativeMixed Method/Model Internal validity External validity Reliability Objectivity Credibility Transferability Dependability Confirmability Authenticity Transformative Multiple purposes/questions Match purposes/questions to appropriate method Adherence to criteria that define quality for the quantitative portion of the study Adherence to criteria that define quality for the qualitative portion of the study Addresses conflicting demands between paradigms Limitations in supplemental data collection Integration of results, and explanation of conflicting findings Responsiveness to practical and cultural needs of subgroups 13

References Mertens, D. (2010). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods, 3 rd edition. Thousand Oaks, CA: Sage. Denzin, N. K. and Lincoln, Y. S. (Eds.) (2000). Handbook of qualitative research, 2 nd edition. Thousand Oaks, CA: Sage. Lewins, A. and Silver, C. (2007). Using software in qualitative research: A step-by-step guide. Thousand Oaks, CA: Sage. Cronk, B. (2008). How to use SPSS: A step-by-step guide to analysis and interpretation. Glendale, CA: Pyrczak Publishing. 14