Slides to accompany Weathington, Cunningham & Pittenger (2010), Chapter 4: An Overview of Empirical Methods 1

Objectives Internal, statistical conclusion, and external validity Empirical methods Intact groups and quasi-experimental designs Surveys Correlational studies Single-N methods Meta-analysis 2

Internal Validity Shown by the degree to which a study rules out alt. explanations for IV  DV Requires ruling out alternative explanations Threats include sources of confounding variables –4 general categories 3

Threats to Internal Validity Unintended sequence of events –Carryover effects: drug at Time 1 hurts performance at Time 2 (but the drug is not what we wanted to test) –Maturation: Changes in answers between 6 and 10 year olds may be due to normal learning rather than a reading intervention – Intervening events: being burglarized may change your response to a social psychology experiment involving eye witnesses 4

Threats to Internal Validity Nonequivalent groups –Confounds interpretation of cause and effect between IV and DV –Can be caused by: Non-random sampling Mortality/attrition Subject characteristics (variables) 5

Threats to Internal Validity Measurement errors –Non-valid test –Low reliability of measurement –Ceiling and floor effects –Regression to the mean Ambiguity of cause and effect –Which came first, X or Y? 6

Statistical Conclusion Validity Were the proper statistical or analytical methods used when studying the data? “Proper” = best allowing the researcher to: –Demonstrate relationship between IV and DV –Identify the strength of this relationship 7

Threats to Statistical Conclusion Validity Low statistical power: increases risk of missing an effect that really exists Violating assumptions of tests: no statistical tests are perfect in all research situations; you need to know your “tools” Unreliability in measurement and setting: inconsistencies in the measurement process make it impossible for you to draw valid inferences from the statistics 8

External Validity Do our findings/results generalize beyond our sample? –More likely if representative sample Can we generalize our findings to the population? Can we generalize our conclusions from one population to another? 9

Internal vs. External Validity 10

Threats to External Validity NOT always just the “lab setting” Participant recruitment –How + who you select to study matters –Need to be as representative as possible May require replication, extension studies 11

Threats to External Validity Situation effects –Where you do the study matters –Control for what you can and consider replicating in different settings History effects –Be aware that phenomena may change over time 12

True Experiment Best method for testing cause and effect “Easiest” control for internal validity threats Not always a practical/ethical option You know it is a true experiment if: 1.The IV can be controlled/manipulated 2.Random assignment to conditions occurs 3.Control conditions can be created 13

True Experiment Nonrandom differences among the groups in terms of the measured DV leads us to conclude that the manipulations of the IV may have caused those differences Sampling frame Group 1 n= 10 Group2 n= 10 Treatment for Group 1 Group3 n= 10 Treatment for Group 3 Treatment for Group 2 Results for Group 1 Results for Group 2 Results for Group 2 Assuming random assignment into groups, differences among the groups at this stage are due to random effects Separate conditions controlled by the researcher (different levels of IV) Differences among groups due to random effects + effect of treatment (level of IV) Random assignment 14

Intact Groups Design No random assignment possible Multiple samples (by subject variables), from multiple populations Cannot establish cause and effect –Unknown 3 rd variable and temporal order Can compare differences across samples 15

Intact Groups Design 16

Quasi-Experimental Design No random assignment; grouping by some other factor An IV is manipulated One group is treated as a “control”, while the other is exposed to the manipulated IV Still problem with unknown 3 rd variable and temporal order 17

Quasi-experimental Design 18

How does the true experiment differ from the intact groups and quasi- experiment design? 19

Surveys For estimating population parameters Good for large-scale data collection –Quick and inexpensive “Bad” because of respondent error –Honesty and personal bias 20

Correlational Study Usually to estimate population parameters Often data from surveys Good for initial understanding and “prediction” of complex behaviors Bad at supporting cause and effect –Unknown 3 rd variable –Temporal order issues 21

Single-N Methods Sometimes better to focus in-depth on one or a few participants –Single-participant experiment –Case study Good if IV and situational variables are well- controlled Bad for generalizability (potentially) and also because of participant bias/error 22

Meta-analysis Analysis of multiple outcomes from multiple studies Good because takes advantage of more representative sampling of participants and measures/methods Bad because depends on which studies are entered –Principle of GI, GO 23

What is Next? **instructor to provide details 24