Internal Validity and Confounding Variables Jenny and Nasim

Review - Terminology Independent Variable Dependent Variable Researchers study as having a possible effect on one or more variables [1] “mathematical variable that is independent of the other variables in an expression or function and whose value determines one or more of the values of the other variables” [2] Dependent Variable Variable that is maybe affected by independent variable [1] “mathematical variable whose value is determined by that of one or more other variables in a function” [3]

Validity - Review Source: Research Methods Knowledge Base http://www.socialresea rchmethods.net/kb/intr oval.php

Was there an alternate explanation? Internal Validity Can the results of your experiment be attributed to the manipulation of the independent variable? … or … Was there an alternate explanation?

Internal Validity - Demonstrate Causality Covariance - The Dependent Variable changes with the Independent Variable Temporal Precedence - The change in the Independent Variable occurred before the change in the Dependent Variable Did the cause occur before the effect? No Plausible Alternatives Confounding Variables - variation in the experiment other than the Independent Variable

Threats to Internal Validity Single Group Experiment: A: Run Experiment -> Measure Results B: Measure Baseline -> Run Experiment -> Measure Results

Single Group - Threats to Internal Validity History - Something outside of the study is causing the change Maturation - Change in the measure would have occurred naturally over time Testing - Pre-test might prepare participants for post-test Instrumentation - Change in the measurement between pretest and posttest Attrition - Participants dropping out of the study Statistical Regression - Group that doesn’t measure close to the mean in the pre-test moves closer to it in the post-test

Statistical Regression Also called Regression to the mean Nonrandom sample of the population Metric distribution is asymmetrical compared the population in one test Next test shows movement towards the mean even when nothing changed Occurs in both directions Statistical Phenomenon The result of random error Not related to Instrumentation Bias or other experiment design issues Group Phenomenon Relative Phenomenon Not related to the Maturation Threat

Statistical Regression - Example A Health Science Graduate Student is studying the impact a specific set of anaerobic exercises on HDL levels in adults. After months of developing his exercise program, it’s ready for evaluation. The experiment requires participants to follow the exercise program for 10 weeks. Participants are asked not change their diets or add an additional exercise not specified by the program. The Student measures HDL levels both at the beginning and conclusion of the study. To his dismay, the results show no statistically significant change in the mean HDL Levels. After half a year of work, the student has nothing to write about! The student start looking at different subsets in the test results and notices that the participants in the lowest quantile of pretest HDL value showed noticeable improvement in their posttest values. The research effort is saved and the student prepares to write his thesis.

Multiple Group Threats Same threats as single group threat with the added factor of differing impact on each group Focus on the differential changes between each group for each threat History Maturation Testing Instrumentation Attrition Statistical Regression

Social Threats Imitation of Treatment Compensatory Rivalry Resentful Demoralization Equalization of Treatment

Internal Validity - Example 1 A psychology research assistant is tasked with conducting a simple experiment that will take approximately 20 minutes. She decides recruit participants from the two classes goes to her advisors teaches; one is Introduction to the Practice of Psychology, PSY 110, offered at 5:45pm on Monday the other is a Graduate Seminar in Adolescent Psychotherapy, PSY 528, offer at 2:00pm on Tuesday. For convenience, she asks for 15 volunteers who can stay immediately after each class. She gets enough participants from each class and uses students from one class as a control and the others as the experimental group. She conducts the experiment with the same equipment, in similar classroom environments. Unfortunately, she the experiment runs longer than anticipated in both cases and 5 participants from the evening course have to leave before it’s finished to catch the last train home. The student still has data from 25 participants and decides to move forward with it.

Internal Validity - Example 2 A researcher is conducting a visual learning and retention experiment using Geography subject matter. He begins his research conducting a survey to identify regions of the world that would be least familiar to likely participants in his experiment. He then recruits participants who did not take his survey to attend seminars studying 1 of 2 geographical regions and randomly assigns them control and experimental groups for each region (a total of 4 groups in the study). He begins with a pretest for each group and concludes that the mean test scores are have similar distributions between control and experimental groups for both regions. This experiment requires groups to take a 4-week seminar, taking up approximately 30 minutes per week of the participant’s time. The participants are compensated with Starbucks gift cards to motivate them to continue taking the seminar. After the first week, a humanitarian crisis puts the one of the regions in the news regularly. Over the course of 4 weeks, the researcher found that participants in a group that meets at 6pm had attendance rates at 2/3 of the other groups and 25% more dropouts, this group was the control group for the region in the news. The pretest and posttest were similar to each other for both regions and the control and experimental group for region were administered the same test. In the end, the research found experimental group the region that wasn’t in the news showed noticeable improvement with the experimental learning program compared to the controlled, while control and experimental groups for the region in the news both showed similar improvements in test scores.

Mitigating Threats to Internal Validity Use a control group Hawthorne Effect Use of a Placebo Participants might be denied existing beneficial treatments or services Requirements for Informed consent must discuss use of the placebo Participants in placebo group should be offered treatment afterwards, if it’s beneficial Research should weigh value of study vs. risk to participants Keep things constant Constant factors can’t be confounding variables

Mitigating Threats to Internal Validity Randomly assign groups Any differences are due to chance Use pretest(s) Determine the equivalence of different groups Useful when groups cannot be random Expose participants to all treatments Within-subjects variable - Independent variable is systematically manipulated for all participants Statistically Control of Confounding Variables

References Leedy, P. D., & Ormrod, J. E. (2016). Practical research: planning and design. Boston: Pearson. Independent Variable. (n.d.). Retrieved April 30, 2017, from https://www.merriam- webster.com/dictionary/independent%20variable Dependent Variable. (n.d.). Retrieved April 30, 2017, from https://www.merriam- webster.com/dictionary/dependent%20variable Introduction to Validity. (n.d.). Retrieved May 01, 2017, from http://www.socialresearchmethods.net/kb/introval.php