Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research

Leading Questions ‘A measure is not valid if it is not reliable.’ Why do you think this can be the case? In academic research, why do you think we need to pay attention to research validity? Do you know various types of research validity? If so, what are they?

Validity Validity is related to the accuracy, correctness and legitimacy of the measurements and observations made during data collection. Validity is also related to the soundness of the inferences made on the basis of the data collected. To consider validity, we need to consider theories, how a construct is defined, how instruments are developed and used and how data are analyzed and interpreted.

Reliability Reliability is often related to the issue of the consistency of research instruments, observations, or measurements of a construct. An experimental study cannot be considered valid if it uses unreliable data from participants to analyze and answer the research questions. Example, using a reliable and valid reading score to claim how someone is good at writing is invalid and inappropriate.

Reliability The reliability of instruments is concerned with the degree to which the results of a questionnaire, test or other measuring instrument are consistent. The reliability of the research result of a study is concerned with the degree to which the research result (e.g., the difference between experiment and control groups) is likely to reappear.

Internal and External Validity Internal validity is closely related to what is considered and done during research processes. External validity is related to a generalization of the study to other participants and settings

Internal Validity Is concerned with the logic of the causal-like relationship between the independent and dependent variables under examination. Is related to the extent to which other confounding variables influence the research outcomes. The less confounding variables interference, the better the internal validity of the study prerequisite of external validity

Threats to Internal Validity Two major groups of threats that need to be considered: Threats related to research participants (e.g. selection bias, history effect, maturation, attrition) Threats related to research instruments and procedures (e.g. experimenter/researcher effect, testing effect, instrumentation effect, statistical regression effect )

Threats to Internal Validity

Threats relevant to research participants Selection bias : When there are major pre- existing differences between participants in the treatment and control groups. The history effect : when a specific situation or event that takes place during an experimental study and interferes the dependent variable of interest. The maturation effect : biological, cognitive or psychological developments that occur naturally within the participants of the experimental study.

Threats relevant to research participants The attrition or mortality effect : when there is an imbalance in the loss of participants across comparison groups. The Hawthorne effect : when experimental participants change their behaviour due to the mere fact of the experiment taking place, rather than the specific treatment of the experiment). The Placebo effects : when experimental participants believe that they are receiving a special treatment that improve their current condition.

Threats relevant to research participants The John Henry effect : when participants in the control group invest more effort in their learning to compete with those in the experimental group. The demoralized effect : how students in the control group feel that they are not being treated fairly. The diffusion effect : situations in which the participants in an experimental group share what the details of the special treatment with those in the control group.

Threats relevant to research instruments/procedures The experimenter/researcher effect : the unintentional influence of the experimenters on the research outcome. The testing effect : when researchers use the identical pretest and posttest. The instrumentation effect : when researchers use different instrument for measuring the dependent variable for the pretest and posttest.

Threats relevant to research instruments/procedures The novelty effect : the innovative look of a treatment or method that may excite learners, thereby causing them to be enthusiastic about the treatment. The statistical regression effect : when participants with extreme scores (e.g., the highest or lowest) in the pretest achieve scores in the posttest which are closer to the mean score (i.e., the average group score).

External Validity generalizability of the inferences made on the basis of an experimental finding to other learners (with similar characteristics) and other settings whether the study can be useful for other people -- the importance of the problem being addressed by an empirical study

Specific Aspects of Validity Construct, content, criterion-related, and statistical validities are related to the internal validity of the study. Construct Validity : related to research constructs The construct validity of a research instrument concerns the extent to which an instrument measures what it is intended to measure. The construct validity of research concerns the question of whether the inferred causal-like results support the theory behind the research.

Specific Aspects of Validity Content Validity : the extent to which sample behaviors or abilities are relevant to and representative of the construct being defined. Criterion-related Validity : an instrument should have a strong relationship with other instruments that measure the same or similar construct. Statistical Validity : appropriate statistical analyses leading to the inferences and conclusions; use of the probability value (e.g. p < 0.05)

Specific Aspects of Validity Predictive Validity : the level of predictability of current test scores, or reported behaviours to future scores, or behaviors. Predictive validity is related to the external validity. Face Validity : is not validity in the sense in which the term has been used so far. Face validity is related to the appearance of a research instrument being used or an experiment being conducted.

Discussion In your view, why is it difficult to separate reliability from validity? Which one do you think more important for an experimental study: internal validity versus external validity? Why? If you had to choose only one of the following types of validity for your study (namely construct validity, predictive validity, or statistical validity), which one would you choose? Please explain your reasons.