1. PhD Research Seminar Series: Valid Research Designs
Dr. K. A. Korb
University of Jos

2. Outline Threats to valid Research Design Construct Validity
Internal Validity
External Validity
Reliability
Dr. K. A. Korb
University of Jos

3. Reliability: Consistency of results
Reliable
Unreliable
Dr. K. A. Korb
University of Jos

4. Validity: Measuring what is supposed to be measured
Invalid
Dr. K. A. Korb
University of Jos

5. Research Example
The effect of regular leisure reading on reading achievement in primary school.
Students in a primary school will be randomly assigned to either a treatment or a control group.
The treatment group will spend 10 minutes everyday reading a book with an adult.
In an attempt to keep the treatment and control groups as similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems.
Dr. K. A. Korb
University of Jos

6. Research Example IV: Regular Leisure Reading DV: Reading Achievement
Experimental Group
Reading
Achievement
Read Book
Every Day
with Adult
Reading
Achievement
Control Group
Reading
Achievement
Do Maths
Every Day
with Adult
Reading
Achievement
Dr. K. A. Korb
University of Jos

7. Construct Validity
Construct Validity: Establishing valid operational measures for the concepts being studied
This relates to:
Independent Variable
Dependent Variable
Review:
Construct Definition: General explanation of the construct by relating it to other constructs
Operational Definition: Statement of specifically how the construct will be measured or implemented in your study
Dr. K. A. Korb
University of Jos

8. Review: Construct vs. Operational Definitions
WAEC Scores
Final Course Exam
Academic Achievement
NECO Scores
State Exams
Self-Report of Parents’
Yearly Income
Economic
Status
Free/Reduced
Lunch Eligibility
Dr. K. A. Korb
University of Jos

9. Research Example Construct validity focuses on:
IV:
Regular
Reading
DV:
Reading
Achievement
Experimental Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Reading
Achievement
Control Group
Reading
Achievement
Dr. K. A. Korb
University of Jos

10. Construct Validity
When you develop your research project, you start with a theory.
For example, regular leisure reading will lead to increased reading achievement scores.
Then you operationalize your theory into your research study.
You translate “regular leisure reading” into how it will be implemented in your study – reading a book with an adult everyday for 10 minutes.
You translate “reading achievement scores” into how it will be measured in your study – classroom exam scores.
When you finish collecting data, you hope to be able to move back to theory.
You want to be able to say that not only did reading a book for 10 minutes everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general.
However, in order to do that, you must have construct valid measures of your independent and dependent variables.
Dr. K. A. Korb
University of Jos

11. Research Example Theory: Research Study: Theory: IV: Regular Reading
DV:
Reading
Achievement
Theory:
Read book
every day with
an adult for
10 min.
Scores on
reading section
of classroom
exams
Research
Study:
IV:
Regular
Reading
DV:
Reading
Achievement
Theory:
Dr. K. A. Korb
University of Jos

12. Construct Validity
Enables you to generalize from the treatment and measures in your study to the general constructs of your treatment and measures
Threats to Construct Validity
Inadequate explanation of constructs in your paper
Flawed matching of operationalization to constructs
The measures do not align with the constructs they were designed to measure.
Dr. K. A. Korb
University of Jos

13. Internal Validity
Internal validity: Extent to which variables other than the treatment provide plausible explanations to the experimental results
Only relevant for experimental designs
Primary consideration for interventions
Extraneous variable: Any variable other than treatment variable that if not controlled can affect the experimental outcome
Hold constant or eliminate all extraneous variables that might affect the posttest
Goal of Research Design: Create set of conditions so any observed changes in your dependent variable can be attributed to experimental treatment instead of extraneous variables
Do variables other than the treatment provide plausible explanations to the experimental results?
Dr. K. A. Korb
University of Jos

14. Research Example Maturation IV: Regular Reading DV: Reading
Achievement
Interest
Experimental Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Reading
Achievement
Control Group
Reading
Achievement
Reading
Ability
Adult
Attention
Dr. K. A. Korb
University of Jos

15. Threats to Internal Validity
History: If treatment lasts over time, other events may influence dependent variable
To prevent: Keep the control group equivalent in all aspects but treatment
Maturation: Physical or psychological changes may influence the dependent variable
To prevent: Use a control group
Testing: Giving a pretest may increase performance on the post-test
To prevent: Don’t use a pre-test OR Use different pre- and post-tests
Instrumentation: Nature of measuring instrument has changed between pre- and post-test
To prevent: Do not use a pre-test OR Use the same for pre- and post-test
Selection Bias: Treatment and control groups are different on an important extraneous variable
To prevent: Random assignment OR Use a pre-test
NOTE: Many of these preventions contradict. You must think about which threats are most relevant to your study and then develop a strategy for overcoming the most important threats.
Dr. K. A. Korb
University of Jos

16. External Validity
External Validity: Establishing the group of people to which the research findings can be generalized
Dr. K. A. Korb
University of Jos

17. Population Sample Research Example Experimental Group Control Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Sample
Control
Group
Reading
Achievement
Reading
Achievement
Dr. K. A. Korb
University of Jos

18. Types of External Validity
Population Validity: Generalizing the results from experimental sample to a defined population
To establish: Give a thorough explanation of the constitution of your sample and how your sample was selected.
Ecological Validity: Generalizing the results of a study from the artificial conditions created by researcher to real-life conditions
To establish:
Give an adequate description of treatment
Give an adequate description of the measure for the dependent variable
Threats to Ecological Validity:
Novelty: The treatment was different than typical treatment
Experimenter Effect: The particular person delivering the treatment affects results
Dr. K. A. Korb
University of Jos

19. Reliability
Reliability: Demonstrating that the operations of the study can be repeated with the same results
Conduct research as if someone were always looking over your shoulder
Report the procedure of your study in explicit detail
Dr. K. A. Korb
University of Jos

20. Other Issues in Valid Research Design
Experimenter Bias: Researcher’s expectations about the outcome of experiment influence participants’ response
To prevent: Use research assistants who are blind to the study (aka do not know the purpose of the study)
Treatment Fidelity: Extent to which treatment conditions are implemented according to the researcher’s specifications
To support:
Carefully train research assistants
Periodically check up on research assistants unexpectedly to determine if they are following guidelines
Strong Experimental Treatments: Developing a treatment that will make a robust effect on the dependent variable
Develop a thorough understand of your dependent variable and how it can be influenced.
Spend much thoughtful time developing your treatment.
Dr. K. A. Korb
University of Jos