PhD Research Seminar Series: Valid Research Designs Dr. K. A. Korb University of Jos

Outline Threats to valid Research Design Construct Validity Internal Validity External Validity Reliability Dr. K. A. Korb University of Jos

Reliability: Consistency of results Reliable Unreliable Dr. K. A. Korb University of Jos

Validity: Measuring what is supposed to be measured Invalid Dr. K. A. Korb University of Jos

Research Example The effect of regular leisure reading on reading achievement in primary school. Students in a primary school will be randomly assigned to either a treatment or a control group. The treatment group will spend 10 minutes everyday reading a book with an adult. In an attempt to keep the treatment and control groups as similar as possible, the control group will also spend 10 minutes with an adult everyday, but will instead do math problems. Dr. K. A. Korb University of Jos

Research Example IV: Regular Leisure Reading DV: Reading Achievement Experimental Group Reading Achievement Read Book Every Day with Adult Reading Achievement Control Group Reading Achievement Do Maths Every Day with Adult Reading Achievement Dr. K. A. Korb University of Jos

Construct Validity Construct Validity: Establishing valid operational measures for the concepts being studied This relates to: Independent Variable Dependent Variable Review: Construct Definition: General explanation of the construct by relating it to other constructs Operational Definition: Statement of specifically how the construct will be measured or implemented in your study Dr. K. A. Korb University of Jos

Review: Construct vs. Operational Definitions WAEC Scores Final Course Exam Academic Achievement NECO Scores State Exams Self-Report of Parents’ Yearly Income Economic Status Free/Reduced Lunch Eligibility Dr. K. A. Korb University of Jos

Research Example Construct validity focuses on: IV: Regular Reading DV: Reading Achievement Experimental Group Reading Achievement Read a Book Every Day Reading Achievement Reading Achievement Control Group Reading Achievement Dr. K. A. Korb University of Jos

Construct Validity When you develop your research project, you start with a theory. For example, regular leisure reading will lead to increased reading achievement scores. Then you operationalize your theory into your research study. You translate “regular leisure reading” into how it will be implemented in your study – reading a book with an adult everyday for 10 minutes. You translate “reading achievement scores” into how it will be measured in your study – classroom exam scores. When you finish collecting data, you hope to be able to move back to theory. You want to be able to say that not only did reading a book for 10 minutes everyday with an adult lead to higher classroom exam scores on reading achievement, but also that regular leisure reading will lead to higher reading achievement scores in general. However, in order to do that, you must have construct valid measures of your independent and dependent variables. Dr. K. A. Korb University of Jos

Research Example Theory: Research Study: Theory: IV: Regular Reading DV: Reading Achievement Theory: Read book every day with an adult for 10 min. Scores on reading section of classroom exams Research Study: IV: Regular Reading DV: Reading Achievement Theory: Dr. K. A. Korb University of Jos

Construct Validity Enables you to generalize from the treatment and measures in your study to the general constructs of your treatment and measures Threats to Construct Validity Inadequate explanation of constructs in your paper Flawed matching of operationalization to constructs The measures do not align with the constructs they were designed to measure. Dr. K. A. Korb University of Jos

Internal Validity Internal validity: Extent to which variables other than the treatment provide plausible explanations to the experimental results Only relevant for experimental designs Primary consideration for interventions Extraneous variable: Any variable other than treatment variable that if not controlled can affect the experimental outcome Hold constant or eliminate all extraneous variables that might affect the posttest Goal of Research Design: Create set of conditions so any observed changes in your dependent variable can be attributed to experimental treatment instead of extraneous variables Do variables other than the treatment provide plausible explanations to the experimental results? Dr. K. A. Korb University of Jos

Research Example Maturation IV: Regular Reading DV: Reading Achievement Interest Experimental Group Reading Achievement Read a Book Every Day Reading Achievement Reading Achievement Control Group Reading Achievement Reading Ability Adult Attention Dr. K. A. Korb University of Jos

Threats to Internal Validity History: If treatment lasts over time, other events may influence dependent variable To prevent: Keep the control group equivalent in all aspects but treatment Maturation: Physical or psychological changes may influence the dependent variable To prevent: Use a control group Testing: Giving a pretest may increase performance on the post-test To prevent: Don’t use a pre-test OR Use different pre- and post-tests Instrumentation: Nature of measuring instrument has changed between pre- and post-test To prevent: Do not use a pre-test OR Use the same for pre- and post-test Selection Bias: Treatment and control groups are different on an important extraneous variable To prevent: Random assignment OR Use a pre-test NOTE: Many of these preventions contradict. You must think about which threats are most relevant to your study and then develop a strategy for overcoming the most important threats. Dr. K. A. Korb University of Jos

External Validity External Validity: Establishing the group of people to which the research findings can be generalized Dr. K. A. Korb University of Jos

Population Sample Research Example Experimental Group Control Group Reading Achievement Read a Book Every Day Reading Achievement Sample Control Group Reading Achievement Reading Achievement Dr. K. A. Korb University of Jos

Types of External Validity Population Validity: Generalizing the results from experimental sample to a defined population To establish: Give a thorough explanation of the constitution of your sample and how your sample was selected. Ecological Validity: Generalizing the results of a study from the artificial conditions created by researcher to real-life conditions To establish: Give an adequate description of treatment Give an adequate description of the measure for the dependent variable Threats to Ecological Validity: Novelty: The treatment was different than typical treatment Experimenter Effect: The particular person delivering the treatment affects results Dr. K. A. Korb University of Jos

Reliability Reliability: Demonstrating that the operations of the study can be repeated with the same results Conduct research as if someone were always looking over your shoulder Report the procedure of your study in explicit detail Dr. K. A. Korb University of Jos

Other Issues in Valid Research Design Experimenter Bias: Researcher’s expectations about the outcome of experiment influence participants’ response To prevent: Use research assistants who are blind to the study (aka do not know the purpose of the study) Treatment Fidelity: Extent to which treatment conditions are implemented according to the researcher’s specifications To support: Carefully train research assistants Periodically check up on research assistants unexpectedly to determine if they are following guidelines Strong Experimental Treatments: Developing a treatment that will make a robust effect on the dependent variable Develop a thorough understand of your dependent variable and how it can be influenced. Spend much thoughtful time developing your treatment. Dr. K. A. Korb University of Jos