Research Methods

Research In your “Testing Claims” project you will be investigating research studies that back up a claim. You use the C.R.A.P. test to evaluate sources, but what do you do if you find a research study from what appears to be a reliable source, but may have other problems. How do you analyze the problems that may exist in a research study?

Strength of Evidence Weak Strong Sample size (1 pt) Small sample size Large sample size Sample size (1 pt) Non-random sample Random sample Sample selection (1 pt) Confounds (1 pt) Significant Confounds Insignificant Confounds Researcher Bias (1 pt) Researcher Bias Present Researcher Bias Absent

Example: Based on results from a recent experiment, Dr. Lite claims that she has found a way to prevent 95% of hair loss in men and women (aged 25-80), with no adverse side effects. You are interested in investing in Dr. Lite’s product, but first you need to determine the quality of her study. You would like to make sure that Dr. Lite’s research findings would replicate if the study was repeated with a different sample. You know that research findings from studies with rigorous methods are more likely to replicate, so you decide to ask Dr. Lite a few questions.

Clicker Question 1 Which of the following questions about the study’s research methodology would you ask Dr. Lite first? Did the participants think the treatment was easy to use? How did you select your sample? Did participants report hair regrowth? Was there a significant difference between treatment and control groups?

Clicker Question 2 After learning that Dr. Lite randomly selected participants (50% female, aged 25-80) from the population of interest, what would you ask Dr. Lite next? How many participants were in your sample? Was this a convenience sample? Were subjects paid to participate? Did family members notice a change in the participants’ hair following the treatments?

Clicker Question 3 Dr. Lite informs you that there were approximately 1000 participants in the study. Half of the participants were in the experimental group (used the treatment) and half of the participants were in the control group (used a placebo treatment). What would you want to ask Dr. Lite next? Was there a significant difference in hair loss between the treatment and control group? How many weeks did the treatment effects last? How were subjects assigned to conditions? Were there any confounds in this study?

Clicker Question 4 You learn that the 1000 participants were randomly assigned to the experimental and control groups. Were the investigators and participants blind to the study conditions? How much does the treatment cost? Does the treatment effectiveness change depending on initial hair thickness? Who funded this study?

Clicker Question 5 Dr. Lite informs you that the researchers and participants were blind to the conditions of the study (i.e., double blind). What might you ask next? How many researchers helped with this study? How many weeks did the treatment effects last? Was there a significant difference in hair loss between the treatment and control group? Were the procedures for the experimental and control group the same, with the exception of the actual treatment?

Clicker Question 6 After learning that Dr. Lite used strict experimental control, and that the conditions for each group were exactly the same, how would you rate the strength of the evidence provided by this study’s findings? very weak moderately weak moderately strong very strong Weak Strong Small sample size Large sample size Non-random sample Random sample Significant Confounds Insignificant Confounds Researcher Bias Present Researcher Bias Absent

Your Project: Use one of the references from your project with the scale below to evaluate the strength of the evidence reported in the study. For each of the 4 categories, explain why you thought the study earned it’s +1 point rating. Then provide an overall rating out of 4. Weak Strong Small sample size Large sample size Non-random sample Random sample Significant Confounds Insignificant Confounds Researcher Bias Present Researcher Bias Absent