Reading Science Critically Debi A. LaPlante, PhD Associate Director, Division on Addictions

First Sources Reading primary sources can be daunting Reading primary sources can be daunting –Complexity of information –Researchers are marketing their ideas and findings –Time Benefits Benefits –Current findings –Promotes and enables replication –Data (often)

What is the Purpose of Scientific Papers? Concisely report information, ideas, and innovation Concisely report information, ideas, and innovation Build the common knowledge-base Build the common knowledge-base Contribute to scientific debate Contribute to scientific debate Resume building Resume building

Why is important to read science critically? Peer-review is state of the art, but imperfect Peer-review is state of the art, but imperfect –Author bias –Unintentional errors –Conflicts of interest –Author self-marketing

More challenges to understanding and evaluating scientific literature Writing by scientists, not writers Writing by scientists, not writers Marketing: Trojan Ns Marketing: Trojan Ns Marketing: Assertive Sentence Titles Marketing: Assertive Sentence Titles Statistical versus Clinical significance Statistical versus Clinical significance Publication bias Publication bias –Tough to publish negative results

Finding Articles Citation lists of published papers Citation lists of published papers Select journals’ table of contents Select journals’ table of contents Specialized search engines (e.g., Medline; PsycInfo) Specialized search engines (e.g., Medline; PsycInfo) Web searches (e.g., Google Scholar) Web searches (e.g., Google Scholar) Personal referrals Personal referrals Citation indexes (e.g., Social Science Citation Index) Citation indexes (e.g., Social Science Citation Index)

Components of Scientific Papers Abstract Abstract Introduction Introduction –Hypotheses or research questions Methods Methods –Participants –Materials –Protocol Results Results Discussion Discussion –Interpretation of results –Advances –Limitations Conclusion Conclusion

How to get through a paper Strategy depends on expertise Strategy depends on expertise General approach: General approach: –Don’t read straight through –Read title and abstract –Skim Intro –Read results –Track back to Methods –Read Discussion

Resources Literature summary services Literature summary services – – Greenhalgh (1997) /305 Greenhalgh (1997) / / /305 Zaccai (2004) ull.pdf Zaccai (2004) ull.pdf ull.pdf ull.pdf

Is the study original? Does the research advance what we know? Does the research advance what we know? –Bigger, longer, more substantial? –More rigorous? –New population? –Will it inform or change clinical practice? Greenhalgh (1997)

Whom is the study about? What was the recruitment method? What was the recruitment method? –Representative and generalizable?  Refusal rate? Homogeneity? Random? What are the inclusion criteria? What are the inclusion criteria? –Disorder severity What are the exclusion criteria? What are the exclusion criteria? –Co-existing illness, other medication, English, literate How “true to life” is the study setting? How “true to life” is the study setting? Greenhalgh (1997)

Is the design sensible? What was done? What was done? –Appropriate comparison groups? What was the measured outcome? What was the measured outcome? Is there a sufficient description of the design? Is there a sufficient description of the design? Greenhalgh (1997)

Ambiguous Research Methods Greenhalgh (1997) Author Statement Actual Method Problem “We measured how often GPs ask patients whether they smoke.” “We looked in patients’ medical records and counted how many had had their smoking status recorded.” Assumes medical records are 100% accurate. “We measured how doctors treat low back pain.” “We measured what doctors say they do when faced with a patient with low back pain.” Assumes that self-report reflects behavior. “We compared a nicotine- replacement patch with placebo.” “Subjects in the intervention group were asked to apply a patch containing 15 mg nicotine twice daily; those in the control group received identical-looking patches.” Fails to provide adequate methodological details.

Is systematic bias avoided or minimized? Designs Designs –Randomized trials –Non-randomized trials –Cohort studies –Case studies Methods Methods –Blind assignment and assessment –Validated measurement tools –Control confounding (e.g., baseline group differences) Greenhalgh (1997)

Autumn Season Student arrival to campus Falling Leaves Confounder of the Season-Falling Leaves relationship

Are the results credible? Is there a sufficient sample size? Is there a sufficient sample size? Are the results clinical significant? Are the results clinical significant? How long is follow up? How long is follow up? –Is the follow-up appropriate to the outcome? (e.g., post-operative pain versus pediatric growth patterns) –What is retention rate? Greenhalgh (1997)

Take Away Messages First Source publications provide important benefits to science and practice First Source publications provide important benefits to science and practice Unintentional and intentional errors occur Unintentional and intentional errors occur Readers should read critically and not merely take such publications at face value Readers should read critically and not merely take such publications at face value