1. Introduction to Critical Appraisal : Quantitative Research
South East London
Outreach Librarians
January 2008

2. Learning objectives
Understand the principles of critical appraisal and its role in evidence based practice
Be able to appraise quantitative research and judge its validity
Be able to assess the relevance of published research to your own work

3. Daily Mail exercise
Would you treat a patient based on this article? Why?
Validity
Reliability
Transferable to practice

4. What is evidence based practice?
Evidence-based practice is the integration of
individual clinical expertise
with the
best available external clinical evidence from systematic research
and
patient’s values and expectations

5. The evidence-based practice process.
Decision or question arising from a patient’s care.
Formulate a focused question.
Search for the best evidence.
Appraise the evidence.
Apply the evidence.
This is the step-by-step process.
We will be concentrating on step 4 today.
Critical appraisal is very much part of patient care.

6. Why does evidence from research fail to get into practice?
75% cannot understand the statistics
70% cannot critically appraise a research paper
Using research for Practice: a UK experience of the barriers scale.
Dunn, V. et al.

7. What is critical appraisal?
Weighing up evidence to see how useful it is in decision making
Balanced assessment of benefits and strengths of research against its flaws and weaknesses
Assess research process and results
Skill that needs to be practiced by all health professionals as part of their work

8. What critical appraisal is NOT
Negative dismissal of any piece of research
Assessment of results alone
Based entirely on statistical analysis
Only to be undertaken by researchers/ statisticians

9. Why do we need to critically appraise?
“It usually comes as a surprise to students to learn that some (the purists would say 99% of) published articles belong in the bin and should not be used to inform practice” (Greenhalgh 2001)
Find that 1% - save time and avoid information overload

10. How do I appraise? Mostly common sense.
You don’t have to be a statistical expert!
Checklists help you focus on the most important aspects of the article.
Different checklists for different types of research.
Will help you decide if research is valid and relevant.
There are different checklists for different types of research – available from CASP website.
These will help you focus on the main points to consider.
Today we are looking at a systematic review so I will focus on the points to consider for this type of research.
Focus on the Methods and the Results in a paper. Abstract can be useful but be aware that is may not give a full picture.

11. Research methods Quantitative Uses numbers to describe and analyse
Useful for finding precise answers to defined questions
Qualitative
Uses words to describe and analyse
Useful for finding detailed information about people’s perceptions and attitudes

12. Levels of quantitative evidence
Levels of quantitative evidence. (In order of decreasing scientific validity.)
Systematic reviews.
Randomized controlled trials.
Prospective studies (cohort studies).
Retrospective studies (case control).
Case series and reports
Opinions of respected authorities.
Before critically appraising an individual paper you need to appraise and select the best type of research to read.
Does anyone know?
If you are looking for evidence to support your practice, you would first try to find a systematic review, followed by Randomised CTs.
If no SR or RCT has been carried out follow the trail to the next best form of external evidence and work from there.

13. Systematic Reviews. Thorough search of literature carried out.
All RCTs (or other studies) on a similar subject synthesised and summarised.
Meta-analysis to combine statistical findings of similar studies.
What is a systematic review?
What is a meta-analysis?
Not the same as a review article or a literature review, necessarily. Has to be systematic.

14. Randomised Controlled Trials (RCTs)
Normal treatment/placebo versus new treatment.
Participants are randomised.
If possible should be double-blinded.
Intention to treat analysis
What are the main characteristics of an RCT?

15. Cohort studies prospective groups (cohorts) exposure to a risk factor
followed over a period of time
compare rates of development of an outcome of interest
Confounding factors and bias

16. Case control studies Retrospective
Subjects confirmed with a disease (cases) are compared with non-diseased subjects (controls) in relation to possible past exposure to a risk factor.
Confounding factors and bias

17. Appraising original research
Are the results valid?
Is the research question focused?
Was the method appropriate?
How was it conducted, e.g. randomisation, blinding, recruitment and follow up?
What are the results?
How was data collected and analysed?
Are they significant?
Will the results help my work with patients?

18. Appraising systematic reviews.
In addition to the above:
Was a thorough literature search carried out ?
Publication bias - papers with more ‘interesting’ results are more likely to be:
Submitted for publication
Accepted for publication
Published in a major journal
Published in the English language
These are the main points to consider.
Bear in mind that you are not trying to pull the paper to pieces. You are just trying to consider “is the article good enough”. It can’t be perfect.

19. Reviews in general medical journals
50 reviews in 4 major journals
No statement of methods
Summary inappropriate
“Current systematic reviews do not routinely use scientific methods to identify, assess and synthesise information” (Mulrow, 1987)

20. Is the research question focused?
Patient (e.g. child)
Intervention (e.g. MMR vaccine)
Comparison (e.g. single vaccines)
Outcome (e.g. autism)
The main components of a focused question. If the research question is not focused, the article will not be able to answer your clinical question so there is no point in reading it.
Don’t have to have a Comparison.

21. Are results significant?
How was data collected?
Which statistical analyses were used?
How precise are the results?
How are the results presented?
These are some of the aspects that the checklist will help you to look at.
You don’t have to be a statistical expert!
Be wary of complicated statistical analyses (data-torturing). Repeated calculations could eventually prove that black is white. Could be used to get the results they wanted to find. The basic analyses should always have been carried out and presented, e.g.
Confidence Intervals and/or p values
Meta-analyses (possibly)
Odds ratios

22. Intention to treat analyses
Analysing people, at the end of the trial, in the groups to which they were randomised, even if they did not receive the intended intervention

23. Statistical analyses Odds ratios, absolute and relative risks/benefits
The likelihood of something happening vs the likelihood of something not happening
Numbers needed to treat (NNT)
The number of people you would need to treat to see one additional occurrence of a specific beneficial outcome

24. Odds Ratio Diagrams. (Blobbograms or Forest Plots.)

25. Odds Ratio Diagrams
Line of no effect – no difference between treatment and control group
Result (blob) to the Left of the line of no effect = Less of the outcome in the treatment group.
Result to the Right of the line = More of the outcome.
BUT - Is the outcome good or bad?
Results of SRs are often presented as odds ratio diagrams.
Pooled statistical results or meta-analyses.
It is useful to understand how to interpret these.
Does anybody know?

26. Cardiac deaths – Less = good
What does this result tell us?
Outcome is unfavourable (death).
Left = less of the outcome with treatment, so good!
Diamond at the bottom = overall results.

27. Smoking cessation – More = good
So what does this result tell us?
Outcome is favourable (smoking cessation).
To the right = more of the outcome with treatment, so good!
You don’t always get the ‘Favours treatment’ bit at the bottom.

28. Confidence Intervals.
Longer confidence interval = less confident of results – wider range.
Shorter confidence interval = more confident – narrower range.
Crosses line of no effect/no significance = Inconclusive results.
Studies are weighted according to their size.
Does anybody know how the lines and blobs work?

29. Confidence intervals
Big blob and small line means it is a big study and we can be quite confident of the results.
Small blob and long line means small study and less confidence in results.
If confidence interval crosses line of no effect, we cannot be sure whether the favour control or favour treatment.
Sometimes individual studies cross line but overall results are conclusive.
Here results are inconclusive. We don’t know.

30. P Values.
P stands for probability - how likely is the result to have occurred by chance?
P value of less than 0.05 means likelihood of results being due to chance is less than 1 in 20 = “statistically significant”.
P values and confidence intervals should be consistent
Another thing to look for is the p value.
Does anybody know what this is?
Look for a p value of less than This means it is not likely that the results came about by chance.

31. Number Needed to Treat
The number of people you would need to treat to see one additional occurrence of a specific beneficial outcome.
The number of patients that need to be treated to prevent one bad outcome.
The NNT can be calculated by finding the Absolute Risk Reduction (ARR)

32. Events or outcomes are used for reporting results
Events or outcomes are used for reporting results. The event rate is the proportion of patients in a group in whom the event is observed
Outcome event
Total
Yes No
Experimental group a b
a + b
Control group c d
c + d
a + c
b + d
a + b + c + d

33. CER and EER
Control Event Rate (CER) is the proportion of patients in the control group in whom an event is observed.
CER = c/(c+d)
Experimental Event Rate (EER) is the proportion of patients in the experimental in whom an event is observed.
EER = a/(a+b)

34. AAR & NNT
Absolute Risk Reduction is the difference between the Control Event Rate (CER) and the Experimental Event Rate (EER).
ARR = CER – EER
Number needed to treat (NNT)
NNT = 1/ARR

35. Outcome event Total Yes No 3 10 5 8 12 20
Outcome event
Total
Yes No
Experimental
group 3 7 10
Control group 5 8 12 20

36. Answers What is the event ? Lack of concentration and sleeping
What is the control event rate (CER)? 
5/10 = 0.50
What is the experimental event rate (EER)?
3/10 = 0.30
Calculate the absolute risk reduction (ARR) 
0.50 – 0.30 = 0.20 
What is the number needed to treat (NNT)? 
1.00/0.20 = 5

37. Are results relevant? Can I apply these results to my own practice?
Is my local setting significantly different?
Are these findings applicable to my patients?
Are findings specific/detailed enough to be applied?
Were all outcomes considered?
Very important.
It is no good spending ages looking at a paper if you can’t apply it to your own patients.
SRs are good in that respect as they usually include a wide range of patient groups from different RCTs.
Are any differences between this and my setting important or not?

38. The good news!
Some resources have already been critically appraised for you.
An increasing number of guidelines and summaries of appraised evidence are available on the internet.
The good thing about Cochrane is that if you find one of their systematic reviews then you don’t need to worry so much about the reliability or quality of the research. They will evaluate it for you and you can be more confident in trusting the opinions of these experts.
There are also other evidence-based research summaries available – see handout for further information.

39. Summary.
Search for resources that have already been appraised first, e.g. Guidelines, Cochrane systematic reviews.
Search down through levels of evidence, e.g. systematic reviews, RCTs.
Use checklists to appraise research.
How can these results be put into practice?
Try to use ready-appraised resources first if you can, saves time and effort.
Use hierarchy of evidence to find suitable papers.
Always be thinking about putting the evidence into practice (if the research is good quality) – how should policy or practice change?