1. Why do we read research articles?
To increase our knowledge in a particular area of interest
To search for evidence for our practice
To review literature in preparation for a research
Why do we read research articles?

2. Should we believe everything we read?

3. Critical Appraisal OPST 199 Methods of Research
Kristofferson G. Mendoza, PTRP
Department of Physical Therapy
College of Allied Medical Professions
University of the Philippines Manila
27 May 2009

4. Learning Objectives
At the end of the session, the learner should be able to:
Identify the level of evidence presented in a research study on the basis of the research design used
Discuss theoretical and methodological aspects of a research study in relation to internal and external validity
Rate the usefulness of evidence presented in a research study using key criteria

5. Critical Appraisal
the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.

6. OPST 199 Methods of Research Critical Appraisal
Levels of Evidence
OPST 199 Methods of Research
Critical Appraisal

7. Levels of Evidence
Various study designs have different levels of rigour

8. Classic Levels of Evidence
By Sackett et al.
For Studies on intervention, prevention, etiology and harm
RCT the most rigorous study design
Modification of the classic level of evidence later on included systematic reviews bec of the proliferation of RCTs
Randomization of large numbers of patients – by chance, tend to evenly distribute prognostic variables between comparison groups – removing bias

9. Level 1 Level 2 Level 3 Level 4 Level 5
HIGH
Level 1
(minus) Randomization
Level 2
(minus) Prospective Data Collection
Level 3
(minus) Comparison Groups
Level 4
(minus) Clinical data
Level 5
LOW

10. Levels of Evidence For studies on Prognosis
optimal individual study – individual inception cohort study with greater than 80% ff up
Where the clinical decision rule has been validated in a single population
Clinical decision rule are algorithms or scoring systems that lead to prognosis estimation or diagnostic category

11. Levels of Evidence For studies on Diagnosis
optimal individual study – cohort study with good reference standards or clinical decision rule tested within one clinical center

12. Levels of Evidence
For studies on Differential diagnosis / Symptom prevalence
Cohort study with good follow up

13. Levels of Evidence
Despite differences in the optimal study design, certain consistencies are evident across different types of questions
A systematic review of high-quality studies always provides the highest level of rigor
An individual study using the optimal design for that type of clinical question is considered level 1
Prospective data collection indicates higher study quality than retrospective data collection
Expert opinion, bench research, conceptual framework/theories/first principles are always considered the lowest (level 5) evidence

14. Other Levels of evidence Classification System
Greenhalgh (1997)
Systematic Reviews and meta-analysis
RCTs with definitive results
RCTs with non-definitive results
Cohort studies
Case-control studies
Cross sectional surveys
Case reports

15. Levels of evidence For quantitative studies
Primary Research
Experimental
Observational
For qualitative studies? No consensus about the relative rigour of different methods

16. Research design issues
OPST 199 Methods of Research
Critical Appraisal

17. Research Design Issues in Quantitative Research
Sampling of Subjects
Objective: A sample that represents the target population
Optimal Design: A random sample selected from the population
Threats to Validity: Differential sampling between groups will affect internal validity; convenience sampling may affect external validity

18. Research Design Issues in Quantitative Research
Sample Size
Objective: The optimal number of subjects is large enough to detect important treatment effects but small enough to conduct the study in a timely, efficient manner
Optimal Design: Correct number of subjects is determined by an accurate sample size calculation
Threats to Validity: Small sample size

19. Research Design Issues in Quantitative Research
Allocation of treatment
Objective: Unbiased allocation
Optimal Design: Random allocation
Threats to Validity: Nonrandom allocation (observational)

20. Research Design Issues in Quantitative Research
Blinding (to treatment allocation and/or outcomes
Objective: To minimize sources of bias introduced by study personnel or participants
Optimal Design: Ideally everyone involved in the study would be blinded to the full extent possible
Threats to Validity: Blinding in rehab is difficult; may result in bias from differential diagnosis, outcome assessments, attention and follow-up procedures

21. Research Design Issues in Quantitative Research
Outcome ascertainment
Objective: To accurately determine pretreatment status; To reflect all important changes in outcome post-treatment
Optimal Design: Outcomes measured at all likely relevant time points; Outcomes measured using reliable, valid, and responsive measures
Threats to Validity: important effects will be missed if relevant time points are not assessed or the outcome measures are not sensitive; outcome measures that reflect only certain domains may be biased towards specific treatments; poor reliability/validity may invalidate the conclusion

22. Research Design Issues in Quantitative Research
Follow-up
Objective: To accurately portray the treatment effects obtained by all the participants
Optimal Design: 100% follow-up
Threats to Validity: differential loss to follow-up can introduce bias in estimate of effects

23. Research Design Issues in Quantitative Research
Statistical analysis
Objective: To provide accurate estimates of the size and significance of the observed effects
Optimal Design: Accurate and appropriate analysis of all data
Threats to Validity: Inappropriate analysis may lead to faulty conclusion

24. OPST 199 Methods of Research Critical Appraisal
Critical review form
Law, M., Stewart, D., Pollock, N., Letts, L., Bosch, J., & Westmorland, M.,
1998. McMaster University
OPST 199 Methods of Research
Critical Appraisal

25. Was the purpose stated clearly?
Comments
STUDY PURPOSE:
Was the purpose stated clearly?
Yes No
Outline the purpose of the study. How does the study apply to OT/PT/SP and/or your research question?
LITERATURE:
Was relevant background literature reviewed?
Describe the justification of the need for the study.
DESIGN:
RCT
Cohort
Single case design
Before and after
Case-control
Cross-sectional
Case study
Describe the study design. Was the design appropriate for the study question (e.g., for knowledge level about this issue, outcomes, ethical issues, etc.)?
Specify any biases that may have been operating and the direction of their influence on the results.

26. Appropriateness of Study Design
Knowledge of topic/issue
If little is known, more exploratory (case study, cross- sectional design)
as level of knowledge increases, study design must be more rigorous (RCT most rigorous)
Outcomes
If outcomes are easily quantifiable and well-developed standardized assessment tools are available, design should be rigorous
If outcomes are not fully understood yet (e.g., quality of life), use design that explore different factors (case control)

27. Appropriateness of Study Design
Ethical Issues:
If there is no ethical issue re: withholding treatment, design should have control groups
Study purpose/questions
Effectiveness of treatment: RCTs, before-after design, single-case studies
Learn more about an issue, pilot study to determine treatment: case control an cross sectional

28. Biases Sample/Selection Bias
Volunteer bias – usually favors treatment group
Seasonal bias – could work either way
Attention bias – favors treatment group (extra attention)

29. Biases Measurement/Detection Bias Number of outcome measures used
Only one outcome measure – could favor either groups
Too many outcome measures for sample size – favors control group
Lack of masked or independent evaluation – treatment group is usually favored
Recall or memory bias – favors treatment group

30. Biases Intervention/Performance Bias
Contamination – favors control group
Co-intervention – can influence either groups
Timing of intervention
Site of treatment
Different therapists

31. Was the sample described in detail?
Comments
SAMPLE: N=
Was the sample described in detail?
Yes No
Was the sample size justified?
Sampling (who; characteristics; how many; how was sampling done?) If more than one group, was there similarity between groups?
Describe ethics procedure. Was informed consent obtained?
OUTCOMES:
Were the outcome measures reliable?
Not addressed
Were the outcome measures valid?
Specify the frequency of outcome measurement (i.e. pre, post, follow-up).
Outcome areas (e.g., self care, productivity, leisure)
List measures used

32. Intervention was described in detail?
Comments
INTERVENTION:
Intervention was described in detail?
Yes No
Not addressed
Contamination was avoided?
N/A
Cointervention was avoided?
Provide a short description of the intervention (focus, who delivered it, how often, setting). Could the intervention be replicated in OT/PT/SP practice?

33. Results were reported in terms of statistical significance?
Comments
RESULTS:
Results were reported in terms of statistical significance?
Yes No
N/A
Not addressed
Were the analysis method(s) appropriate?
What were the results? Were they statistically significant (i.e., p<0.05)? If not statistically significant, was study big enough to show an important difference if it should occur? If there were multiple outcomes, was that taken into account for the statistical analysis?
Drop-outs were reported?
Did any participant drop out from the study? Why? (Were reasons give and were drop outs handled appropriately?)

34. CONCLUSIONS AND CLINICAL IMPLICATIONS:
Comments
CONCLUSIONS AND CLINICAL IMPLICATIONS:
Conclusions were appropriate given study methods and results?
Yes No
What did the study conclude? What are the implications of these results for OT/PT/SP practice? What were the main limitations or biases in the study?

35. OPST 199 Methods of Research Critical Appraisal
Thank you
OPST 199 Methods of Research
Critical Appraisal