1. Assessing the Evidence:
11/9/2018
Assessing the Evidence:
Search, Evaluate, and Outline
March 26, 2015
CONFIDENTIAL

2. Topics How to Approach the Literature Search
Evaluating the Quality of the Identified Literature
Framing the Story and Next Steps

3. How to Approach the Literature Search

4. How to Approach the Literature Search
Specify your objectives
Formulate the key research questions
Build the search strategy
Conduct the search

5. Specify your Objectives
11/9/2018
Specify your Objectives
Why are you reviewing the literature?
Make a decision: Should we or shouldn’t we use repeat anti-TNF-alpha therapy?
Comparative effectiveness: How does a new ACE inhibitor compare with an existing ACE inhibitor?
Assess knowledge base: What is hereditary angioedema and how is it treated in the US?
Prepare a summary: Drafting an introduction section in a manuscript or an AMCP dossier
The core question is WHY are you reviewing the literature? What is the intent? This will inform your approach to the search strategy and even to the literature evaluation.
CONFIDENTIAL

6. Determine Type of Search Based on Objectives
Systematic
Narrative
Targeted
Make a decision
Literature review intended for publication
To support the development of a core value dossier or specific dossier requirements (eg, NICE submission dossier, AMNOG dossier)
To conduct a meta-analysis or indirect treatment comparison
Landscape assessment of new therapeutic area / asset
Answer a specific research question
Inform internal planning/research questions (broad)
To support the development of an AMCP dossier
Economic model conceptualization
Answer a specific research question
Inform internal planning/research question
Summary of literature for publications
Identify model inputs
$$$ $$ $

7. Formulate Your Key Research Questions
My need is basic, so why do I have to formulate the research questions?

8. Formulate Your Key Research Questions
My need is basic, so why do I have to formulate the research questions?

9. Using PICO (PICOS) to Formulate the Questions
Population
Interventions
Comparators
Outcomes
Study design
“The well-formed question makes it relatively straightforward to elicit and combine the appropriate terms…in the query language”.

10. Using PICO
Centre for Evidence-based Medicine

11. Build the Search Strategy
Purpose
11/9/201811/9/201811/9/2018

12. Build the Search Strategy
Purpose
As a result…
You can repeat what you’ve done
science needs to be replicable
Someone else can do the same thing and get the same results
scalable, replicable
You can maintain quality control
You can modify and improve.

13. What Does a Literature Search Strategy Look Like?
Elements of a search strategy
Inclusion criteria
Study question
Language
Time horizon
Exclusion criteria
Search strategy
Data extraction
Study selection
It depends…
Databases / Sources
Pick a few critical elements to focus on

14. Conduct the Search

15. Evaluating the Quality of the Evidence

16. Approaches to Grading the Evidence
11/9/2018
Approaches to Grading the Evidence
Multiple approaches exist and vary depending on the study design
The following represents a sample of commonly used checklists
CHEERS
GRACE
CRD
NICE
Sponsor
ISPOR
NPC/Quintiles
Centre for Reviews and Dissemination
Study Type Evaluated
Health economic evaluations
Observational cohort studies
Stated Aims
To provide recommendations, in the form of a checklist, to optimize reporting of health economic evaluations.
To assess observational studies
of comparative effectiveness in terms of their quality and usefulness for
decision-making
Evaluate the elements of the economic evaluation that can have an impact on the validity of the overall results
To determine whether an economic evaluation provides evidence that
is useful to inform the decision-making of the GDG
Checklist Available?
Yes
# of Items 24 11 36 22
CRD: Referral to compare the cost analyses with published best practice guidance documents. Elements to consider include:
Methods of deriving the effectiveness data
Measurement of resource data
Valuation of resource data
Measurement and valuation of health benefits (utilities)
Method of synthesizing the costs and effects
Analysis of uncertainty
Generalizability of the results
CONFIDENTIAL

17. Approaches to Grading the Evidence
11/9/2018
Approaches to Grading the Evidence
Cont’d
Assessment of quality of all study types
Cochrane Collaboration Risk of Bias
Assessing reporting quality or reporting of a RCT
Jadad 5-point Scale
CONSORT statement
Assessing quality of observational studies
Newcastle-Ottawa Scale 16-item checklist
STROBE statement 22-tiem checklist
Assessing quality or reporting of systematic literature reviews
PRISMA Statement 27-item checklist
MOOSE 35-item checklist
AMSTAR 11-item questionnaire
NICE Methodology checklist
Assessing the body of evidence
GRADE
SIGN
Source: McMaster University (
For SLRs:
AMSTAR assesses methodological quality
Other quality tools are Scottish Intercollegiate Guideline Network (SIGN) checklist, CEBM critical appraisal checklist
Others assess reporting: PRISMA, MOOSE, NICE
And many many others!
CONFIDENTIAL

18. The Problem with Choice

19. Practical Evaluation of the Quality of the Evidence
What is the Relevance of Grading to the Research Question?
Consider what is your purpose in grading the literature:
Grading a body of evidence for purposes of synthesis (quantitative)
Grading a select number of studies for inclusion in a dossier or other use (qualitative)
Consider whether you are grading the quality of the evidence or the reporting of the evidence
“The strength of evidence grade summarizes the reviewers’ confidence in the findings based on either approach to evidence synthesis.” – AHRQ 2013

20. Practical Evaluation of the Quality of the Evidence
11/9/2018
Practical Evaluation of the Quality of the Evidence
Source:
Shamliyan TA, Kane RL, Taylor FR. Migraine in Adults: Preventive Pharmacologic Treatments. Comparative Effectiveness Review No (Prepared by the University of Minnesota Evidence-based Practice Center under Contract No I) AHRQ Publication No. 13- EHC068-EF. Rockville, MD: Agency for Healthcare Research and Quality; April
Arena R, Cahalin LP, Borghi-Silva A, Myers J. The effect of exercise training on the pulmonary arterial system in patients with pulmonary hypertension. Prog Cardiovasc Dis Mar-Apr;57(5): doi: /j.pcad Epub 2014 Mar 27.
CONFIDENTIAL

21. Framing the Story and Next Steps

22. What to Do with the Body of Evidence?
11/9/2018
What to Do with the Body of Evidence?
Or… so what now?
Building an outline that reflects the literature
Depends on what is the literature being used for?
To inform (literature report, landscape assessment)
To support (publication / manuscript, dossier)
Turn THIS
Into THIS
CONFIDENTIAL

23. Step 1 Step 2 Step 3 Step 4 Frame the Story Create buckets
11/9/2018
Frame the Story
Corralling the Literature
Steps to developing the outline and framing the story
Systematic literature reviews have their own worksteps and best practices
Identify the story or value proposition behind the writing (theme)
Group the available evidence for each major topic or outcome (Level 1 outline)
Assess where the majority or quality of evidence is (relative to the theme)
Develop an outline around the evidence (Level 2 outline)
Organize the literature around the outline
Drop in the relevant citations and 2-3 key points to convey from each study
Remove the non-relevant data/studies
Assess where there are no citations: missed or a gap in the evidence?
Determine if this is incorporated into the writing or go back to your search strategy
Create buckets
Develop an outline
Drop in the literature
Gap analysis
Step 1
Step 2
Step 3
Step 4
Kallestinova ED. How to write your first research paper. Yale J Biol Med ;84(3):
CONFIDENTIAL

24. Next Steps
Write and publish!

25. Thank you!