1. Quality Standards Working Group Meeting
ERS 2017
Milan, 9th September 2017

2. Agenda Working group progress update
Discussion of the two main papers
Priorities for future research, actions, publications, promotion and implementation
Review potential studies
Are they still relevant, feasible, valid and a priority
Other research ideas
Setting priorities
Additional items

3. Attendees
N Roche (chair)

4. Active study update REG_P005:
Phase I: Assess the quality of the asthma comparative effectiveness literature to identify evidence for consideration by future guideline developers
Phase II: Develop (implementable) tools to facilitate quality appraisal of published real-world evidence

5. Assess the quality of the asthma comparative effectiveness literature to identify evidence for consideration by future guideline developers
Paper
Focus
Journal
Lead author
Status 1
Overview of the taskforce work
Allergy
Nicolas
First draft due to be submitted to EAACI and REG 2
Development of the tool and link to (and differentiation from) other work in this area
ERJ?
Jon
First draft almost complete 3
Results of literature review
Clinical and experimental allergy?
On hold – determine whether required following publication of paper 1.

6. REG_P005: Paper 1 Working title Objective Rationale
Quality standards in respiratory real-life effectiveness research: a Respiratory Effectiveness Group – European Academy of Allergy and Clinical Immunology Task Force report
Objective
Provide an overview of how this Task Force developed a quality assessment tool and applied it to some selected PICOT questions for which RCTs provide only limited evidence.
Rationale
In observational research as well as in RCTs, ensuring high-quality methodology is crucial to avoid biases that would compromise the reliability of results.
While quality standards for RCTs are well-defined and extensively disseminated, e.g., by the CONSORT statements, this is less so in the field of observational comparative effectiveness research.
Several tools assess the quality of evidence provided by observational research, they have not been specifically designed for effectiveness research evaluation

7. REG_P005: Paper 1 Methodology
Development and testing of the quality assessment tool
Systematic review of tools proposed in the literature.
Results discussed during Taskforce meetings and used to build a first version of the REG/EAACI tool.
Tested by the taskforce
Refined and formatted
Tested again on a larger basis using all papers retrieved by the literature search
Selection of topics
A first set of 21 PICOT questions was identified during a Taskforce meeting, selected based on their relevance for asthma management, limited ability of available RCTs to provide robust answers and potential of observational research to address them.
Nine were prioritised, and then ranked and feasibility discussed.
Four final questions were selected.
Assessment of the literature
Identify relevant articles corresponding to the PICOT questions
Papers considered of sufficient quality were sent to at least two reviewers
Summary tables were built, mentioning the main conclusion of the articles, the type of study, the GRADE assessment, the possible impact on clinical practice and whether similar evidence is available from RCTs.

8. REG_P005: Paper 1
Results
SECONDARY CRITERIA
1. Background
1.1. The research is based on a review of the background literature (ideal standard is a systematic review)
2. Design
2.1. Evidence of a priori design, e.g. protocol registration in a dedicated website
2.2 Population justified
2.3 The datasource (or database), as described, contains adequate exposures (if relevant) and outcome variables to answer the research question
2.4 Setting justified
3. Measures
3.1 Sample size / Power pre-specified
4. Analysis
NO SECONDARY CRITERIA
5. Results
5.1. Flow chart explaining all exclusions and individuals screened or selected at each stage of defining the final sample
5.2. The authors describe the statistical uncertainty of their findings (e.g. p-values, confidence intervals)
5.3. The extent of missing data is reported
6. Discussion/Interpretation
6.1. Possible biases and/or confounding factors described
7. Conflict of interests
PRIMARY CRITERIA
1. Background
1.1. Clearly stated research question
2. Design
2.1 Population defined
2.2. Comparison groups defined and justified
3. Measures
3.1. (If relevant), exposure (e.g. treatment) is clearly defined
3.2. Primary outcomes defined
4. Analysis
4.1. Potential confounders are addressed
4.2. Study groups are compared at baseline
5. Results
5.1. Results are clearly presented for all primary and secondary endpoints as well as confounders
6. Discussion/Interpretation
6.1. Results consistent with known information or if not, an explanation is provided
6.2 The clinical relevance of the results is discussed
7. Conflict of interests
7.1. Potential conflicts of interest, including study funding, are stated
*For guideline development, secondary criteria only appear if all primary criteria are met

9. REG_P005: Paper 1 Results PICOT questions dealt with:
The influence of adherence/persistence on asthma outcomes
The influence of smoking on asthma outcomes and treatment effectiveness
The impact of the device on outcomes
The relation between particle size of maintenance therapy and asthma outcomes
46 papers were selected
The main limitation of currently available database studies was the lack of detailed clinical data regarding:
control as assessed by validated questionnaires,
severity as assessed by symptoms, and lung function,
symptoms suggestive of comorbidities such as rhinitis or gastro-oesophageal reflux disease.
As a consequence, authors used proxies.
In most retained studies, these factors were used to match and/or adjust analyses, and so the level of evidence could be raised to moderate.
The level could not be raised to high, since residual confounding by hidden factors could not be ruled out. For instance, data on behaviours are frequently lacking, while smoking history or BMI are absent from several databases.

10. REG_P005: Paper 1 Conclusions
The final tool was considered easy to use by members of the REG and EAACI networks who were involved in the testing phase (n=66)
Fair levels of agreement were obtained in the implementation phase
≥73% for primary items and ≥69% for secondary ones.
The high number of PICOT questions (n=21) identified, outlines the evidence gaps that are still present regarding asthma treatment.
This contrasts with the high number of RCTs assessing asthma care, illustrating that classical RCTs are not sufficient to answer all questions regarding real-life effectiveness of therapeutic strategies.
The number of published articles available for each of the PICOT questions was surprisingly low, illustrates that observational effectiveness research in asthma is still infrequently performed.
46% of retrieved papers were considered of sufficient quality to be analysed further in terms of their results and how these could inform current knowledge and recommendations.

11. REG_P005: Paper 2 Working title Objective
The Real Life Evidence AssessmeNt Tool (RELEVANT): development of a novel quality assurance asset
Objective
To outline the methodology and process that lead to the development of the Real Life Evidence AssessmeNt Tool (RELEVANT). The tool was created to assist the Task Force in discriminating flawed CER studies from those of sufficiently high quality to warrant consideration by asthma guideline bodies

12. REG_P005: Paper 2 Methodology
Development and testing of the quality assessment tool was across 6 key phases
Phase I: Review of the literature
Phase II: Initial tool creation
Phase III: Taskforce review and feedback
Phase IV: Pilot application of the quality assessment tool
Phase V: Extended pilot
Phase VI: Development of an online tool

13. REG_P005: Paper 2 Results Initial tool creation (Phase I-III)
A 43-item checklist was created from primary reference papers, and was reduced to 25 quality criteria by the task force members’ discussions.
Tool appraisal and item reduction (Phase IV & V)
Initial pilot:
Evaluation of inter-rater agreement of the initial tool indicated wide variation (33-100%) in concordance across individual criteria
The qualitative feedback suggested this variation was largely driven by differences in semantic interpretation of some of the criteria
This was addressed by rephrasing, removal of any redundancies and eliminating subjective words
Extended pilot:
The inter-rater agreement of the revised tool found concordance in excess of 70% for 94% of primary criteria and 93% of secondary criteria
Qualitative feedback from the pilot raters identified some remaining criteria in the tool that could be further simplified to reduce potential inter-rater variability.

14. REG_P005: Paper 2 Results RELEVANT (Phase VI)
Guides systematic appraisal of the quality of published observational CER papers across seven domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, Conflicts of Interest
Raters must indicate fulfillment (Yes / No / NA) of 11 quality criteria across these seven domains
RELEVANT quality” is defined as fulfillment of all 11 primary criteria.
Failure to meet any one primary criterion reflects a potential “fatal flaw” in a study’s design or, if failure reflects absence of the necessary detail, a lack of necessary transparency in reporting
If all primary criteria are fulfilled, assessment of ten additional, secondary, parameters is prompted to enable further characterization of the relative strengths and weaknesses of the paper

15. REG_P005: Paper 2 Conclusions
RELEVANT is a user-friendly quality appraisal tool comprising 21 quality criteria (11 primary; 10 secondary) across seven core quality domains.
It is the first of its kind to support quality appraisal of published research and to have been developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation.
Possibly look at reviewing these papers side by side and minimising overlap.

16. Next steps Publication of papers
The quality standards in respiratory real-life effectiveness research tool:
Finalise tool
Make it available – where?
Promote it

17. Other research ideas?

18. Setting priorities Are these projects:
Relevant?
Feasible?
Valid?
A priority?
How do we set priorities in quality standards research?
How to we ensure these priorities are pursued?
What are the two most important projects to push forwards?

19. REG Summit 2018: “Working together to make an impact”
Who do we partner with for greatest impact
How do we increase acceptability/validity of real-life research

20. Any other business?