Applicability of the AGREE II Instrument in Evaluating the Development Process and Quality of Current National Academy of Clinical Biochemistry Guidelines A.C. Don-Wauchope, J.L. Sievenpiper, S.A. Hill, and A. Iorio October © Copyright 2012 by the American Association for Clinical Chemistry

© Copyright 2009 by the American Association for Clinical Chemistry Background  Practice guidelines are everywhere in healthcare  They have been present for decades but have been more prevalent in recent times for the following factors: >Assessment and accountability Variability, cost, and utilization  Clinical practice should be guided by the best available evidence  This is a daunting task considering the number of journals and the volume of studies  See accompanying Editorial by Bossuyt PMM

© Copyright 2009 by the American Association for Clinical Chemistry Background (continued)  Guideline methodology  This has improved over time and now there are a number of standardized methodologies available for people who write guidelines  Applicability to Laboratory Medicine  Additional information may need to be considered, in relation to laboratory tests

© Copyright 2009 by the American Association for Clinical Chemistry Question  Are guidelines including laboratory tests important for clinical laboratories?

© Copyright 2009 by the American Association for Clinical Chemistry Evaluation of Guidelines  AGREE collaboration  International collaboration that published an appraisal instrument initially in 2003 and a revised version in 2009  Application to Laboratory Medicine  The instrument has been used to evaluate guidelines with laboratory medicine content but no evaluation of the National Academy of Clinical Biochemistry Laboratory Medicine Practice Guidelines has yet been performed  The 2009 version has not been validated for use in Laboratory Medicine Practice Guidelines

© Copyright 2009 by the American Association for Clinical Chemistry Objectives  Demonstrate the applicability of the AGREE II instrument to Laboratory Medicine Practice Guidelines (LMPGs)  Report the quality of the LMPGs  Make recommendations for improving future LMPGs

© Copyright 2009 by the American Association for Clinical Chemistry AGREE II instrument  6 domains and an overall score  Scope and Purpose  Stakeholder Involvement  Rigor of development  Clarity of presentation  Applicability  Editorial independence  23 key items (see Table 1*)  Scored with a 7-point scale  The largest number of key items is in the rigor of development section. This is important because the methodology for the guideline development process is critical to the quality of the guideline. *Clin Chem 2012; 58: AGREE Trust:

© Copyright 2009 by the American Association for Clinical Chemistry Methods  Guideline selection  National Academy of Clinical Biochemistry website  Guideline appraisal  AGREE II instrument  Training of appraisers using on-line training from AGREE trust website  Random allocation of LMPGs  Independent appraisal with team discussion if scores varied by ≥ 4  Data analysis  As described in AGREE II instrument  Weighted (Quadratic) kappa for appraiser agreement  Mann-Whitney test to compare guidelines before or after 2009

© Copyright 2009 by the American Association for Clinical Chemistry Question  Are these methods reproducible and appropriate for the objectives of the study?

© Copyright 2009 by the American Association for Clinical Chemistry Results  Guideline selection  National Academy of Clinical Biochemistry website had 11 guidelines posted in December 2011  Currently, there are 8 active guidelines and 6 older guidelines that are archived but still available Recommendations For The Use Of Laboratory Tests To Support Poisoned Patients Who Present To The Emergency Department Emerging Biomarkers For Primary Prevention Of Cardiovascular Disease And Stroke. Maternal-Fetal Risk Assessment And Reference Values In Pregnancy. Use Of Tumor Markers In Testicular, Prostate, Colorectal, Breast, And Ovarian Cancers Evidence-Based Practice For Point-Of-Care Testing.Laboratory Analysis and Application of Pharmacogenetics to Clinical Practice. Biomarkers of Acute Coronary Syndromes and Heart Failure. Use of Tumor Markers in Liver, Bladder, Cervical, and Gastric Cancers. Use of Tumor Markers in Clinical Practice: Quality Requirement. Guidelines and Recommendations for Laboratory Analysis in the Diagnosis and Management of Diabetes Mellitus. Follow-up Testing for Metabolic Diseases Identified by Expanded Newborn Screening Using Tandem Mass Spectrometry

© Copyright 2009 by the American Association for Clinical Chemistry Table 2. Assessment of the NACB LMPGs with the AGREE II Instrument

© Copyright 2009 by the American Association for Clinical Chemistry Figure 2 Counts of individual scale (1-7) scores for Domain 3 (Rigor of Development) showing key items The y axis is set to 10 individual counts (max 22 counts). Each item was scored twice for the eleven LMPGs.

© Copyright 2009 by the American Association for Clinical Chemistry Results (continued)  Guideline appraisal  AGREE II instrument was successfully applied  Only 1 item (#16: the different options for management of the condition or health issue are clearly presented) was not applicable in 1 LMPG (Tumor Marker Quality Requirements)  The overall quality of the LMPGs was generally poor -5 had overall scores of > 50% -the median score was 42% -the range was 8%-92%  Sensitivity analysis  There was no significant change between the 2 groups of LMPGs.  There was a suggestion of improving scores.

© Copyright 2009 by the American Association for Clinical Chemistry Question  Are these results important for adapting the LMPGs to your clinical laboratory?

© Copyright 2009 by the American Association for Clinical Chemistry Discussion  Domain 3 (Rigor of development)  3 key items (of 8) were very poorly addressed  The other key items had variable results  Domain 3 is crucial to help the reader assess the validity of the LMPG  Poorly addressed key items:  Use of systematic methods to find evidence  Description of the methods used to evaluate and select evidence  Description of an explicit plan to update the guideline

© Copyright 2009 by the American Association for Clinical Chemistry Discussion  Utility of the AGREE II instrument  The AGREE II instrument was useful in appraising the 11 NACB LMPGs  The domains and item questions were applicable and appropriate  Only the 1 item (#16) raised some concerns and the AGREE instrument does allow for this and the item can be excluded for scoring purposes  AGREE II is applicable to LMPGs  4 guidelines considered AGREE II in their planning  The most recent (Diabetes) scored highest and had followed a more robust methodology, including considering the AGREE criteria

© Copyright 2009 by the American Association for Clinical Chemistry Recommendations 1.Specific description of the population 2.Specific description of who has contributed to the guideline 3.Present methodology in an explicit manner 4.Strategy for updating the guideline must be described 5.Factors that implicate resource use should be described 6.The choice of question must be considered carefully for each guideline

© Copyright 2009 by the American Association for Clinical Chemistry Criticism of the AGREE instrument  AGREE relies on the report which may not reflect the full process  AGREE evaluates the process of producing the report and not the content of the report  AGREE definition of quality is many-layered and requires appraiser judgment  See accompanying Editorial by Bossuyt PMM

© Copyright 2009 by the American Association for Clinical Chemistry Conclusion  Guidelines are important to Laboratory Medicine  The AGREE II instrument is useful for evaluating LMPGs  There are a number of areas that could be improved in the development and reporting of LMPGs

© Copyright 2009 by the American Association for Clinical Chemistry Thank you for participating in this month’s Clinical Chemistry Journal Club. Additional Journal Clubs are available at Follow us