1. Education Program Experience for the Appraisers of Guidelines using AGREE II Scoring Guide in Korea Sung-Goo Chang 1) 6), Ein-Soon Shin 2), Ji-Eun Jang 2), Min-Ji Kim 2), Ji-Yun Yeon 2), You-Kyoung Lee 3), Heui-Sug Jo 4) 6), Dong-Ik Kim 5)6) 1) Department of Urology, Kyung Hee University School of Medicine, Seoul, South Korea 2) Research Agency for Clinical Practice Guidelines, KAMS Research Center, The Korean Academy of Medical Sciences, Seoul, South Korea 3) Department of Laboratory Medicine and Genetics, Soonchunhyang University College of Medicine, Bucheon, South Korea 4) Department of Health Management and Policy, Kangwon National University School of Medicine, Chuncheon, South Korea 5) Department of Diagnostic Radiology, Yonsei University College of Medicine, Seoul, South Korea 6) The Korean Academy of Medical Sciences, Seoul, South Korea

2. http://www.guideline.or.kr Can we reduce differences among 4 appraisers? Guideline ratings require a level of judgment 1. All 23 AGREE II items are rated on a 7-point scale. 2. Score 1 (strongly Disagree) should be given when there is no information or if the concept is very poorly reported. 3. Score 7 (Strongly Agree) should be given when the full criteria in ‘ how to Rate ’ and/or ‘ Further Considerations ’ are articulated in the User ’ s Manual. 4. Scores between 2 and 6: a score is assigned depending on the completeness and quality of reporting. Scores increase as more criteria are met and considerations addressed. To reduce differences among appraisers 1.Plan an effective education program to train qualified appraisers 2.Need to define the concept and details on assessment criteria → develop a scoring guide for anchor points 1, 3, 5, 7 in 23 items 3. Have to identify explicit elements that reflect the operational definition → develop a practical working sheet to identify elements for making correct decision and finding elements that show differences among appraisers

3. http://www.guideline.or.kr Objectives To develop an effective education program to train qualified appraisers (multidisciplinary physicians) of guidelines To apply and examine a Korean AGREE II * scoring guide developed by KAMS that is time-saving and increasing reliability for the assessment of quality of CPGs among appraisers Purpose of the AGREE II (Appraisal of Guidelines for Research & Evaluation II) 1)Assess the quality of guidelines 2)Provide a methodological strategy for the development of guidelines 3)Inform what information and how information ought to be reported in guidelines Purpose of the AGREE II (Appraisal of Guidelines for Research & Evaluation II) 1)Assess the quality of guidelines 2)Provide a methodological strategy for the development of guidelines 3)Inform what information and how information ought to be reported in guidelines To maximize the applicability of a Korean AGREE II scoring guide and reduce the disagreement rate among appraisers providing a practical implementation tool such as working sheet to identify elements for correct judgment and finding elements of differences among appraisers

4. http://www.guideline.or.kr Methods: Education program for evaluators The planned education program was provided to 29 participants who are physicians to assess quality of CPGs on November 29, 2013. Providing a series of training courses including one hour lecture, demonstration and maximum two hours practice. To make correct judgment, 29 participants learned how to apply a developed Korean AGREE II scoring guide and practical working sheet to identify elements in detail. ‘Guideline for prevention and treatment of metabolic syndrome in primary care’ was assessed under 23 items of AGREE II instrument. Finally, evaluators made a judgment score after considering both identified elements and scoring guides for anchor points 1, 3, 5, and 7.

5. http://www.guideline.or.kr Korean version of AGREE II User ’ s Manual http://www.agreetrust.org/resource- centre/agree-ii-translations/

6. http://www.guideline.or.kr Korean AGREE II Scoring Guide - Development 92 draft scoring guides for anchor points 1, 3, 5, and 7 (full score) in 23 items was developed by Executive Committee for CPGs, the Korean Academy of Medical Sciences (KAMS) in Korea. In consideration of the importance of using a quantitative measure of satisfaction with the elements described in the AGREE II manual (analyzed the user’s manual, how to rate, and additional consideration), a final draft was developed through a Delphi consensus process. Consensus to finalize the scoring guide was defined as agreement among at least 70% of the raters.

7. http://www.guideline.or.kr Korean AGREE II Scoring Guide - Development Agreement on 88 of 92 draft scoring guidelines was reached in the first Delphi round (95.7%), and agreement for the remaining 4 was achieved in the second round (4.3%).

8. http://www.guideline.or.kr Korean AGREE II Scoring Guide - an example 평가영역 1. 범위와 목적 (Domain 1. Scope and Purpose) 2. 진료지침에서 다루고자 하는 건강 관련 질문들이 구체적으로 서술되어 있다. (2. The health question(s) covered by the guideline is (are) specifically described.) 7 진료지침에서 다루는 질문이 PICO요소를 포함하고 있고, 별도의 리스트로 되어 있어 찾기 쉽고 내용이 명확히 서술된 경우 (Reported the clinical/health questions with PICO(H) components, easy to find because of provided separate list, and descriptions are clear and concise) 5 질문이 일목요연한 문장이지만, PICO의 요소가 없거나 부족하면서, 별도의 리스트로 되어 있지 않은 경우 (Reported as a structured statements with lack of PICO(H) components and/or no separate list) 3 질문이 단어로만 이루어진 소제목과 같이 최소의 정보로 제시된 경우 (poorly reported or providing a minimum information such as a form of heading with couple of words combination without PICO(H) components) 1 진료지침에서 다루는 질문이 제시되어 있지 않은 경우 (not provided the clinical/health questions or no information related PICO(H))

9. http://www.guideline.or.kr Korean AGREE II Scoring Guide

10. http://www.guideline.or.kr Define disagreement among appraisers Disagreement was defined by more than 4 judgment score differences among appraisers on the same item using the 7-point rating scale. Judgment score of the appraiser #1 : 1 Judgment score of the appraiser #2 : 6 Judgment score of the appraiser #3 : 5 Judgment score of the appraiser #4 : 6 Domain 5. Applicability. 18. The guideline describes facilitators and barriers to its application. Domain 5. Applicability. 18. The guideline describes facilitators and barriers to its application. 1 Strongly Disagree 23456 7 Strongly Agree Disagreement

11. http://www.guideline.or.kr Practical implementation tool to identify explicit elements Develop and apply a practical working sheet to identify elements for correct judgment and easily find elements that show differences among appraisers 영역 3. 개발의 엄격성 (Domain 3 Rigour of Development) 14. 진료지침의 갱신 절차가 제시되어 있다. (A procedure for updating the guideline is provided) 5 (How to rate) Comment ( 평가근거 서술, page 표기 ) (Yes/No) 지침이 개정될 것이라는 언급 (a statement that the guideline will be updated) 명시되어 있음 (p.10), 이에 대한 자세한 일정은 추후 대한소화기기능성질환, 운동학회 웹사이트를 통해 공지할 예정임을 언급함. Y 개정기간을 명시하거나 개정을 결정하는 판단 기준의 명시 (explicit time interval or explicit criteria to guide decisions about when an update will occur) 개정기간 (4-5 년 ) 명시되어 있음 (p.10), 개정결정 판단기준 ( 향후 새로운 검사법이나 약제, 치료법 등이 개발되고 새로운 연구결과가 축적되어 우리나라 국민 건강증진에 필요하다고 판단될 경우 ) 가 명시되어 있음 (p.10) Y 개정과정의 방법론을 제시 (methodology for the updating procedure is reported) 향후에 신규방법으로 개정을 할지 혹은 수용개발 방법으로 개정 할지에 대한 구체적인 언급이 없음 (not described) N

12. http://www.guideline.or.kr Results AGREE II Domains Ite m Agreem ent Disagr eement No respon se Total number of apprais ers N%N%N%N% 1. Scope a nd Purpose 129 100.0 00.00 29 100.0 228 96. 6 13.400.029 100.0 324 82. 8 5 17. 2 00.029 100.0 2. Stakeholde r Involvemen t 429 100.0 00.00 29 100.0 519 65. 5 8 27. 6 26.929 100.0 628 96. 6 00.013.429 100.0 Stakeholder Involvement. Item 5. showed the highest disagreement. The views and preferences of the target population (patients, public, etc.) have been sought.

13. http://www.guideline.or.kr Reasons for the highest disagreement (27.6%) - Stakeholder Involvement. Item 5 Possible reasons: 1.Do not have any qualified data on the view and preferences of patients and/or public in Korea 2.No guidelines to consider the patient’s value and preferences in Korea 3.Appraisers do not understand the definition for the views and preferences of the target population (patients, public, etc.) 4.Showed the highest disagreement because appraisers could not identify the proper elements Terminology: values and preferences (Jeff Andrews et al. GRADE Guidelines 14. 2012) Values and preferences is an overarching term that includes patients’ perspectives, beliefs, expectations, and goals for health and life. More precisely, they refer to the processes that individuals use in considering the potential benefits, harms, costs, limitations, and inconvenience of the management options in relation to one another. For some, the term ‘‘values’’ has the closest connotation to these processes. For others, the connotation of ‘‘preferences’’ best captures the notion of choice. Thus, we use both words together to convey the concept. Domain 2. Stakeholder Involvement. Item 5. showed the highest disagreement. The views and preferences of the target population (patients, public, etc.) have been sought. Domain 2. Stakeholder Involvement. Item 5. showed the highest disagreement. The views and preferences of the target population (patients, public, etc.) have been sought.

14. http://www.guideline.or.kr Results Rigour of Development. Item 11. The health benefits, side effects, and risks have been considered in formulation the recommendations. Rigour of Development. Item 11. The health benefits, side effects, and risks have been considered in formulation the recommendations. AGREE II Domains Ite m Agreem ent Disagr eement No respo nse Total number of apprai sers N%N%N%N% 3. Rigour of Developme nt 729 10 0. 0 00.00 29 100.0 826 89.7 13.42 6. 9 29 100.0 925 86.3 13.43 10.3 29 100.0 1024 82.8 3 10.3 2 6. 9 29 100.0 1120 69.0 7 24. 1 2 6. 9 29 100.0 1223 79.2 26.84 14.0 29 100.0 1323 79.2 26.84 14.0 29 100.0 1422 76.0 4 13.7 3 10.3 29 100.0

15. http://www.guideline.or.kr Reasons for the high disagreement (24.1%) - Rigour of Development. Item 11 Domain 3. Rigour of Development. Item 11. The health benefits, side effects, and risks have been considered in formulation the recommendations. Domain 3. Rigour of Development. Item 11. The health benefits, side effects, and risks have been considered in formulation the recommendations. Possible reasons: 1.Appraisers did not identify the proper elements because of poor understanding - need a plan to provide a repeated education session to understand the exact definition of AGREE II items and educate how to identify the elements 2. Appraisers did not consider the description scoring guide by anchor point 1, 3, 5, 7. - need to make/revise a clear and concise description and educate how to apply scoring guide 3. Showed pretty high disagreement because of limited time to read and evaluate guideline - to make correct judgment after reading and identifying the elements

16. http://www.guideline.or.kr Results 6 to 9 of 29 appraisers (20.7~31.0%) did not rate the 7-point scale on the 3 domains, 9 items among 23 items (39.1%). AGREE II Domains Ite m Agreem ent Disagr eement No respo nse Total number of apprai sers N%N%N%N% 4. Clarity of Presentati on 1523 79.3 00.06 20.7 29 100.0 1619 65.5 26.98 27.6 29 100.0 1722 75.9 13.46 20.7 29 100.0 5. Applicabil ity 18 62.1 26.99 31.0 29 100.0 1920 69.0 13.48 27.6 29 100.0 2018 62.1 26.99 31.0 29 100.0 2119 65.6 13.49 31.0 29 100.0 6. Editorial Independen ce 22 75.9 13.46 20.7 29 100.0 2317 58.6 4 13.8 8 27.6 29 100.0

17. http://www.guideline.or.kr No response: did not rating the 7-point scale - the 3 domains, 9 items among 23 items Main Reason:. Appraisers did not evaluate domain 4, 5 and 6 (9 items of 23) and showed high rate of no response with ranged 20.7% to 31.0% because of limited time to read and evaluate guideline during a practice session -Need to provide enough time to make correct judgment after reading guideline carefully and identifying the proper elements Main Reason:. Appraisers did not evaluate domain 4, 5 and 6 (9 items of 23) and showed high rate of no response with ranged 20.7% to 31.0% because of limited time to read and evaluate guideline during a practice session -Need to provide enough time to make correct judgment after reading guideline carefully and identifying the proper elements Domain 4. Clarity of Presentation 20.7%- 27.6% (3 of 3 items) Domain 5. Applicabili ty 27.6%-31.0% (4 of 4 items) Domain 6. Editorial Independence 20.7%- 27.6% (2 of 2 items)

18. http://www.guideline.or.kr Limitation and p roposed way forward Limitations Proposed way forward Difficult to select a good quality of guideline for education purpose - using an evidence-based ADAPT guideline, could not meet the criteria Limited time (maximum 2 hours for practice) was one of the difficulties to educate and evaluate a guideline Hard to understand and rate the patient values & preferences (Item 5) because have no experience to identify and collect this element in Korea Could not consider the weight among anchor points (1, 3, 5, 7) in 23 items Need constant monitoring and updating of the scoring guide to reflect user’s feedback Providing an easy, clear operational definition for scoring guide Development of the scoring guide in English version will be beneficial not only to domestic appraisers but also to international appraisers

19. http://www.guideline.or.kr Lessons learned and Conclusions Minimizing differences among appraisers is important when measuring quality of CPGs. We demonstrated that Korean AGREE II scoring guide and practical working sheet were time-saving and useful tool to educate appraisers about CPGs with low disagreement rate. The detailed scoring guide for 23 items of AGREE II instrument can successfully clarify the guideline assessment process. Applying a practical implementation tool such as working sheet to identify elements for correct judgment was very easy and useful, and helped finding the elements that caused differences among appraisers.

20. Authors are free of any real or perceived bias introduced by receipt of any benefit in cash or kind, any hospitality, or any subsidy derived from any source that may have or be perceived to have an interest in the outcome of the study. Especially there are no conflicts of interest that relate to industry. For copies of slides: shin2738@kams.or.kr Ein-Soon Shin, PhD, MPH, Lic.Ac.