Developing Practice Guidelines for the U.S. Preventive Services Task Force: A Hands-On Approach Kenneth Lin, MD Medical Officer, AHRQ STFM Annual Spring Conference May 2, 2009

Advancing Excellence in Health Care Learning Objectives Define important attributes of good practice guidelines. Define important attributes of good practice guidelines. Review the guideline development process of the U.S. Preventive Services Task Force. Review the guideline development process of the U.S. Preventive Services Task Force. Using USPSTF methods and evidence from a systematic review, update a recommendation on screening for oral cancer. Using USPSTF methods and evidence from a systematic review, update a recommendation on screening for oral cancer.

When do you consult practice guidelines? How do you use practice guidelines? What are some attributes of good practice guidelines?

Advancing Excellence in Health Care Attributes of Good Practice Guidelines Comprehensive, systematic evidence search Comprehensive, systematic evidence search Evidence linked directly to recommendations via strength of recommendation grading system Evidence linked directly to recommendations via strength of recommendation grading system Recommendations based on patient-oriented rather than disease-oriented outcomes Recommendations based on patient-oriented rather than disease-oriented outcomes Development process is transparent Development process is transparent Potential conflicts of interest identified and addressed Potential conflicts of interest identified and addressed Prospective validation Prospective validation Clinical flexibility Clinical flexibility

Advancing Excellence in Health Care Attributes of Good Practice Guidelines Comprehensive, systematic evidence search Comprehensive, systematic evidence search – Evidence-based vs. opinion-based – How to distinguish a systematic from a narrative review – How quickly reviews become out of date

Advancing Excellence in Health Care Attributes of Good Practice Guidelines Evidence linked directly to recommendations via strength of recommendation grading system Evidence linked directly to recommendations via strength of recommendation grading system – Evidence-based is not always evidence-linked – SORT / USPSTF / GRADE Recommendations based on patient-oriented rather than disease-oriented outcomes Recommendations based on patient-oriented rather than disease-oriented outcomes – DOEs versus POEMs

Advancing Excellence in Health Care History of the USPSTF Canadian Task Force on PHE Canadian Task Force on PHE USPSTF established by USPHS USPSTF established by USPHS Community Task Force (CDC) Community Task Force (CDC) rd USPSTF reconvened by AHRQ rd USPSTF reconvened by AHRQ Standing USPSTF Task Force Standing USPSTF Task Force

Advancing Excellence in Health Care Composition of the USPSTF Government support but independent Government support but independent Experts in primary care, prevention, methods Experts in primary care, prevention, methods Family medicine, internal medicine, pediatrics, obstetrics/gynecology, nursing, behavioral medicine, preventive medicine Family medicine, internal medicine, pediatrics, obstetrics/gynecology, nursing, behavioral medicine, preventive medicine Scientific support from a university Evidence- Based Practice center (currently OHSU) Scientific support from a university Evidence- Based Practice center (currently OHSU) Liaisons from primary care subspecialty societies, Federal agencies Liaisons from primary care subspecialty societies, Federal agencies

Advancing Excellence in Health Care Steps in the Recommendation Development Process 1. Define questions and outcomes of interest 2. Define and retrieve relevant evidence 3. Evaluate QUALITY of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits

Advancing Excellence in Health Care Step 1: Analytic Framework on Screening for a Condition

Advancing Excellence in Health Care Step 1: Analytic Framework for Prostate Cancer Screening Early Prostate Cancer Reduced prostate cancer morbidity, mortality Asymptomatic Men Screen: PSA, DRE Treat radiation, prostatectomy 3 Adverse effects of screening: false positive, false negative, inconvenience, Labeling, worry Adverse effects of Rx: Impotence, incontinence, death, overtreatment

Advancing Excellence in Health Care Steps in the Recommendation Development Process 1. Define questions and outcomes of interest 2. Define and retrieve relevant evidence 3. Evaluate QUALITY of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits

Advancing Excellence in Health Care Steps in the Recommendation Development Process Create inclusion/exclusion criteria based on the key questions from the analytic framework Create inclusion/exclusion criteria based on the key questions from the analytic framework – Types of studies – Outcomes – Population – Setting – Time period Sources of Evidence Sources of Evidence – Medline, Cochrane literature searches – Reference mining- key articles, editorials, review articles – Expert recommendations

Advancing Excellence in Health Care Steps in the Recommendation Development Process 1. Define questions and outcomes of interest 2. Define and retrieve relevant evidence 3. Evaluate quality of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits

Advancing Excellence in Health Care Step 3: Evaluate Quality of Individual Studies Good: Good: – Evaluates relevant available screening tests – Uses a credible reference standard – Interprets reference standard independently of screening test – Large sample size, ~ 100 broad spectrum patients Fair: Fair: – Evaluates relevant available screening tests – Uses reasonable although not best standard; – Interprets reference standard independent of screening test; – Moderate sample size, ~ “medium” spectrum patients Poor: Has fatal flaw such as: Poor: Has fatal flaw such as: – Uses inappropriate reference standard – Screening test improperly administered – Biased ascertainment of reference standard – Very small sample size or very narrow selected spectrum of patients.

Advancing Excellence in Health Care Steps in the Recommendation Development Process 1. Define questions and outcomes of interest 2. Define and retrieve relevant evidence 3. Evaluate quality of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits

Advancing Excellence in Health Care Step 4: Synthesize and Judge Strength of Overall Evidence Evidence reports Evidence reports – Evidence tables summarizing characteristics and quality of studies – Narrative discussing overall strength of evidence Systematic reviews from others – Systematic reviews from others – Other AHRQ reports, Cochrane, etc. Other AHRQ reports, Cochrane, etc. Meta-analysis Meta-analysis Modeling Modeling – Projected outcomes tables, decision analysis

Advancing Excellence in Health Care Critical Appraisal Questions Do the studies have the appropriate research design to answer the key question? Do the studies have the appropriate research design to answer the key question? To what extent are the existing studies high quality? To what extent are the existing studies high quality? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? How many studies have been conducted that address the key question? How large are the studies? How many studies have been conducted that address the key question? How large are the studies? How consistent/coherent are the results of the studies? How consistent/coherent are the results of the studies? Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e.g., presence or absence of dose-response effects; fit within a biologic model) Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e.g., presence or absence of dose-response effects; fit within a biologic model)

Advancing Excellence in Health Care Step 4: Synthesize & Judge Strength of Evidence for Each Key Question Convincing: Well-designed, well-conducted studies in representative populations that directly assess effects on health outcomes Adequate: Evidence sufficient to determine effects on health outcomes, but limited by number, quality, or consistency of studies, generalizability to routine practice, or indirect nature of the evidence. Inadequate: Insufficient evidence to determine effect on health outcomes due to limited number or power of studies, important flaws in their design or conduct, gaps in the chain of evidence, or lack of information on important health outcomes

Advancing Excellence in Health Care Step 4: Synthesize and Judge Strength of Overall Evidence: Certainty Definition: The U.S. Preventive Services Task Force defines certainty as “likelihood that the USPSTF assessment of the net benefit of a preventive service is correct”. Definition: The U.S. Preventive Services Task Force defines certainty as “likelihood that the USPSTF assessment of the net benefit of a preventive service is correct”. The net benefit is defined as benefit minus harm of the preventive service as implemented in a general, primary care population. The USPSTF assigns a certainty level based on the nature of the overall evidence available to assess the net benefit of a preventive service. The net benefit is defined as benefit minus harm of the preventive service as implemented in a general, primary care population. The USPSTF assigns a certainty level based on the nature of the overall evidence available to assess the net benefit of a preventive service.

Advancing Excellence in Health Care Levels of Certainty: High, Moderate, or Low High: The available evidence usually includes consistent results from well-designed, well- conducted studies in representative primary care populations. High: The available evidence usually includes consistent results from well-designed, well- conducted studies in representative primary care populations. Moderate: The available evidence is sufficient to determine the effects of the preventive service on health outcomes. As more information becomes available, the magnitude or direction of the observed effect could change, and this change may be large enough to alter the conclusion. Moderate: The available evidence is sufficient to determine the effects of the preventive service on health outcomes. As more information becomes available, the magnitude or direction of the observed effect could change, and this change may be large enough to alter the conclusion. Low: The available evidence is insufficient to assess effects on health outcomes. Low: The available evidence is insufficient to assess effects on health outcomes.

Advancing Excellence in Health Care Critical Appraisal Questions Do the studies have the appropriate research design to answer the key questions? Do the studies have the appropriate research design to answer the key questions? To what extent are the existing studies high quality? To what extent are the existing studies high quality? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? How many studies have been conducted that address the key questions? How large are the studies? How many studies have been conducted that address the key questions? How large are the studies? How consistent/coherent are the results of the studies? How consistent/coherent are the results of the studies? Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e.g., presence or absence of dose-response effects; fit within a biologic model) Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e.g., presence or absence of dose-response effects; fit within a biologic model)

Advancing Excellence in Health Care Steps in the Recommendation Development Process 1. Define questions and outcomes of interest 2. Define and retrieve relevant evidence 3. Evaluate quality of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to net benefits

Advancing Excellence in Health Care Step 5: Determine Balance of Benefits and Harms Estimate Magnitude of Net Benefit Benefits – Harms = Net Benefit

Advancing Excellence in Health Care Step 5: Determine Balance of Benefits and Harms: Assessing Harms Potential harms real but hard to quantify Potential harms real but hard to quantify Include psychological and physical consequences of false-positives, “labeling,” overtreatment Include psychological and physical consequences of false-positives, “labeling,” overtreatment Opportunity costs Opportunity costs – Time and effort required by patients and the health care system to provide the preventive service – Not the same as financial costs Magnitude of harm subjective, hard to compare to benefits Magnitude of harm subjective, hard to compare to benefits – NNH (eg, GI bleeds from aspirin)

Advancing Excellence in Health Care Step 5: Determine Balance of Benefits & Harms: Assessing Magnitude of Net Benefit No explicit criteria for magnitude categories No explicit criteria for magnitude categories – Zero/negative, small, moderate, substantial Substantial benefit : impact on condition with high population burden or major effect on uncommon outcome Substantial benefit : impact on condition with high population burden or major effect on uncommon outcome Problems: requires evidence on harms and common metric for benefit and harms Problems: requires evidence on harms and common metric for benefit and harms Always requires judgment Always requires judgment

Advancing Excellence in Health Care Steps in the Recommendation Development Process 1. Define questions and outcomes of interest 2. Define and retrieve relevant evidence 3. Evaluate quality of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to net benefits

Advancing Excellence in Health Care Step 6: Link recommendation to net benefits: USPSTF Grades of Recommendations Certainty of Net Benefit Magnitude of Net Benefit SubstantialModerateSmallZero/negative HighABCD ModerateBBCD LowInsufficient

Advancing Excellence in Health Care Step 6: Link recommendation to net benefits: Insufficient Evidence Lack of studies on harms or benefits Lack of studies on harms or benefits Poor quality of existing studies Poor quality of existing studies Good quality studies with conflicting results Good quality studies with conflicting results

Advancing Excellence in Health Care Step 6: Link recommendation to net benefits: USPSTF Wording of Recommendations Grade Grade Definition Suggestion for Practice A The USPSTF recommends the service. There is high certainty that the net benefit is substantial. Offer or provide this service. B The USPSTF recommends the service. There is high certainty that the net benefit is moderate or there is moderate certainty that the net benefit is moderate to substantial. Offer or provide this service. C The USPSTF recommends against routinely providing the service. There may be considerations that support providing the service in an individual patient. There is moderate or high certainty that the net benefit is small. Offer or provide this service only if there are other considerations that support offering or providing the service in an individual patient. D The USPSTF recommends against the service. There is moderate or high certainty that the service has no net benefit or that the harms outweigh the benefits. Discourage the use of this service. I The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of the service. Evidence is lacking, of poor quality, or conflicting, and the balance of benefits and harms cannot be determined. Read “Clinical Considerations” section of USPSTF Recommendation Statement. If offered the service, patients should understand the uncertainty about the balance of benefits and harms.

Recommendation Exercise: Screening for Oral Cancer

Advancing Excellence in Health Care USPSTF Recommendation, 2004 The USPSTF concludes that the evidence is insufficient to recommend for or against routinely screening adults for oral cancer. The USPSTF concludes that the evidence is insufficient to recommend for or against routinely screening adults for oral cancer. I recommendation

Advancing Excellence in Health Care USPSTF Rationale and Clinical Considerations, 2004 The USPSTF found no new good quality evidence that screening for oral cancer leads to improved health outcomes for either high- risk adults (ie, those over the age of 50 who use tobacco) or for average-risk adults in the general population. The USPSTF found no new good quality evidence that screening for oral cancer leads to improved health outcomes for either high- risk adults (ie, those over the age of 50 who use tobacco) or for average-risk adults in the general population. Clinicians should be alert to the possibility of oral cancer when treating patients who use tobacco or alcohol. Clinicians should be alert to the possibility of oral cancer when treating patients who use tobacco or alcohol.

Advancing Excellence in Health Care Epidemiology 2 percent of all cancer cases in U.S. 2 percent of all cancer cases in U.S. More than 7,500 deaths attributed to oral cancer in U.S. in 2008 More than 7,500 deaths attributed to oral cancer in U.S. in 2008 Overall 5-year survival rate of about 60% Overall 5-year survival rate of about 60% 50-60% have regional or distant metastases at time of diagnosis 50-60% have regional or distant metastases at time of diagnosis 4 in 5 patients with oral cancer have used tobacco products 4 in 5 patients with oral cancer have used tobacco products

Advancing Excellence in Health Care Potentially premalignant lesions Defined based on visual appearance Defined based on visual appearance Leukoplakia Leukoplakia – White plaque-like lesion – Precursor for up to 85% of oral cancers – Up to half regress spontaneously Erythroplakia Erythroplakia – Red patch of oral tissue not associated with inflammation – Rarer than leukoplakia, but much more likely to transform into cancer

Advancing Excellence in Health Care Leukoplakia, erythroplakia, and oral cancer Sources: Sources:

Advancing Excellence in Health Care The Oral Cancer Examination Includes inspection and palpation of all the extraoral and intraoral soft tissues Includes inspection and palpation of all the extraoral and intraoral soft tissues Extraoral: lips, salivary glands, submandibular and cervical lymph nodes Extraoral: lips, salivary glands, submandibular and cervical lymph nodes Intraoral: interior lips, buccal mucosa, tongue, floor of mouth, gingivae, tonsils, soft palate, and retromolar trigone (area behind the wisdom teeth) Intraoral: interior lips, buccal mucosa, tongue, floor of mouth, gingivae, tonsils, soft palate, and retromolar trigone (area behind the wisdom teeth) In 1998, 13.8% of U.S. adults age 40 and older reported having had an oral cancer examination within the past year (HP 2010 goal is 20%) In 1998, 13.8% of U.S. adults age 40 and older reported having had an oral cancer examination within the past year (HP 2010 goal is 20%)

Advancing Excellence in Health Care The Oral Cancer Examination Source: American Family Physician 2002;65: Source: American Family Physician 2002;65:

Advancing Excellence in Health Care Current clinical context for oral cancer screening Since 2004, multiple “adjunctive diagnostic aids” have been developed and aggressively promoted for oral cancer screening by commercial interests Since 2004, multiple “adjunctive diagnostic aids” have been developed and aggressively promoted for oral cancer screening by commercial interests While organizations such as the American Dental Association have generally concluded that evidence for the usefulness of these aids is lacking, they have also reaffirmed the necessity of routine oral cancer screening with visual examination While organizations such as the American Dental Association have generally concluded that evidence for the usefulness of these aids is lacking, they have also reaffirmed the necessity of routine oral cancer screening with visual examination

Advancing Excellence in Health Care Critical Evidence Gaps identified from previous USPSTF reviews No direct evidence linking screening for oral cancer to improved health outcomes No direct evidence linking screening for oral cancer to improved health outcomes Limited information on the accuracy of the oral examination for detecting cancers or potentially pre-malignant lesions Limited information on the accuracy of the oral examination for detecting cancers or potentially pre-malignant lesions

Advancing Excellence in Health Care Analytic Framework Adults without oral cancer symptoms Screening for oral cancer Improved morbidity or mortality Harms KQ1 - Does screening for oral cancer reduce morbidity or mortality? (KQ 1) KQ2 – What is the yield and accuracy of screening for oral cancer? Identification of oral cancer or pre-malignant lesions Harms

Advancing Excellence in Health Care 2009 Update Critical Key Questions KQ1: Does screening for oral cancer reduce morbidity or mortality? KQ1: Does screening for oral cancer reduce morbidity or mortality? KQ2: What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer? KQ2: What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer?

Advancing Excellence in Health Care Contextual Questions What are the incidence and mortality rates from oral cancer in U.S. adults who are not at increased risk? What are the incidence and mortality rates from oral cancer in U.S. adults who are not at increased risk? How much time does it take for a primary care clinician to perform a comprehensive screening oral examination? How much time does it take for a primary care clinician to perform a comprehensive screening oral examination? What are the observed rates of progression from potentially premalignant oral lesions to oral cancers? Does HPV infection affect these rates? What are the observed rates of progression from potentially premalignant oral lesions to oral cancers? Does HPV infection affect these rates? Does treatment of potentially premalignant lesions reduce the risk of developing oral cancer? Does treatment of potentially premalignant lesions reduce the risk of developing oral cancer?

Contextual Questions

Advancing Excellence in Health Care Oral cancer incidence and mortality rates in “average risk” U.S. adults No evidence found No evidence found

Advancing Excellence in Health Care How much time does an oral cancer examination take? Expert opinion ranged from 1-5 minutes Expert opinion ranged from 1-5 minutes – “Perform a death-defying act: the 90-second oral cancer examination” – “Oral cancer screening: 5 minutes to save a life” No objective studies have measured actual oral examination times for various health professionals No objective studies have measured actual oral examination times for various health professionals

Advancing Excellence in Health Care Natural history of premalignant lesions. Does HPV matter? Greater than half of all leukoplakias spontaneously regress without treatment Greater than half of all leukoplakias spontaneously regress without treatment Spontaneous regression is more likely to occur in tobacco users who quit Spontaneous regression is more likely to occur in tobacco users who quit Rate of progression from leukoplakia to oral cancer has been estimated at less than 1% per year in Western European populations Rate of progression from leukoplakia to oral cancer has been estimated at less than 1% per year in Western European populations HPV types 16 and 18 are predominantly associated with cancers of the pharynx that are not detectable on visual examination HPV types 16 and 18 are predominantly associated with cancers of the pharynx that are not detectable on visual examination

Advancing Excellence in Health Care How effective is treatment of premalignant lesions? Treatments for leukoplakia include surgery, systemic and topical medications, and counseling for tobacco and alcohol cessation Treatments for leukoplakia include surgery, systemic and topical medications, and counseling for tobacco and alcohol cessation 2006 Cochrane review identified no RCTs of surgery that included a placebo arm; no benefit of treatment seen with non-surgical interventions 2006 Cochrane review identified no RCTs of surgery that included a placebo arm; no benefit of treatment seen with non-surgical interventions Incidence of oral cancer after surgical excision of leukoplakia in retrospective studies is similar to that observed in some populations without treatment Incidence of oral cancer after surgical excision of leukoplakia in retrospective studies is similar to that observed in some populations without treatment

Critical Key Questions

Advancing Excellence in Health Care Methods Searched PubMed and the Cochrane Library from January 2001 – October 2008 (KQ1) and 1966 – October 2008 (KQ2) Searched PubMed and the Cochrane Library from January 2001 – October 2008 (KQ1) and 1966 – October 2008 (KQ2) – KQ 1: RCTs, systematic reviews, and meta- analyses that reported morbidity or mortality outcomes in a screened group vs. usual care – KQ 2: Diagnostic accuracy studies comparing the result of a uniformly applied screening test for oral cancer (which could include diagnostic aids) to a reference standard

Advancing Excellence in Health Care Literature Search Results Potentially relevant articles retrieved Potentially relevant articles retrieved – KQ1: 252 – KQ2: articles pulled for full-text review 42 articles pulled for full-text review 8 articles met inclusion criteria for KQ1 or KQ2 8 articles met inclusion criteria for KQ1 or KQ2

Advancing Excellence in Health Care Key Question 2 What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer? What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer?

Advancing Excellence in Health Care KQ2: Results One systematic review and 4 primary studies met all inclusion criteria One systematic review and 4 primary studies met all inclusion criteria All of the primary studies were published in 1997 or earlier All of the primary studies were published in 1997 or earlier Reported the accuracy of the oral examination for detecting potentially premalignant lesions, since detection of true oral cancers was uncommon (5 out of 310 positive findings from all 4 studies) Reported the accuracy of the oral examination for detecting potentially premalignant lesions, since detection of true oral cancers was uncommon (5 out of 310 positive findings from all 4 studies)

Advancing Excellence in Health Care KQ2: Limitations Not best reference standard Not best reference standard Possibility that reference standard may have been affected by patient’s knowledge of screening exam result Possibility that reference standard may have been affected by patient’s knowledge of screening exam result Community or dental clinic samples may not be representative of primary care population Community or dental clinic samples may not be representative of primary care population Some detected lesions (e.g. lichen planus, submucous fibrosis) now believed to have no malignant potential Some detected lesions (e.g. lichen planus, submucous fibrosis) now believed to have no malignant potential

Advancing Excellence in Health Care KQ2: Summary There is reasonable agreement between screening oral examination findings of generalists and oral medicine specialists There is reasonable agreement between screening oral examination findings of generalists and oral medicine specialists Accuracy of visual examination for detecting asymptomatic oral cancer is not known Accuracy of visual examination for detecting asymptomatic oral cancer is not known Similarly, meaningful false positive rates cannot be calculated from existing data Similarly, meaningful false positive rates cannot be calculated from existing data

Advancing Excellence in Health Care Convincing, Adequate, or Inadequate Evidence? What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer? What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer?

Advancing Excellence in Health Care Key Question 1 Does screening for oral cancer reduce morbidity or mortality? Does screening for oral cancer reduce morbidity or mortality?

Advancing Excellence in Health Care KQ1: Results 3 articles met inclusion criteria 3 articles met inclusion criteria 1 systematic review and 2 separate reports (interim and final) of a single study of screening for oral cancer in Kerala, India 1 systematic review and 2 separate reports (interim and final) of a single study of screening for oral cancer in Kerala, India

Advancing Excellence in Health Care KQ1: the Kerala trial - methods Cluster RCT involving more than 190,000 adults age 35 years and older Cluster RCT involving more than 190,000 adults age 35 years and older Screening group invited to 3 rounds of screening between 1996 and 2004 (30% of subjects participated in all 3 rounds) Screening group invited to 3 rounds of screening between 1996 and 2004 (30% of subjects participated in all 3 rounds) Screeners were health workers with non-medical university degrees who received 3 months of training in interviewing, tobacco and alcohol use counseling, and oral cancer examination Screeners were health workers with non-medical university degrees who received 3 months of training in interviewing, tobacco and alcohol use counseling, and oral cancer examination Persons with positive screens were referred to a specialized clinic and re-examined by a dentist or oncologist; one pathologist evaluated all biopsy samples Persons with positive screens were referred to a specialized clinic and re-examined by a dentist or oncologist; one pathologist evaluated all biopsy samples

Advancing Excellence in Health Care KQ1: the Kerala trial - results Primary outcome: death from oral cancer Primary outcome: death from oral cancer Overall, no effect of screening on outcome Overall, no effect of screening on outcome – 77 OC deaths in screening group, 87 OC deaths in control group – RR 0.79 – NS – (95% CI, ) Benefit of screening seen in “users of tobacco, alcohol, or both” (mostly men) Benefit of screening seen in “users of tobacco, alcohol, or both” (mostly men) – 70 OC deaths in screening group, 85 OC deaths in control group – RR 0.66 (95% CI, 0.45 – 0.95)

Advancing Excellence in Health Care Limitations of the Kerala trial Small number of clusters relative to study population size and geographical proximity Small number of clusters relative to study population size and geographical proximity Low participation (30%) in all 3 rounds of screening Low participation (30%) in all 3 rounds of screening Question about generalizability of “high risk” results to U.S. patients given very high rates of oral cancer seen in Kerala population Question about generalizability of “high risk” results to U.S. patients given very high rates of oral cancer seen in Kerala population

Advancing Excellence in Health Care KQ1: Summary One fair quality randomized controlled trial showed no difference in oral cancer mortality from screening a population with a very high incidence of oral cancer One fair quality randomized controlled trial showed no difference in oral cancer mortality from screening a population with a very high incidence of oral cancer

Advancing Excellence in Health Care Convincing, Adequate, or Inadequate Evidence? Does screening for oral cancer reduce morbidity or mortality? Does screening for oral cancer reduce morbidity or mortality?

Advancing Excellence in Health Care What Is the Net Benefit of Screening for Oral Cancer? Estimate Magnitude of Net Benefit Benefits of Screening – Harms of Screening = Net Benefit Net Benefit may be Zero/Negative, Small, Moderate, or Substantial.

Advancing Excellence in Health Care High, Moderate or Low Certainty? Do the studies have the appropriate research design to answer the key questions? Do the studies have the appropriate research design to answer the key questions? To what extent are the existing studies high quality? To what extent are the existing studies high quality? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? How many studies have been conducted that address the key questions? How large are the studies? How many studies have been conducted that address the key questions? How large are the studies? How consistent/coherent are the results of the studies? How consistent/coherent are the results of the studies? Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e.g., presence or absence of dose-response effects; fit within a biologic model) Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e.g., presence or absence of dose-response effects; fit within a biologic model)

Advancing Excellence in Health Care Step 6: Link recommendation to net benefits: USPSTF Grades of Recommendations Certainty of Net Benefit Magnitude of Net Benefit SubstantialModerateSmallZero/negative HighABCD ModerateBBCD LowInsufficient

Advancing Excellence in Health Care Vote on one of 3 letter grades C – “The USPSTF recommends against routinely screening for oral cancer. There may be considerations that support screening in an individual patient.” C – “The USPSTF recommends against routinely screening for oral cancer. There may be considerations that support screening in an individual patient.” D – “The USPSTF recommends against screening for oral cancer.” D – “The USPSTF recommends against screening for oral cancer.” I – “The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of screening for oral cancer.” I – “The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of screening for oral cancer.”

Advancing Excellence in Health Care What did the USPSTF decide? Stay tuned… Stay tuned… Updated recommendation to be released in late 2009 or early 2010 Updated recommendation to be released in late 2009 or early 2010 Sign up for Task Force updates at Sign up for Task Force updates at

Advancing Excellence in Health Care Acknowledgements Evidence review co-authors Evidence review co-authors – Melissa Camp, MD, MPH Johns Hopkins Bloomberg School of Public Health Johns Hopkins Bloomberg School of Public Health – Val W. Finnell, MD, MPH Uniformed Services University of the Health Sciences Uniformed Services University of the Health Sciences – Lolita Ramsey, MSN, RN George Mason University School of Nursing George Mason University School of Nursing

Advancing Excellence in Health Care Acknowledgements Expert Reviewers Expert Reviewers – Celeste Abraham Baylor College of Dentistry, Texas A&M Health Science Center Baylor College of Dentistry, Texas A&M Health Science Center – Daniel Deschler Department of Otology and Laryngology, Harvard Medical School Department of Otology and Laryngology, Harvard Medical School – William Kohn Division of Oral Health, NCCDPHP, CDC Division of Oral Health, NCCDPHP, CDC