1. cap.org v. 1 Gynecologic Cytopathology Quality Consensus Conference Working Group 4: Cytologic-Histologic Correlations June 4, 2011

2. Barbara A Crothers, DO FCAP, Chair Bruce A Jones, MD FCAP, Senior Author Leigh Ann Cahill, CT (ASCP) CMIAC Ann T Moriarty, MD FCAP Dina R Mody, MD FCAP William D Tench, MD FCAP Rhona J Souers, MS, CAP Biostatistician Working Group 4 Members © 2011 College of American Pathologists. All rights reserved. 2

3. Cytologic-Histologic Correlation (CHC) o CHC is first as valuable QA measure − 4.2 out of 5 points o Most find CHC statistical assessment worthwhile o 94% actively monitor the correlation between Pap test and biopsy results o 64% use CHC as part of QA reports o Corrective action for CHC is infrequent − 15% of 133 participants Summary of Laboratory Trends © 2011 College of American Pathologists. All rights reserved. 3

4. §493.1274(c)(2)- Laboratory comparison of clinical information, when available, with cytologic reports and comparison of all gynecologic reports with a diagnosis of HSIL+ with the histopathology report, if available in the laboratory, and determination of the causes of any discrepancies CAP LAP CYP.01900,.07530,.07543,.07556, and.07569 CLIA ‘88/ CAP © 2011 College of American Pathologists. All rights reserved. 4

5. “This monitor should not be viewed as just an evaluation of the performance of the cytopathology laboratory. This is a true ‘system’ monitor. The statistics represent the performance of all personnel and processes in the system of obtaining, processing, and evaluating both cytology and biopsy specimens. Performance must be evaluated at each step of the process to identify improvement opportunities.” © 2011 College of American Pathologists. All rights reserved. 5 Pap – Biopsy Correlation Jones BA, Davey DD. Quality Management in Gynecologic Cytology Using Interlaboratory Comparison Arch Pathol Lab Med. 2000;124:672–681

6. Traditionally, a measure of Pap test performance Actually represents many quality variables Biopsy not a gold standard Poor interobserver reproducibility o ASCUS, CIN1, LSIL, CIN2 Major cause of discordant pairs is sampling* o Both Pap test and cervical biopsy o Of FN cytology (n=1444), 85% sampling error o Of FP cytology (n=1527), 95% sampling error Focus of CHC- Pap test © 2011 College of American Pathologists. All rights reserved. 6 *Jones and Novis. Cervical biopsy-cytology correlation: a CAP Q-Probes study of 22,439 correlations in 348 laboratories. Arch Pathol Lab Med 1996;120:532-531.

7. “Real Time” correlation- review of slides prior to issue of biopsy report o Provides critical information for patient follow-up o Resolves/confirms discrepancies o Timely report to healthcare providers “Retrospective” correlation- review of slides after issuance of both reports o Monitor performance and processes of cytology and biopsy for laboratory quality improvement Dual Role of CHC © 2011 College of American Pathologists. All rights reserved. 7

8. 67% of laboratories resolve discrepancies at biopsy sign-out and have a written policy (70%) 61% address discrepancies in the biopsy report 85% have written policy requiring retrospective CHC for statistical purposes 58% review cases retrospectively* (Sec 4, Table 33*) Bidirectional correlation was also the most commonly described CH activity in the post-survey Web-based public comment (n=13/27); only 33% (9/27) initiated correlation solely based on the biopsy result 59% ( n=16/27) felt optimum time for CHC was during sign out of biopsy “Bidirectional” CHC © 2011 College of American Pathologists. All rights reserved. 8 * Multiple answers were permitted

9. CHC Consensus Statement #1 Cytologic-histologic correlation may be performed “real time,” retrospectively or both.

10. “ Real time” CHC o Preferentially impacts immediate patient care o Strongly preferred for HSIL Pap test/ negative biopsy regardless of the result of the review (TP or FP) Retrospective correlation o Ease of statistical data collection o Ease of identifying laboratory trends Must notify provider for HSIL Pap and negative biopsy Monitor results weekly, monthly, quarterly or annually Justification: Literature, survey, professional opinion Consensus Statement #1: CHC may be performed “real time,” retrospectively or both. © 2011 College of American Pathologists. All rights reserved. 10

11. Laboratories should be allowed to define the process for CHC (“real time,” retrospective, or both). A. Agree 87.5% B. Disagree 12.5% CS 4.1 QUESTION #37: © 2011 College of American Pathologists. All rights reserved. 11

12. CHC Consensus Statement #2 At a minimum, review all available slides for HSIL Pap tests with a negative biopsy.

13. Recommended time interval between Pap and biopsy-within 6 months of biopsy; with 3-4 months preferred when possible Method for notification of care giver and documentation o May be in biopsy report o May be in QA report o Does not require disclosure of QA information- may choose to issue generic comment(s) For “real time” review, assumes that pathologist takes necessary steps to ensure adequate biopsy orientation and leveling Justification: Survey, professional opinion, literature Consensus Statement #2: At a minimum, review all available slides for HSIL Pap-negative biopsy discrepancies. © 2011 College of American Pathologists. All rights reserved. 13

14. Post-survey Web-based public comment: o 77% review slides for both biopsy and cytology in discordant cases o 59% correlate the Pap and biopsy but only review the Pap if biopsy not available o 25% attempt to obtain the biopsy from another laboratory o 66% obtain levels to expose a deeper lesion − 48% because s/o dysplasia − 24% do not obtain add’l levels o Reviews are sometimes at the discretion of the pathologist or requested by clinicians Survey Results © 2011 College of American Pathologists. All rights reserved. 14

15. For “incident” (first Pap with…) HSIL/AIS/cancer Pap test target- correlate with most abnormal tissue within 6-month interval o Exclude Pap test in conjunction with biopsy unless no access to incident Pap test o Exclude ECC without cervical biopsies unless containing SIL/AIS/cancer Include LEEP/Cone/ Hysterectomy Specimens to correlate for standard statistics: Pap Target © 2011 College of American Pathologists. All rights reserved. 15

16. For HSIL/AIS/cancer biopsy target, correlate with the most abnormal prior Pap test within 6-month interval Exclude Pap tests taken at the time of biopsy unless no earlier Pap test is available* As additional quality monitor, laboratories may choose to review less abnormal Pap tests when multiple Pap tests are available Specimens to correlate for standard statistics: Biopsy target © 2011 College of American Pathologists. All rights reserved. 16 * Panos et al. Usefulness of concurrent Papanicoloau smear at time of cervical biopsy. Diag Cytopathol. 2001;25(4):270-273.

17. The correlation interval between the Pap test and the biopsy should preferably be within 3-4 months, but no greater than 6 months. A. Agree as stated 89% B. Disagree - too long 3.03% C. Disagree - too short 7.58% CS 4.2 QUESTION #38: © 2011 College of American Pathologists. All rights reserved. 17

18. CHC Consensus Statement #3 Standardization of metrics and CHC process is desirable.

19. Allows for inter-laboratory comparison Monitor: o Total number of CHC pairs o Number of positive correlations (“true positive,” as defined prior to review) o Number of negative correlations (“false positive,” as defined prior to review) Calculate Positive Predictive Value (PPV) of a positive Pap test Tabulate statistics at least annually o More frequently for high-volume laboratories Justification: Survey, professional opinion Consensus Statement #3: Standardization of metrics and CHC process is desirable. © 2011 College of American Pathologists. All rights reserved. 19

20. 50% want CHC standardized among laboratories 35% unsure Prefer to see: o 67%- Acceptable actions for discrepancies and statistics o 72%- Type of statistics to maintain o 50%- Time frame limit from abnormal Pap test to cervical biopsy 62% support development of standardized comments for follow up recommendations for positive Pap, negative biopsy Post-Survey Web Comments © 2011 College of American Pathologists. All rights reserved. 20

21. 78% monitor the percent of positive Pap tests that correlate with biopsies, but only 13% monitor the “PPV” of a positive Pap test (the same computation) (Sec 4, Table 35) Larger laboratories are more likely to measure screening/interpretive sensitivity or Pap sensitivity/ specificity (Survey volume analysis data) Survey Results © 2011 College of American Pathologists. All rights reserved. 21

22. Standardization of metrics and CHC process is desirable. A. Agree 93.65% B. Disagree 6.35% CS 4.3 QUESTION #39: © 2011 College of American Pathologists. All rights reserved. 22

23. CHC Consensus Statement #4 PPV of a positive Pap test is the preferred standard CHC metric.

24. For Pap tests, PPV = TP / TP + FP, where o True Positive= “positive” pair o False Positive= “positive” Pap test but “negative” biopsy “Screening” role of Pap test Standardized correlation methods required Assumes biopsy gold standard A minimum of 20 total events necessary for meaningful data Justification: Literature, survey, professional opinion Consensus Statement #4: PPV of a positive Pap test is the preferred standard CHC metric. © 2011 College of American Pathologists. All rights reserved. 24

25. Screening result Diagnosis as determined by biopsy Metric PositiveNegative Positive Pap testTrue positive (TP) False positive (FP) TP/ TP+FP= Positive predictive value Negative Pap test False negative (FN) True negative (TN) TN/ TN+FN= Negative predictive value Metric TP/ TP+FN= Sensitivity TN/TN+FP= Specificity Cytologic-Histologic Calculation © 2011 College of American Pathologists. All rights reserved. 25

26. There are few or no correlations for negative Pap results False positive Paps over-represented PPV o A measurement close to the % of positive Pap tests correlating with biopsies, the most measured CHC metric in survey Sensitivity is preferred o Biased due to low number of negative Pap correlations PPV median = 83-88% o Range 71-94% (CAP Q-Track data, 2005-2010) Evidence for PPV © 2011 College of American Pathologists. All rights reserved. 26

27. Any biopsy interpretation and Pap interpretation, in any combination, from the “positive” list: o LSIL/ CIN1/ HPV changes o HSIL/ CIN 2, 3/ CIS o SIL, indeterminate o AIS* o Carcinoma (squamous, adenocarcinoma, NOS or other o Other malignant diagnosis ASCUS, ASC-H, AGC are excluded “Positive” Correlation © 2011 College of American Pathologists. All rights reserved. 27

28. Any normal/ negative/ reactive biopsy with a Pap result from the previous list Any NILM/ reactive/ infectious Pap test with a biopsy result from the previous list “Negative” Correlation © 2011 College of American Pathologists. All rights reserved. 28

29. A positive Pap test (from the list) with a negative biopsy o Review of both specimens may reveal that the Pap test interpretation is correct- for purposes of PPV, this remains a ‘false positive’ value o Laboratories may tabulate Pap/biopsy review results separately as a correct interpretation “False Positive” Pap test © 2011 College of American Pathologists. All rights reserved. 29

30. The PPV of a positive Pap test is the preferred standard CHC metric. A. Agree 69.84 % B. Disagree 26.98% C. Other 3.17% CS 4.41 QUESTION #40: © 2011 College of American Pathologists. All rights reserved. 30

31. Investigate cytologic diagnostic accuracy and intradepartmental variability o Group consensus slide review o Outlier individuals associated with discordance − Correlate with individual/group statistics of other monitors: − ASC:SIL ratio − Rescreen variances − Diagnostic category rates o Investigate biopsy quality o Investigate colposcopic quality Suggested approaches to Low PPV performance © 2011 College of American Pathologists. All rights reserved. 31

32. High PPV may indicate identification of only the most obvious lesions, indicating under- recognition of subtle lesions o Compare with overall laboratory/ individual abnormal rates (lower than benchmarks?) o Rescreen error rates (higher than benchmarks?) o PT performance evaluation Lack of biopsies of subtle colposcopic lesions or presence of transformation zone- discussion with care givers Suggested approaches to High PPV performance © 2011 College of American Pathologists. All rights reserved. 32

33. 1.Where possible, only one final result per patient (one pair) 2.Original interpretations used for “positive” or “negative” correlation statistics rather than the review interpretation 3.Exclude ASC-US, ASC-H, AGC, “equivocal” biopsy results 4.Maintain laboratory statistics-a minimum of 20 events required for statistical significance Difficult to collect sufficient number of events to monitor for individuals unless in high-volume laboratories CHC guidelines for statistical standardization © 2011 College of American Pathologists. All rights reserved. 33

34. Laboratories should use the positive predictive value (for the whole laboratory) to formulate QA monitors. A. Agree 65.22% B. Disagree 28.99% C. Other 5.80% CS 4.42 QUESTION #41: © 2011 College of American Pathologists. All rights reserved. 34

35. CHC Consensus Statement #5 When there is a negative biopsy with confirmed HSIL+ Pap test, the caregiver is notified of the discrepancy resolution in a timely manner.

36. Optimizes patient care o Minimizes over-emphasis of biopsy findings o Prevent unnecessary additional procedures Documentation of notification may be in the biopsy report, cytology report or QA document Justification: Professional opinion, survey, literature o Of 53 cases of negative biopsy/SIL Pap test, 24 (45%) were found to have subsequent SIL- indicating necessity to follow up women with discordant pairs* Consensus Statement #5: When there is a negative biopsy with confirmed HSIL+ Pap test, the caregiver is notified of the discrepancy resolution in a timely manner. © 2011 College of American Pathologists. All rights reserved. 36 * Anderson and Jones. False positive CV cytology-: follow-up study, Acta Cytol 1997;41(6):1697-1700.

37. A negative biopsy with a confirmed HSIL+ Pap test requires caregiver notification. A. Agree 92.19% B. Disagree 6.25% C. Other 1.56% CS 4.5 QUESTION #42: © 2011 College of American Pathologists. All rights reserved. 37

38. CHC Consensus Statement #6 Laboratories should attempt to obtain correlation biopsy information for all patients with a HSIL Pap test.

39. Laboratories that perform primarily Pap test interpretation may not have access to biopsy follow-up, but should ask for follow- up information in the Pap report or by other means. Laboratories should notify the care provider if a Pap test is HSIL and no follow-up information is obtained within a laboratory- defined period of time (suggest within 6 months). Laboratories should be able to show documentation that an attempt to gain follow-up information has been performed. The follow up method should be determined by the laboratory, be described in a written procedure and results documented in their QA program. Justification: Survey, professional opinion, literature Consensus Statement #6: Laboratories should attempt to obtain “HSIL follow-up.” © 2011 College of American Pathologists. All rights reserved. 39

40. CYP.07556: When a follow-up histologic report or material is not available within the laboratory, there is a documented effort to obtain follow-up histologic information for correlative review when gynecologic cases with significantly abnormal (HSIL) or malignant cytologic findings are reported. §493.1274 (c)(5)(iv): [Laboratories must keep] An annual statistical laboratory evaluation of the number of---Gynecologic cases with a diagnosis of HSIL, adenocarcinoma or other malignant neoplasm for which histology results were available for comparison. Regulatory Requirements © 2011 College of American Pathologists. All rights reserved. 40

41. 83% have written policy on how to proceed for HSIL+ Pap with no documented follow-up received (Sec 4, Table 34) 91% notify the physician’s office o Physician’s office contact time ranged from 0-45 months, average 5 months (Sec 4, Table 34) Post-survey Web-based public comments: o 75% (n=24/32) find it useful to track % HSIL+ for which follow-up is received Survey Results © 2011 College of American Pathologists. All rights reserved. 41

42. When you do not have biopsy results for an HSIL+ Pap test, the best way to obtain follow up is to send a formal request for this information to the caregiver that collected the Pap test? A. Agree 87.3% B. Disagree 12.7% CS 4.6 QUESTION #43: © 2011 College of American Pathologists. All rights reserved. 42

43. CHC Consensus Statement #7 Microscopic review of all slides from discordant Pap/biopsy pairs is desirable.

44. If slides are unavailable, the original interpretation stands as the review interpretation Laboratories may define other “non-correlation” metrics (beyond HSIL mismatch) for QA purposes Results of microscopic review should be documented by the laboratory o May be within the biopsy or cytology report o May be in a separate QA document Justification: Professional opinion, survey Consensus Statement #7: Microscopic review of all slides from discordant Pap/biopsy pairs is desirable. © 2011 College of American Pathologists. All rights reserved. 44

45. 61% laboratories address discrepancies in the biopsy report (Sec 4, Table 33) Most (71%) address NILM Pap with CIN 2,3 biopsy in report, and HSIL Pap with negative biopsy (89%) (Sec 4, Table 38) Laboratories are divided on whether to report concurrences (Sec 4, Table 34) Post-survey public comments o 77% (n=20/31) review all available glass slides for both Pap/biopsy in discordant cases o 19% (n=5/31) review all available glass slides for both biopsy and cytology in ALL cases, concordant and discordant (suggesting this is the sign-out practice) o 15% (n=4/31) review only Pap tests but annotate the concordance statistics Survey Results © 2011 College of American Pathologists. All rights reserved. 45

46. It is desirable to microscopically review all discordant pairs (as laboratory- defined) for CHC. A. Agree as stated 86.79% B. Disagree – should be required 3.77% C. Disagree – not necessary 5.66% D. Other 1.89% E. Other 1.89% CS 4.7 QUESTION #44: © 2011 College of American Pathologists. All rights reserved. 46

47. CHC Consensus Statement #8 CHC is optimum with a “multilayered” approach.

48. Multilayered, laboratory-directed approach “drills down” in potential problem areas and can be tailored to laboratory size, issues and practice Additional QA monitors may be continuous or interval efforts o Interval efforts may target specific pairs for a pre-defined period (i.e., quarterly) to acquire a “snapshot” of laboratory performance for that indicator o Continuous efforts may be desirable for laboratories with high personnel turn-over, disruptive environments, or mitigating variables outside of the laboratory’s control Justification: Professional opinion, survey, literature Consensus Statement #8: CHC is optimum with a “multilayered” approach. © 2011 College of American Pathologists. All rights reserved. 48

49. A favored activity among laboratories- learn from mistakes Leverages group experience Encourages uniformity of interpretation through consensus Survey Results (Section 6, Table 62): o 60.4% of laboratories conduct in-house review of Pap tests o 89.1% share interesting cases o 46.6% compare criteria for increased diagnostic precision Suggested Additional Quality Practices: Group Educational Review of Cases © 2011 College of American Pathologists. All rights reserved. 49

50. CHC is ideal time to investigate biopsy quality o Reorient tissue in block o Obtain additional levels o Perform ancillary studies o Record transformation zone presence/ absence Laboratories should develop trend based policies: o Number of routine serial sections o Number of levels on cervical biopsies and ECCs Openly discuss methods to improve biopsy samples © 2011 College of American Pathologists. All rights reserved. 50 Suggested Additional Quality Practices: Optimize the Biopsy

51. Record # events where subsequent actions are necessary for discrepant biopsies Record negative cervical biopsies that: o Lack transformation zone o Are less than 2 mm size or consist of only 1-2 fragments o Are not associated with additional biopsies or ECC o Are poorly oriented o Require additional levels Suggested Additional Quality Practices: Monitor biopsy characteristics © 2011 College of American Pathologists. All rights reserved. 51

52. Investigate variables adversely influencing interpretation Tabulate “both correct”- Sampling error Record false positive/ false negative Pap tests that: o Demonstrate staining or processing irregularities o Have obscuring factors o Show atrophic changes o Did not have subsequently-detected cells marked (screening variance) o Demonstrate difficult patterns of detection (“Litigation cells,” “hyperchromatic crowded groups”) (interpretive variance) Suggested Additional Quality Practices: Monitor Pap test characteristics © 2011 College of American Pathologists. All rights reserved. 52

53. LSIL Pap and HSIL biopsy ASC-H Pap and HSIL biopsy AIS / LSIL discordance LSIL/ CIN1 and negative discordances ASC-US, HPV+ and LSIL/HSIL biopsies ASC-US, HPV- and LSIL/HSIL biopsies AGC Pap and negative cervical biopsies, ECC/ EMBx HSIL Pap in pregnant patient and post-partum biopsy © 2011 College of American Pathologists. All rights reserved. 53 Suggested Additional Quality Practices: Periodically monitor other discrepancy pairs

54. Obtain additional opinion for disagreement with original interpretation Triage specific cases for automatic peer review (HSIL Pap, HSIL biopsy, AGC, disagreement between CT/Path) For retrospective review- blinded review of all Paps and biopsies, then correlate results with original and review interpretations Different individuals performing slide reviews Different CT review of Pap for discrepancies Review with original observer or in consensus conference © 2011 College of American Pathologists. All rights reserved. 54 Suggested Additional Quality Practices: Minimize review bias

55. A multi-layered approach to CHC, suited to laboratory size and staffing, optimizes opportunities for improvement. A. Agree 97% B. Disagree 3% CS 4.8 QUESTION #45: © 2011 College of American Pathologists. All rights reserved. 55

56. Working Group 4 Additional Voting Questions © 2011 College of American Pathologists. All rights reserved. 56

57. 86.For standardization of the PPV of a positive Pap test calculation, should ASCUS interpretations be included? A.Yes 23.08% B.No 61.54% C.Uncertain 15.38% Voting WG4 © 2011 College of American Pathologists. All rights reserved. 57 © 2011 College of American Pathologists. All rights reserved.

58. 87.For standardization of the PPV of a positive Pap test calculation, should ASC-H interpretations be included? A.Yes 55.10% B.No 34.69 C.Uncertain 10.20% Voting WG4 © 2011 College of American Pathologists. All rights reserved. 58 © 2011 College of American Pathologists. All rights reserved.

59. 88.For standardization of the PPV of a positive Pap test calculation, should AGC interpretations be included? A.Yes 53.85% B.No 38.46% C.Uncertain 7.69% Voting WG4 © 2011 College of American Pathologists. All rights reserved. 59 © 2011 College of American Pathologists. All rights reserved.

60. 89.Laboratories should be allowed to define the process for CHC (“real time”, retrospective, or both). A.Agree as stated 80.77% B.Disagree – not robust enough 15.38% C.Disagree – too restrictive 3.85% Formerly voting question #37. Voting WG4 © 2011 College of American Pathologists. All rights reserved. 60 © 2011 College of American Pathologists. All rights reserved.

61. 90.The correlation interval between the Pap test and the biopsy should preferably be within 3-4 months, but no greater than 6 months. A.Agree as stated 72.73% B.Disagree – not robust enough 5.45% C.Disagree – too restrictive 21.82% Formerly voting question #38. Voting WG4 © 2011 College of American Pathologists. All rights reserved. 61 © 2011 College of American Pathologists. All rights reserved.

62. 91.Laboratories should use the positive predictive value (for the whole laboratory) to formulate QA monitors. A.Agree as stated 58.18% B.Disagree – not robust enough 12.73% C.Disagree – too restrictive 29.09% Formerly voting question #41. Voting WG4 © 2011 College of American Pathologists. All rights reserved. 62 © 2011 College of American Pathologists. All rights reserved.

63. 92.Caregivers should be personally notified when review of a discordant pair shows a negative biopsy and an HSIL+ Pap reinterpreted as NILM. A.Agree 92.59% B.Disagree 7.41% Voting WG4 © 2011 College of American Pathologists. All rights reserved. 63 © 2011 College of American Pathologists. All rights reserved.

64. COMMENTS? Group 4: CHC © 2011 College of American Pathologists. All rights reserved. 64