The Progress of npcr audits What have we done, what have we learned, and where are we going now Click to edit subtitle Click to enter your Division Name here

History of Npcr audits Public Law 102-515: “Participate in an independent audit of compliance with NPCR standards, authorized by Public Law 102-515, conducted by CDC approved organization/entity. . . . . . .” Technical Assistance & Audit Data Completeness & Quality Audit Data Quality Evaluations – Version 1 & 2 Read first part of slide – then relate that today I will touch base on the last four (4) audits

Technical Assistance and Audit (TAA) 2001-2005 Assessment of Completeness and Data Quality of Female Breast Colon and Rectum Lung and Bronchus Prostate Data Years - 1998 - 2003 TAA timeframe: October, 2000 – September, 2005 – reviewing data from 1998-2002 encompassing 46 CCR’s. The primary focus of these audit activities were on female breast, colorectal, lung and prostate primaries. The “source” of these audits were an evaluation of hospital resources and processes. The first being an assessment of reporting completeness through a case-finding exercise. Pathology reports were the primary source reviewed for inclusion in the CCR level – assessing whether the patient and tumor information was reported to the state registry. If time allowed, then a review other sources – MRDI and/or clinic logs was also performed. Data quality was evaluated via re-abstracting from source/hospital records. The results of both these activities was then compared to the CCR merged information resulting in Reconciliation by the CCR to either a. explain missing cases or differences in data elements based on additional information at the CCR level or b. or the identification of cases that were not reported to the CCR

Completeness Review of MRDI demonstrated that the most often missed cases were: Hematopoietic Diseases Clinical only cases: Lung, Pancreas, Liver, and Brain/CNS Complete review of Pathology Cases revealed that: Breast Digestive were more often missed Out-patient clinics and logs had it’s own issues Completeness rates were between 95.1% and 97.7% over the five year period

Data completeness and Quality audit (DCQA) 2006-2011 Assessment of Completeness and Data Quality of ALL Sites Summary Stage and Collaborative Stage ALL Treatment information Data Years - 2004 - 2009 With the overall completeness in reporting well within NPCR standards, Casefinding was enhanced with a look at all sites. And data quality took a close look at Summary stage and collaborative stage and an evaluation of all treatment information.

Year 1 case-finding Results: Most often missed cases: Reportable Hematopoietic Diseases Non-malignant CNS Digestive Respiratory Breast Year 1 Completeness: Ranged from: 92% to 99% Average: 97% Most Common Missed: RHD, non-malignant CNS, digestive, respiratory and female breast

And we took a snap-shot look at the stage errors related to our most common sites. CS errors were an issue – with an overall error rate between 6-13% for Summary Stage. Not all the Summary Stage issues were related to errors in the data elements. We found that if a change/update had been made to a portion of CS – Extent, Lymph Nodes or Mets – sometimes SS was not recalculated.

I presented this information a while back, and you can see from this slide that Surgery information needed some additional attention. Results of our audits revealed various issues that were consistent from 2004 – 2006.

NPCR Data Quality Evaluation 2012 - 2016 Focus: Data within the Central Cancer Registry (CCR) for Breast, Colon/Rectum, Lung, Corpus Uteri and Prostate Purpose: Data Completeness and Quality Identify Data Limitations Identify Training Needs: Data Reporters Central Cancer Registry Staff In 2012, NPCR audits took a new focus. The purpose was to evaluate the completeness and quality from the data within the CCR for the major sites. The results help NPCR understand some of the CCR procedures regarding data completeness and overall data quality, as well as to understand some of the data limitations - either due to the accuracy or the completeness of the information that is recorded at the source level and subsequently the summary information that is ultimately used in research As well as to identify the training needs – both of the data reporters as well as the central cancer registry staff.

NPCR Data quality evaluation - dqe: Process: Visual Review – Source Level Manual Consolidation Comparison to CCR Merged Data Evaluation of QC Proceduress Power Point Presentations Ascertainment of Treatment Data: Breast Colorectal The process involves a visual review of the abstract information; manual consolidation of multiple abstracts with a comparison to the CCR merged data. WE also did a review of the CCR QC procedures in comparison to CDC-NPCR recommended guidelines and provided a summary of findings to the CCR via a written report and a power point presentation. We also run clinical checks on breast and colo-rectal primary cases; The primary purpose of the Clinical Check edits is to evaluate reported prognostic and treatment items for cancer cases with specific tumor characteristics. For example – was the required prognostic/staging information reported for the breast and colon primaries If the reported treatment does not appear to be consistent with widely recognized standards of care or cases fail to contain known prognostic characteristics, a warning is generated.

Common Issues: Required Data Elements: Consolidation: Is there text? Is it accurate? Are dates included in text? Consolidation: Was it done? Was the source reviewed? The NPCR Data Quality Evaluation reviewed the “source information” in the CCR – rather than reviewing the records at the hospital level – through a visual review of required data elements – including TEXT DOCUMENTATION. Text is a critical piece related to quality and validates whether or not each source record or abstract contains the necessary/correct information – to validate the code. And if there is text – does it correspond to the coded data value? Are dates included in the text fields to validate items like the date(s) of surgery, chemotherapy and/or radiation therapy so a researcher could determine if patient received adjuvant tx? Text is evaluated with the coded data values – for accuracy and completeness from the reporting source. Because if the pieces of the puzzle are not correct – it could impact the outcome. Let me show you some of the common errors related to stage that we encountered – next slide

Common issues: Tumor Size: 999 vs. Specific Value Using the Correct Report Lack of Documentation Size Eval: Applying Correct Rules Update/Consolidation Neoadjuvant therapy Let me start with Tumor Size: we found that it was coded to unknown or to 990/991 – C50 – when there was a specific size available in the text fields. There were also issues regarding whether the size should come from the imaging, the physical exam or the pathology specimen – when all three were available. And our favorite – there were often times where we could find no text to validate the code that had been entered. And related to Tumor size – the Size Eval field also had data quality issues. While this may have been a CS data item, and we know that CS is no longer required by NPCR– the correct coding or knowledge related to this data item results in the correct assignment to either a clinical or pathologic category. We found issues related to applying the rules – bx vs. resection; or often this item wouldn’t get updated when additional information was added. And we saw data quality errors when patient received neoadjuvant therapy and the clinical size should have been used instead of the pathologic size.

Data Item Accuracy Data Item Accuracy 95% CI Minimum Maximum Primary Site 99.90% 99.8%--100.0% 99.00% 100.00% Histologic Type 98.30% 97.8%--98.7% 94.30% Behavior 99.70% 99.5%--99.9% 97.90% Grade 94.50% 93.8%--95.3% 84.60% 99.20% Laterality 98.90% Derived Summary Stage 2000 96.10% 95.5%--96.8% 90.30% This slide is for all sites combined. Overall, the quality of the tumor-specific data items was very good. However accurate information re: tumor grade had remained an issue

Data Quality Evaluation – Version 2 Data sources Consolidated record level file - Diagnosis years 2008 – 2014 Focus: Select cancer sites: Abstract record level file Urinary Bladder* Accuracy & Consolidation Melanoma of skin* Colon Policy and Procedure Manual Lung Female Breast Multiple Primary Rules Prostate NPCR is currently in the process of completing the Data Quality Evaluation – Version 2 with our states. The cancer sites that are included in the data item review has changed slightly where you see that we added bladder, melanoma, to the original sites. The overall focus is again a review of the central cancer registry’s source information – the abstracts that make up the summary information – for accuracy of text to coded values and an evaluation of the consolidation process/logic as well as a review of cases related to the application of the multiple primary rules – thus this is why bladder and melanoma of skins were added to the mix.

Methodology Reconsolidation process MPH Rules Assess data agreement and reproducibility Consistent interpretation and abstracting Estimate agreement rates Identify training opportunities MPH Rules Determine agreement and consistency in applying rules During the reconsolidation process, data were assessed for agreement and reproducibility – to determine the consistency in interpretation and abstracting, estimate agreement rates, and identify training opportunities (if needed). The sample size included 438 cases from each state, depending upon data availability. Then, the multiple primary and histology rules were applied to determine agreement and consistency in applying the rules. A sample of records with a sequence number >01, for all sites, was selected and each tumor for that patient was reviewed.

Data Quality Factors Use most accurate or specific information DQE – Version 2 results shows Consistently, each CCR’s identified errors are related to these issues: Most accurate or specific information wasn’t used to assign the proper code; e.g., date of diagnosis, primary site, histology An unknown or less specific code was recorded when text provided information for the known or more specific code; e.g., incorrect grading system used or highest grade not assigned; specific or combination histology information was available Text information was lacking to support the assigned code Information from a metastatic site rather than the primary site was used to assign codes; e.g., histology, behavior, grade; subsequent primary assigned for metastatic disease Important to code the full extent of the stage; e.g., LN involvement or metastatic disease was missed, progression of disease was recorded as mets, incorrect behavior assigned which directly affects stage Subsequent treatment was recorded as first course; when using COC treatment guidelines, failed to record date of death as reason no treatment; no treatment and active surveillance are considered treatment; diagnostic procedures were recorded as definitive treatment or were used to assign dates of first course. Use most accurate or specific information Code known over unknown or less specific Document codes with text Primary site vs. metastatic site Stage Treatment First course treatment vs. subsequent treatment No therapy/Active Surveillance Diagnostic vs. definitive procedures

Multiple Primary Factors Primary site Synchronous tumors in paired organs Metastatic disease Progression of disease Same histology Coding error Non-reportable histology Ambiguous terminology Sequence number Factors associated with the multiple primary issues consistently related to assigning the primary site; e.g., 4th digit for colon cases, lesions in both breasts, metastatic site or progression of disease recorded as new primaries, same histology w/n or beyond established timeframe (colon >1 yr apart, lung w/ same laterality/histology w/n 3 yrs). Coding errors in either the site or histology lead to incorrect disposition of multiple records. Histologies without a definitive diagnosis that includes “malignant” are not reportable but were assigned as a new primary. An ambiguous term was used to assign a new primary; e.g., suspicious cytology for bladder. Sequence number didn’t reflect the tumors over the patient’s life; e.g., h/o previous cancer.

This infographic provides a snapshot of the 2014-2015 evaluation year for all programs combined. You may have trouble reading this but I wanted to show it again. In the right hand corner you can see the errors by primary site, lung having the most errors, 24%. In the bottom left hand corner you see the errors categorized. Stage and treatment have about equal percent of errors. And in the bottom right, it shows the data elements with lowest accuracy proportions. Overall a 97% accuracy rate for the ten registries that participated is a job well done. I just want to thank Dylan Holt from Westat for putting this together for us.

For year 2 of this DQE, all CCRs combined, the data quality and the multiple primary accuracy rate is 97.9%. Treatment information contributed the most to the errors identified. Ten CCRs were evaluated contributing >4,000 reconsolidated cases with >93,000 data items reviewed and consolidated. The MPH sample contained >8,500 tumors from >3,800 patients. For the sites shown on this slide, other sites had the lowest agreement for the MPH evaluation followed by colon. I’ll show another slide later that breaks this down further. For the data quality portion, melanoma had the highest agreement while bladder had the lowest.

NPCR DQE Plan for 2020-2023 Three focus areas: Completeness of staging, biomarker, treatment, and treatment- related information Validation of staging, biomarker, treatment, and treatment-related information Evaluation of duplicate rates Soon we will be launching our next version of the NPCR Data Quality Evaluation. It will run 2020 – 2023 and will focus on: READ SLIDE

Thank you Mary Lewis, CTR Thank you for this opportunity to share with you the progress of the NPCR Audits. Click to enter your Division Name here