Evaluation of Summary Stage Data Quality in the CDC-NPCR Cancer Surveillance System Reda Wilson Qiming He Cheryll Thomas Jessica King Karen Ledford Christie Eheman

Discussion Points Describe Summary Stage issues Describe effect of Collaborative Staging Provide evidence for quality control Provide evidence to re-derive stage prior to use To maximize the quality of data maintained by CDC’s National Program of Cancer Registries, NPCR has implemented activities to more fully evaluate data quality to enhance data utilization in comprehensive cancer control activities. I will attempt to describe issues identified in Summary Stage and the effect of the Collaborative Staging system and provide evidence for carrying out quality control activities and re-deriving/recalculating stage prior to data utilization.

Background Data quality definition Assessment and surveillance goal Quality is defined as an encompassing term comprising utility (usefulness), objectivity (accuracy, clearness, completeness, unbiased), & integrity (security, confidentiality). Through ongoing strategic planning, CDC’s Div Ca Prev & Control identified assessment & surveillance as an important division attribute. DCPC’s overarching assessment & surveillance goal is to provide for & promote the use of high quality data to monitor the cancer burden & guide cancer control planning & policy. Data quality is integral to every step in the development of that data, including creation, collection, maintenance, & dissemination. Prior to providing & promoting the use of NPCR data, these data must be reviewed & categorized as to their quality, completeness, & suitability. Variables must also be identified which require additional review & assessment before use, or which require extensive evaluation. In one DQ activity, NPCR eval summary stage data as reported to the NPCR Ca Surv System. Data quality definition Assessment and surveillance goal Provide and promote use of high quality data To monitor cancer burden Guide cancer control planning and policy Data quality activities Review data Categorize quality, completeness, suitability

Objective Evaluate quality of CDC-NPCR CSS Summary Stage Identify potential solutions assuring data quality Confirm suitability of stage information The objective was to evaluate the quality of summary stage data in the NPCR-CSS and to identify potential solutions to assure data quality so that stage information may be used in cancer control activities.

Methods NPCR data 2007, 2008, 2009 submissions Data from dx yrs 2001-2005, 2001-2006, & 2004-2007 from the NPCR-CSS 2007, 2008, & 2009 submissions, respectively, were analyzed. As appropriate, NPCR pgms were excluded from some eval due to a lack of information. Comparisons were made between submissions to identify areas that had improved & areas where continued improvement was needed. The proportion of unstaged invasive cases was eval by state over time & included directly-coded SS00 & Derived SS00. Eval were performed for all sites combined, female breast cancer, & colorectal cancer, including & excluding cases identified through DCO or pathology only. The dataset for all sites combined was evaluated with & without unknown primary site. The DCO, path only, & unknown primary site cases were excluded as an unknown stage is the appropriate code for these cases. The female breast & colorectal datasets were eval by race. Additional eval by sex was performed for the colorectal dataset. Comparisons were carried out between reported Derived SS00 & recalc Derived SS00. Datasets excluding younger age groups were compared to datasets including all ages. Another eval was conducted comparing behavior & stage using the 2008 & 2009 datasets, & included all NPCR-funded programs. Where discrepancies were found between the invasive beh & in situ stage, the CS Ext data item & appropriate edits were eval further. In situ beh cases were included in this dataset to identify discrepancies between in situ beh & stage other than in situ. NPCR data 2007, 2008, 2009 submissions Evaluations of unstaged invasive cases Evaluations by site, race, sex Derived Summary Stage 2000 recalculated Evaluations by behavior and stage

Results1 Proportion unstaged cases Comparison 2007 to 2008 submissions Overall, a decrease in the proportion of unstaged cases was found in dx yrs 2004 & 2005 (Derived SS00) as compared to dx yrs 2001-2003 (directly-coded SS00), as we would expect. For most NPCR-funded programs, there were minor differences in the proportion of unstaged cases between submissions. However, some outliers were identified. One NPCR program had a 3-4 point increased difference in the proportion of unstaged cases from the 2007 submission to the 2008 submission. Another NPCR program had a significant increase in unstaged cases in dx yr 2004, in the 2007 submission, but not in the 2008 submission. And a third NPCR program was found to also have a significant increase in the proportion of unstaged cases in dx yr 2004, in the 2007 submission, that carried over to the 2008 submission. We found, from the NPCR DC&Q audit results, that this pgm does not receive CS data items from their non CoC hosp. This may be the contributing factor for the large proportion of unstaged cases in this state. When the 2009 dataset was evaluated for the proportion of unstaged cases, there were minimal differences between the 2008 & 2009 datasets, with improvement in proportions for 2 of the 3 programs just described. This same evaluation was done for the reported & recalc unknown stage & showed a slight drop for unknown in the recalc dataset. Proportion unstaged cases Comparison 2007 to 2008 submissions Decreased proportion unstaged 3 programs as outliers Comparison 2008 to 2009 submissions Minimal differences proportion unstaged 1 program as outlier

Results2 Unstaged breast and colorectal cancer cases Overall, a higher proportion of unstaged breast ca cases were found in the 2008 submission when younger patients were excluded but DCO & path only cases were included. Usually, a large difference was not seen, with only one NPCR program showing an increase greater than 3 points in the 2004 dx yr & 2 points in the 2005 dx yr. A similar pattern was found in the unstaged colorectal ca cases. The proportion of unstaged female breast ca cases by race approximated the individual states’ racial distribution. The proportion of unstaged colorectal ca cases by race also approximated the individual states’ racial distribution in all but one NPCR program. In this program, there was almost a 50% decrease in unstaged colorectal cases in the white population while there was almost a 30% increase in unstaged colorectal cases in the black population. Further stratification by sex showed no change in the racial dist for this pgm. Racial distributions were relatively equal between directly coded SS00 & derived SS00. Unstaged breast and colorectal cancer cases Increased proportion older patients Race distribution similar to state 1 program as outlier

Results3 In the 2008 submission, 193 cases were identified with an invasive behavior and an in situ stage. Of these, the edit “CS Extension, Primary Site, Behavior ICDO3” allowed this configuration to incorrectly pass for the sites stomach, small intestine, and penis (representing 23% of the identified cases). The edit should have failed in 40% of the identified cases. The remaining 37% of identified cases would have passed the edit (invasive behavior with invasive extension code) but appeared to not have had the Derived SS00 corrected. These cases were found to have been submitted by 23 NPCR programs, with the majority (70%) reported by two programs.

Results4 An additional 1,100 cases were identified with an in situ behavior code and a Derived SS00 other than in situ; 671, 764, and 723 in situ cases were reported with a stage other than in situ for diagnosis years 2004-2006 respectively. After recalculating the derived SS00, these cases fell to 22, 185, and 55 for each diagnosis year respectively. 92% of the reported cases were coded as unstaged. An overwhelming majority (84%) of these cases were submitted by a single NPCR program. The CS algorithm was run on the full 2008 and 2009 NPCR-CSS datasets and the discrepancies between behavior and stage decreased dramatically. A review of the primary sites showed these to be consistent over each year. This problem was consistently identified in the same 3 states in both submissions.

Change in Re-Derived Summary Stage Groups for 2004, 2005 Diagnosis Years NPCR-CSS 2008 The stage group dist changed for 18 pgms after Derived SS00 was recalc. The biggest change, as you can see, occurred in 3 pgms. State 6 had an increase in all stage groups. State 13 had a decrease in in situ & local stage groups with an increase in regional & distant stage groups. State 16 had a decrease in regional & unknown stage groups with an increase in local & distant stage groups. For the 2005 dx yr, in the 2008 dataset, there was a shift in the stage group distributions for 19 NPCR pgms. As you can see, the shifts were less for most programs. However, the 3 states with the greatest change in the 2004 data are the same states with the greatest changes in the 2005 data. State 5 had an increase in all stage groups. State 11 had a decrease in in situ & local stage groups with an increase in regional & distant stage groups. State 16 had a decrease in regional & unknown stage groups with an increase in local & distant stage groups. These are the same shifts seen in the 2004 data. There were 13 pgms with shifts in both 2004 & 2005 dx years.

Change in Re-Derived Summary Stage Groups for 2004, 2005, 2006 Diagnosis Years NPCR-CSS 2009 As you can see, the shift in stage group distribution was minimal in the 2009 NPCR-CSS dataset, though 8 programs had shifts in each of the 3 years under evaluation. The states in the 2008 NPCR-CSS dataset with the largest shift are the same in the 2009 NPCR-CSS dataset, though with less change.

Change in Re-Derived Summary Stage Groups by Primary Site for 2004-2006 Diagnosis Years NPCR-CSS 2009 Unknown stage group changed in more primary sites than other stage groups, followed by local & regional. The highest change in distant stage group occurred in dx yr 2006; in the 2008 NPCR-CSS dataset all sites had changes for dx yr 2004 & all sites but ureter in dx yr 2005; in the 2009 NPCR-CSS dataset 8 primary sites had changes in each year 2004-2006 with the 4 highest shown here, 12 sites had changes in 2004, 10 in 2005, & 15 in 2006.

Number of Cases Moving between Re-Derived Summary Stage Groups NPCR-CSS 2008 and NCR-CSS 2009 In the 2008 NPCR-CSS dataset – 2,775 cases moved out of not applicable, unstaged, and blank and into specific groups In the 2009 NPCR-CSS dataset – 806 cases moved out of unstaged, blank, and in situ and into specific groups

Conclusions Data processing delays Incorrect edit check The decreased proportion of unstaged cases from one NPCR-CSS submission to the next may indicate delays in processing or receiving data so that the appropriate stage could be assigned. The overall decrease in unstaged cases from diagnosis years 2004+ as compared to diagnosis years 2001-2003 appears to indicate more appropriate stage assignments are derived from the CS schemas, one of the reasons for this system. Based on information from these evaluations, the edit “CS Extension, Primary Site, Behavior ICDO3” did not identify all cases which should fail this edit. That information was forwarded to the AJCC CS Task Force and the edit ‘tightened’ to prevent these errors in the future. Also based on these evaluations, it appears that central registries, and possibly hospital registries, are not recalculating SS00 after changes have been made to either the behavior or CS extension codes. The CS algorithm contains a feature to automatically recall the program when a change has been made to a data item used in the calculation. However, cancer surveillance software developers have the option of not utilizing this feature, which appears to have occurred. Data processing delays Incorrect edit check Insufficient use of CS algorithm

Anticipated Outcome Implement processes to assure quality Revised edit New edit NPCR-CSS SOP change We have previously reported that quality measurement provides both a means and a justification for training, additional resources, and documenting high quality data for use in comprehensive cancer control efforts. This continues to hold true for these evaluations, which shows that additional training is required to assure data quality of the entire abstract once failed edits are corrected. These evaluations resulted in a correction to the edit “CS Extension, Primary Site, Behavior ICDO3”, development of a new edit “Derived SS2000, Behavior ICDO3”, and a change to NPCR-CSS SOP to recalculate Derived SS00 during dataset preparation.

Recommendations NPCR recommends that its programs take advantage of the CS dll that requires the derived fields be recalculated whenever a change is made to individual CS data items and that registries routinely recalculate Derived SS00 prior to using or releasing this information. Collaborative Stage .dll Routinely re-calculate Derived SS00

Summary Points Described effect of Collaborative Staging Identified Summary Stage issues Provided evidence for quality control Provided evidence to re-derive stage prior to use These evaluations showed the effect of CS, with decreases in unstaged cases; identified issues with Derived SS00, with the algorithm appearing to incorrectly assign stage for some sites; provided evidence for enhancing quality control efforts, with improvement in available edits; and provided evidence that stage should be recalculated prior to use. So, where do we go from here?

NEXT STEPS Data quality plan Data use plan All fed agencies, under the Paperwork Reduction Act, are directed to dev info resources mgmt proc for reviewing & substantiating the quality of info before it is disseminated. To assure that this directive is met, NPCR is developing a plan for systematic review & quality check or analyses of NPCR-CSS variables. This plan includes a description of needed resources for quality review & improvement including staff & methods for data eval. Using information obtained through these reviews, NPCR will dev an appropriate plan for data use. This process enables NPCR to substantiate the quality of data available for use &/or dissemination. The 1st step in this process is the establishment of NPCR’s Data Evaluation, Enhancement, & Promotion Activity (DEEPA). DEEPA is a new org unit with ongoing support from each CSB team. The focus of this activity is to assess & promote NPCR-CSS data by assuring appropriate documentation is in place, reviewing DQ, developing & implementing a data use & DQ plan, & integrating processes for creating & documenting internal & external data sets. A guiding principle is to eval & use currently avail data. NPCR continues to work with pgms to assist in opportunities for improving DQ, such as those described in this eval, & to provide feedback whenever eval are conducted. The specific NPCR pgms will be contacted regarding the results of this eval so that they may address & implement appropriate quality control activities. Data quality plan Data use plan Data Evaluation, Enhancement, and Promotion Activity Continue work with programs Provide feedback

Reda J. Wilson, MPH, RHIT, CTR dfo8@cdc.gov 770-488-3245 http://www.cdc.gov/cancer/npcr/index.htm The findings and conclusions in this report are those of the author(s) and do not necessarily represent the official position of the Centers for Disease Control and Prevention.