Reasons for Rework in Clinical Data Provisioning: A Root-Cause Analysis Amanda L. Do, MPH1,2, Ruby Y. Wan, MS1,2, Robert W. Follett1,2, Marianne Zachariah1,2, and Douglas S. Bell MD, PhD1,2 1Department of Medicine, UCLA David Geffen School of Medicine, Los Angeles, CA 2UCLA Clinical & Translational Science Institute, Los Angeles, CA Accelerating Discoveries Toward Better Health Secondary use of electronic health record (EHR) data in clinical research has become more prevalent with the deployment of large-scale EHR systems. Challenges frequently arise in repurposing clinical data for research, and many investigators are unable to relate their needs to the existing clinical data. To address these challenges, the UCLA Clinical & Translational Science Institute’s (CTSI) Informatics Program (IP) has implemented, and continues to refine, a data provisioning service model. The model includes consulting and programming services to assist investigators in defining EHR data requests that address their research questions. However, investigators sometimes find that a dataset does not meet their needs only after it has been provided, which costs the IP additional time and resources to remedy. The IP seeks to understand the reasons for this kind of rework in an effort to identify and rectify their root causes. IP hopes to use the findings from this analysis to further enhance the service model and minimize future rework. IP acts as the storefront for provisioning healthcare-related datasets for research projects at UCLA: Provide initial consultation (orient the investigators to the request process and ask them to fill out the data request form) Data request form: standardized template with two components 1) definition of patient selection criteria and 2) selection of demographics and other output variables e.g. diagnoses, lab results, etc. Review, analyze and optimize data request to ensure institutional approvals (IRB and Compliance) and technical feasibility (data requested is attainable from the existing EHR) Explore data sources, program and test data extraction code, quality check data Data source: UCLA’s Integrated Clinical and Research Data Repository (xDR), a clinical data warehouse of UCLA’s CareConnect (Epic) EHR data, linked with data from older legacy systems and other sources (e.g. Transplant Registry) Communicate with the investigators when clarification is needed Deliver data through secure, compliant methods Data requests were considered complete once all template variables were queried, exported, and delivered to the investigators. Study Data 165 data requests completed between March 2014 and February 2017 Reviewing Rework Requests Two IP data consultants reviewed all completed data requests and associated materials: Data request form filled out by the investigator and confirmed by IP staff Consulting and programming notes Project management system records (REDCap and JIRA) Email correspondence with the investigator or amongst IP staff Institutional Review Board (IRB) applications, approval notices, and other related study materials Additional files sent by the investigator e.g. study protocols, recruitment materials Defining Rework and Assigning Rework Reasons Rework was defined as any instance in which the original query needed to be revisited and modified after the original dataset was extracted and delivered. Based on previous work and experience, IP consultants classified reasons for rework into 6 categories (Table 1). Each request was then assigned any one or combination of the 6 reasons by IP consultants and programmers. Calculating Rework Cost Total hours logged by consultants and programmers for each rework request were obtained from project management system records Consultants and programmers confirmed, out of the total number of hours, the time spent on each rework reason. Requests could have more than 1 rework reason. Cost was then calculated for each request based on IP’s current hourly service rates Introduction Results Distribution of Rework Out of 165 data requests completed between March 2014 and February 2017, 39 requests (23.6%) required rework. 16 out of the 39 rework requests (41.0%) had only 1 cause for rework. The remaining 23 requests (59.0%) had more than 1 rework reason: 14 requests with two reasons, 3 with three or four reasons, 2 requests with five reasons, and 1 request with all six possible rework reasons. By year, there were 6 rework requests in 2014, 22 in 2015 and 11 in 2016. Hours and Cost Spent on Rework Total time spent on rework: 795 hours Total cost of rework (based on current hourly rates): $102,000 Adjustment of patient selection criteria was the greatest cause for rework, accounting for ~446 hours and a cost of ~$57,800 (Figure 1, Figure 2) Total time spent on original rework requests (prior to any rework): 1,268 hours Total cost of original rework requests (prior to any rework): ~$149,400 Average cost of original rework request (prior to any rework): ~$3,800 Average cost per any request (with or without rework): ~$2,600 Overall, most investigators were able to use the data provisioned through the UCLA CTSI IP service model without needing to request revisions. Nonetheless, substantial resources ($102K) were still spent on rework, costing the equivalent of 68% of the original data extraction expense. Adjustment of patient selection criteria and additional data variables have imposed the most significant strains on IP resources, accounting for ~80% of all rework. The root causes of rework could be insufficient preparation on the part of the investigators in defining the study scope and planning data analysis or lack of careful upfront analysis of the research plan by IP. However, the requests that ended up needing rework may have been inherently more complex, e.g. patient populations harder to define, as evidenced by the higher average cost ($3,800) compared to that ($2,600) of all requests, regardless of rework. To reduce the occurrence of rework, IP has continually worked to improve the service model to ensure patient selection criteria and output variables will adequately answer the investigators’ research questions. This includes fine-tuning the data request form to make it more comprehensive, e.g. addition of descriptive footnotes. IP has also implemented a more rigorous scientific review during initial consultations. The project described was supported by the National Center for Advancing Translational Science through UCLA CTSI Grant UL1TR001881. Data Provisioning Service Model Methods Figure 1: Total Hours of Rework, by Reason Figure 2: Cost of Rework, by Reason Discussion & Conclusion Table 1: Rework Reasons and Counts Acknowledgments