1. 1 LRT - MEASURES OF DATA QUALITY AND RELIABILITY Jennifer Shafer LMI support to ODASD(SCI) 1

2. 2 Why is data quality important?  Quality data fuels quality analysis.  Data quality issues open the Department to criticism. –GAO has a number of reports critical of data accuracy and reliability in the DoD: DoD High-Risk Supply Management, Financial Management, and GAO-07-780. –To address those and similar criticisms to the monitoring of supply chain metrics, a total data quality management process is needed.

3. 3 What quality characteristics are in DoD Guidelines on Data Quality Management? Data Quality Characteristics Description AccuracyA quality of that which is free of error. A qualitative assessment of freedom from error, with a high assessment corresponding to a small error CompletenessCompleteness is the degree to which values are present in the attributes that require them. ConsistencyConsistency is a measure of the degree to which a set of data satisfies a set of constraints. TimelinessAs a synonym for currency, timeliness represents the degree to which specified data values are up to date. UniquenessThe state of being the only one of its kind. Being without an equal or equivalent. ValidityThe quality of data that is founded on an adequate system of classification and is rigorous enough to compel acceptance. The following table on the DoD core set of data quality requirements is relevant to a discussion on data accuracy and reliability.

4. 4 What is the Total Data Quality Management (TDQM) approach? Drawn from the DoD Guidelines for Data Quality Management, the following table layouts a TDQM process. ActionDescriptionActivities DefineIdentify data quality requirements and establish data quality metrics Scope problem Identify objectives Identify and review documentation Develop quality parameters/metrics MeasureMeasure conformance with established business rules and develop exception reports Apply business rules/metrics Flag suspect data AnalyzeVerify, validate, and assess the causes for poor data quality and analyze opportunities for improvement Identify conformance issues Provide recommendations Prioritize conformance issues Validate conformance issues ImproveSelect data quality improvement opportunities that provide the most benefit and implement the selected improvements. Improving data quality may lead to changing data entry procedures, updating data validation rules, and/or use of DOD data standards to prescribe a uniform representation of data that is used throughout the DOD. Select improvement opportunities Implement improvements Document improved quality Update DoD data standards

5. 5 What is the Supply Chain Metrics Group doing to support data quality?  Members aim for on-time data call submission.  Components check their computational rules against the rules in the metrics guide to see if they conform.  Where Component metrics differ in their content, the SCMG discusses the standardization of those metrics.  Members support TDQM efforts within their Components.  We can check transactional data (i.e., LMARS, SDDB, CWT, line item strat files) data sets and develop metrics to ensure: –Accuracy – what data elements have correct values (e.g., requisitions with transmission to ICP dates after material release order dates) –Completeness – what data elements are missing values (e.g., number of blank fields) –Validity – what data elements have values outside of established formats and domains (e.g., unauthorized MRA discrepancy codes) –Consistency – what data elements have values outside of their norms (e.g., required delivery dates before the requisition date established)

6. Pre-Decisional Working Paper Accuracy Quality characteristicAccuracy DefinitionAssessment of freedom from error. Proposed measureDates are sequential ≥ 95% Data elementsSERIAL_DATE REQ_BIRTH_DATE REQ_TRAN_DATE Method of computation(Count of records where SERIAL_DATE, REQ_BIRTH_DATE, and REQ_TRAN_DATE are sequential) divided by (total records) Performance over 13 months 95.8% 2 of 13 months under goal 6

7. Pre-Decisional Working Paper Completeness Quality characteristicCompleteness DefinitionDegree to which values are present in the attributes that require them. Proposed measureCompletion rates of required fields are ≥ 95% Data elementsCUSTOMER_CODE ICP COS_CODE AREA_CODE DODAAC CONUS_OCONUS COCOM_IND Method of computation(Count of non-null records for each data element) divided by (total records) Performance over 13 months 100% 0 of 13 months under goal 7

8. Pre-Decisional Working Paper Consistency Quality characteristicConsistency DefinitionDegree to which a set of data satisfies a set of constraints. Proposed measureCounts for data element subsets fall within ± 25% of the preceding 13 month average Data elementsCOS_CODE AREA_CODE CONUS_OCONUS Method of computation(Average monthly count for data subset over 13 months) minus (monthly count for data subset in current month) divided by (average monthly count for data subset over 13 months) Performance over 13 months For May 2016 data, 4 subsets fell outside of ± 25% of the preceding 13 month average: COS_CODE 8 COS_CODE U COS_CODE 5 COS_CODE 7 8

9. Pre-Decisional Working Paper Timeliness Quality characteristicTimeliness DefinitionDegree to which data values are up to date. Proposed measurePercentage of late recorded records processed in a month Data elementsCUST_RECEIPT_DATE CREATED_ON_DATE Method of computation(Count of records closing in a prior month) divided by (total records reported) Performance over 13 months 24.6% 9

10. Pre-Decisional Working Paper Uniqueness Quality characteristicUniqueness DefinitionThe state of being the only one of its kind. Proposed measureRecords with a requisition record number that is not a duplicate = 100% Data elementsDOC_NR DOC_NR_SFX Method of computation(Count of records where DOC_NR and DOC_NR_SFX are not duplicates) divided by (total records) Performance over 13 months 97.1% 5 of 13 months under goal 10

11. Pre-Decisional Working Paper Uniqueness Quality characteristicUniqueness DefinitionThe state of being the only one of its kind. Proposed measureNumber of records that were replaced with updated information Data elementsDOC_NR DOC_NR_SFX Method of computation(Count of existing records that were replaced with a newer record with the same DOC_NR and DOC_NR_SFX combination ) Performance over 13 months 2,967,780 (8.6% of all records) 11

12. Pre-Decisional Working Paper Validity Quality characteristicValidity DefinitionPresence of an adequate system of classification that is rigorous enough to accept. Proposed measureValidity rates for fields using defined codes are ≥ 97% Data elementsCUSTOMER_CODE ICP AREA_CODE CONUS_OCONUS DEPOT IMD_CORP_FILL FMS_G_R_CODE COCOM_IND MRAD_DISCR_CODE Method of computation(Count of records where the code is valid against the comparison list) divided by (total records) Performance over 13 months 97.3% 2 of 13 months under goal 12

13. 13 What are the next steps?  Additional analysis based on feedback from the Supply Chain Metrics Group  Finalize and institutionalize the measures  Collect and monitor the measures monthly