Assisting Federal Managers to Gather Reliable Data on Their Performance Measures Herbert M. Baum, PhD Andrew Gluck, MBA

Introduction “A budget is more than simply numbers on a page. It is a measure of how well we are living up to our obligations to ourselves and one another.” – President Barack Obama, with 2010 budget submission

OMB Circular A-11 “Agencies should have in place or begin to develop verification and validation (V & V) techniques that will ensure the completeness and reliability of all performance measurement data contained in their annual performance plans and reports.” Page 8 of Section 230

A-11 (continued) Verification and Validation factors to consider: 1.Standards and procedures 2.Data entry and transfer 3.Data integrity 4.Data quality and limitations 5.Oversight and certifications

Standards and Procedures a.Source data are well defined, documented; definitions are available and used. b.Collection standards are documented/available/used. c.Data reporting schedules are documented/distributed/followed. A-11 P 9 of Section 230

Standards and Procedures (II) d. Supporting documentation is maintained and readily available. e. Collection staffs are skilled/trained in proper procedures. A-11 P 9 of Section 230

Statement of the Problem How can agencies that do not have a good grasp of data-quality issues provide reliable data for decision making?

Solution Overview Background on the metric Increase the number of domains (factors) to make them more specific Provide yes/no questions to allow generation of quality score

Metrics Background Strategic goal Objective Measure number and description Program area Data source Data collection methodology Data owner

Domains Assumptions Validity Reliability Timeliness Accuracy Integrity Data not available Limitations Certification

Data Accuracy 1. Is there a method for detecting missing data? 2. Is there a method for detecting duplicate data? 3. Is there a method for detecting inconsistent data?

Data Accuracy (II) 4. Are the data checked by comparing against original source information after entry? 5. Are data verified by rechecking calculations? 6.Are edit checks built into automated data collection systems so that errors relating to internal inconsistency can immediately be detected and avoided?

Lessons Learned Build quality in by offering repeated clear guidance and training Recognize that staff turnover and competing priorities increase probability of errors creeping in Automate edit checks in the field and at headquarters Make sure program staff, who understand data, see the data that technicians are submitting

Conclusions Use Verification and Validation Checklist as time and resources allow to build in quality Inspect quality by making sure that data are regularly checked to test for outliers and internal inconsistencies

For More Information Herbert Baum   Andrew Gluck  