1. Using Administrative Data for Federal Program Evaluation
Stephanie Shipman and Valerie Caracelli
U.S. Government Accountability Office
American Evaluation Association conference
October 2016
For more information, contact

2. Background
U.S. Office of Management and Budget has been encouraging agencies to use existing administrative data to conduct more rapid, low-cost evaluations and increase evidence on what works
FY2017 Proposed Budget acknowledged challenges and included proposals to improve access to and quality of agency datasets
OMB encouraged: See memoranda, FY2016 and 2017 Budget guidance
FY 2017
Build Census Bureau data access, data linking, and analysis capacity
Help states build integrated longitudinal human services data systems
Facilitate federal researchers’ access to state employment and earnings data
Increase data quality in states’ education and workforce data

3. Background (cont’d)
In March 2016, Congress established an Evidence-Based Policymaking Commission (PL ) to examine how federal administrative and survey data can be made available to facilitate program evaluation, continuous improvement and policy-relevant research
The Commission is supported by Census Bureau statistical and evaluation staff, is gathering information through public meetings, has a website and mailing list CEP.gov, and solicited suggestions through the Federal Register
The Commission closes after it submits a final report in September 2017.

4. How expand the use of administrative data government-wide?
We examined federal guidance, recent GAO reports, social science literature and conference proceedings to identify:
Challenges to using administrative data for evaluating the outcomes of federal programs
Conditions that facilitate the use of agency administrative data
Guidance and resources available to address those challenges

5. Observed analytic and evaluation uses of administrative data
Identify areas of greatest need for program services
Assess program success in reaching targeted populations
Identify predictors of good (or poor) outcomes
Test effects of a specific treatment or service model
Assess effects of a program change on use of services or outcomes
Identify areas of greatest need for program services
Assess program success in reaching targeted populations (seniors, young children)
Identify predictors of good (or poor) outcomes (drug-treatment)
Test effects of a specific treatment or service model on outcomes (job training)
Assess effects of a program change on use of services or outcomes (fee increase, COPS)

6. Challenges to using administrative data
Key outcome data often at state and local level
Interagency barriers to access
Poor quality and documentation of many administrative data
Difficulties in matching records across datasets
Uneven agency capacity in information technology and program evaluation
State and local agencies possess data on key outcomes of interest (e.g., vital statistics, employment and earnings, school achievement, health care use, criminal arrest)
Interagency barriers to access: statutory limitations, security requirements, time and cost of negotiating access, ethics and consent requirements
Poor quality (reliability, completeness), timeliness, and documentation of many administrative data
Difficulties in matching records and data definitions across datasets
Uneven agency capacity in information technology and program evaluation

7. Facilitators for using administrative data
Ongoing interagency agreements
Ongoing relationships between researchers and program staff
Trusted data brokers (e.g., ICPSR, Census, CMS)
Agency investment in ensuring data reliability and validity
Ongoing interagency agreements (within and across states)
Ongoing relationships between researchers and program staff
Trusted data brokers (Inter-Univ. Consortium for Political & Social Research - U. of Michigan; Census Bureau Research Data Centers; Centers for Medicare and Medicaid Services Research Data Assistance Center - U. of Minnesota)
Agency investment in ensuring data reliability and validity (NHTSA FARS, ED and DOL state grants)

8. Resources for addressing challenges
OMB, Guidance for Providing and Using Administrative Data for Statistical Purposes, OMB M (2014)
OMB papers for the Commission on using administrative data and surveys to build evidence (2016)
Federal Committee on Statistical Methodology: Data Quality Assessment Tool (2013); Profiles in Success of Statistical Uses of Administrative Data (2009)

9. Resources for addressing challenges
HHS, Administration for Children and Families, Confidentiality Toolkit (2014); Cheaper, Faster Better: Are State Administrative Data the Answer? (2015)
Federal Chief Information Officer Council (2015). Open Data Prioritization Toolkit. Aug
U.S. Census Bureau. Federal Statistical Research Data Centers. Washington, D.C. ; Data Linkage Infrastructure (2016)

10. Contacts Stephanie Shipman, shipmans@gao.gov
Valerie Caracelli,
Center for Evaluation Methods and Issues
Applied Research and Methods Team
U.S. Government Accountability Office