Using Administrative Data for Federal Program Evaluation Stephanie Shipman and Valerie Caracelli U.S. Government Accountability Office American Evaluation Association conference October 2016 For more information, contact <shipmans@gao.gov>

Background U.S. Office of Management and Budget has been encouraging agencies to use existing administrative data to conduct more rapid, low-cost evaluations and increase evidence on what works FY2017 Proposed Budget acknowledged challenges and included proposals to improve access to and quality of agency datasets OMB encouraged: See 2013-14 memoranda, FY2016 and 2017 Budget guidance FY 2017 Build Census Bureau data access, data linking, and analysis capacity Help states build integrated longitudinal human services data systems Facilitate federal researchers’ access to state employment and earnings data Increase data quality in states’ education and workforce data

Background (cont’d) In March 2016, Congress established an Evidence-Based Policymaking Commission (PL 114-140) to examine how federal administrative and survey data can be made available to facilitate program evaluation, continuous improvement and policy-relevant research http://cep.gov The Commission is supported by Census Bureau statistical and evaluation staff, is gathering information through public meetings, has a website and mailing list CEP.gov, and solicited suggestions through the Federal Register The Commission closes after it submits a final report in September 2017.

How expand the use of administrative data government-wide? We examined federal guidance, recent GAO reports, social science literature and conference proceedings to identify: Challenges to using administrative data for evaluating the outcomes of federal programs Conditions that facilitate the use of agency administrative data Guidance and resources available to address those challenges

Observed analytic and evaluation uses of administrative data Identify areas of greatest need for program services Assess program success in reaching targeted populations Identify predictors of good (or poor) outcomes Test effects of a specific treatment or service model Assess effects of a program change on use of services or outcomes Identify areas of greatest need for program services Assess program success in reaching targeted populations (seniors, young children) Identify predictors of good (or poor) outcomes (drug-treatment) Test effects of a specific treatment or service model on outcomes (job training) Assess effects of a program change on use of services or outcomes (fee increase, COPS)

Challenges to using administrative data Key outcome data often at state and local level Interagency barriers to access Poor quality and documentation of many administrative data Difficulties in matching records across datasets Uneven agency capacity in information technology and program evaluation State and local agencies possess data on key outcomes of interest (e.g., vital statistics, employment and earnings, school achievement, health care use, criminal arrest) Interagency barriers to access: statutory limitations, security requirements, time and cost of negotiating access, ethics and consent requirements Poor quality (reliability, completeness), timeliness, and documentation of many administrative data Difficulties in matching records and data definitions across datasets Uneven agency capacity in information technology and program evaluation

Facilitators for using administrative data Ongoing interagency agreements Ongoing relationships between researchers and program staff Trusted data brokers (e.g., ICPSR, Census, CMS) Agency investment in ensuring data reliability and validity Ongoing interagency agreements (within and across states) Ongoing relationships between researchers and program staff Trusted data brokers (Inter-Univ. Consortium for Political & Social Research - U. of Michigan; Census Bureau Research Data Centers; Centers for Medicare and Medicaid Services Research Data Assistance Center - U. of Minnesota) Agency investment in ensuring data reliability and validity (NHTSA FARS, ED and DOL state grants)

Resources for addressing challenges OMB, Guidance for Providing and Using Administrative Data for Statistical Purposes, OMB M-14-06 (2014) https://www.whitehouse.gov/omb/evidence OMB papers for the Commission on using administrative data and surveys to build evidence (2016) https://www.whitehouse.gov/omb/management/commission_evidence Federal Committee on Statistical Methodology: Data Quality Assessment Tool (2013); Profiles in Success of Statistical Uses of Administrative Data (2009) http://fcsm.sites.usa.gov

Resources for addressing challenges HHS, Administration for Children and Families, Confidentiality Toolkit (2014); Cheaper, Faster Better: Are State Administrative Data the Answer? (2015) http://www.acf.hhs.gov/about/interoperability#chapter-2 Federal Chief Information Officer Council (2015). Open Data Prioritization Toolkit. Aug. 2015. https://cio.gov/cio-council-releases-open-data-prioritization-toolkit/ U.S. Census Bureau. Federal Statistical Research Data Centers. Washington, D.C. http://www.census.gov/fsrdc ; Data Linkage Infrastructure (2016) http://www.census.gov/datalinkage

Contacts Stephanie Shipman, shipmans@gao.gov Valerie Caracelli, caracelliv@gao.gov Center for Evaluation Methods and Issues Applied Research and Methods Team U.S. Government Accountability Office