The VR-ROI Project: Vocational Rehabilitation Return on Investment for Four State VR Agencies Dr. Kirsten Rowe, Virginia Department for Aging and Rehabilitative Services Dr. Robert M. Schmidt, University of Richmond Dr. Steven Stern, University of Virginia 7th Annual Summit on Performance Management in Vocational Rehabilitation Louisville, KY: September 8, 2014

Introduction Roadmap of Presentation History of Project Goal & Issues Early work on ROI model development Current activities and findings

Project Overview NIDRR-funded 3-year grant to U. of Richmond VR agency partners include: Virginia General (DARS) Virginia Blind (DBVI) Maryland Combined (DORS) Oklahoma Combined (DRS) Project consultants from U. of Virginia

Innovations Longitudinal administrative data on: VR services provided Employment and earnings (pre-, during, post-VR) “Crack the black box” of VR services Use applicant cohorts, not closure cohorts Include everyone who applies for VR Control for selection bias using “instruments” Focus on “Rate of Return” vs. ROI

Longitudinal Data on Employment, VR Services, Labor Markets, and SSDI/SSI Earnings & Employment data from state Unemployment Insurance program records 3 years prior through 5-10 years post VR application date using quarterly data for all VR applicants in SFY 2000/2007 VR Service Provision Longitudinal VR services (up to 10 years) to account for multiple cases over time, purchased services and in-house/fixed costs DI/SSI data from Social Security Administration 3 years prior through 5-10 years post; monthly receipt & dollar amounts Local labor market data from U.S. Bureau of Economic Analysis

Categories of Purchased Services First model: DTERMO Diagnosis & evaluation: purchased services for assessing eligibility and developing IPE, plus medical diagnostics Training: expenditures for OJT, GED program tuition and fees, etc. Education: post-secondary education and vocational training Restorative: medical/healthcare services plus AT Maintenance: transportation, clothing, vehicle/home modifications, etc. Other

Account for Variation in VR Consumers and Types of Services Provided VR consumers by type of impairment We estimate separate impacts by types of impairment (mental illness, cognitive impairments, physical impairments, learning disabilities) VR services We allow for different impacts of the VR service categories (DTERMO) We can calculate Rate of Return by disability type or VR service category, as well as agency- wide

Model & Estimation Service receipt equation: binary service receipt as a function of demographic characteristics and DRS office and counselor propensities (multivariate probit) Employment equation: binary employment as a function of demographic characteristics, local labor market conditions, and service receipt (probit)

Model & Estimation Quarterly Earnings: log quarterly earnings as a function of demographic characteristics, local labor market conditions, and service receipt SSI/DI receipt: binary receipt as a function of demographic characteristics and service receipt Division of time into 3 segments: before application; first two years after application (short-run); and more than two years after application (long-run)

Model & Estimation Estimation by maximum simulated likelihood estimation: all parameters estimated jointly with allowance for rich covariance structure across equations

Measuring “Rate of Return” versus “Return on Investment” ROR & ROI both use net earnings impacts and cost of service provision to calculate a measure of VR service efficacy ROI requires the arbitrary selection of an interest rate, the choice of which becomes more important the longer the earnings time horizon ROR for VR can be readily compared to rates of return such as the 10% annual ROR for long-term U.S. stock market performance

2000 VA Results: ID

2000 VA Results: MI

2000 VA Results: PI

2000 VA Work in Progress Working on LTESS Working on LD Important to control for variation in vendor services Working on LD Race-based differentials in LD diagnosis and how it changes over time causes surprising results Working on Autism Data is constructed; estimation will begin soon

Service Aggregation for 2007 DTERMPS Diagnosis & evaluation Training Education Restorative services Maintenance Placement Supported Employment

Maryland 2007 Data Description

Maryland 2007 Data Description

Maryland 2007 Data Description MI Transition Probabilities Initial Service Employment Not Employment Subsequent Service 0.862 0.050 0.087 0.000 0.002 0.842 0.144 0.012 0.039 0.951 0.009 0.057 0.096 0.847

Maryland 2007 Data Description

Maryland 2007 Special Issues and Progress Federal employees Cross-border employment Very high correlation of physical impairments and mental illness MI estimation is proceeding, and we will have results soon

Oklahoma 2007 Data Issues and Progress Enough Native Americans in sample to estimate effects of VR on that group Data construction is mostly done, and estimation is ready to start

VA DBVI 2007 Modeling and Data Issues The model that applies to MI, CI, and PI does not work as well for people who are blind because the nature of service provision, the aggregation of services, and (possibly) the measurement of outcomes are different. We are working with blind agencies to make progress on this issue. There are some issues associated with observation of services in this data. But, we are relatively confident the service data exist, so we should be able to solve this problem.

2007 Analysis Timeline Maryland: estimation for MI group running now; other groups will follow; completion over next 6 months Oklahoma: working through last data issues; estimation will start soon; completion over next 6 months VA DBVI: significant modeling and data issues to resolve; completion over next year

2007 Data and Beyond: Big Picture Potential expansion to other states Potential improvements in efficiency in collecting administrative data Other useful data SSA Federal employment data Other states’ UI data