1. The VR-ROI Project: Vocational Rehabilitation Return on Investment for Four State VR Agencies
Dr. Kirsten Rowe, Virginia Department for Aging and Rehabilitative Services
Dr. Robert M. Schmidt, University of Richmond
Dr. Steven Stern, University of Virginia
7th Annual Summit on Performance Management in Vocational Rehabilitation
Louisville, KY: September 8, 2014

2. Introduction Roadmap of Presentation History of Project Goal & Issues
Early work on ROI model development
Current activities and findings

3. Project Overview NIDRR-funded 3-year grant to U. of Richmond
VR agency partners include:
Virginia General (DARS)
Virginia Blind (DBVI)
Maryland Combined (DORS)
Oklahoma Combined (DRS)
Project consultants from U. of Virginia

4. Innovations Longitudinal administrative data on:
VR services provided
Employment and earnings (pre-, during, post-VR)
“Crack the black box” of VR services
Use applicant cohorts, not closure cohorts
Include everyone who applies for VR
Control for selection bias using “instruments”
Focus on “Rate of Return” vs. ROI

5. Longitudinal Data on Employment, VR Services, Labor Markets, and SSDI/SSI
Earnings & Employment data from state Unemployment Insurance program records
3 years prior through 5-10 years post VR application date using quarterly data for all VR applicants in SFY 2000/2007
VR Service Provision
Longitudinal VR services (up to 10 years) to account for multiple cases over time, purchased services and in-house/fixed costs
DI/SSI data from Social Security Administration
3 years prior through 5-10 years post; monthly receipt & dollar amounts
Local labor market data from U.S. Bureau of Economic Analysis

6. Categories of Purchased Services
First model: DTERMO
Diagnosis & evaluation: purchased services for assessing eligibility and developing IPE, plus medical diagnostics
Training: expenditures for OJT, GED program tuition and fees, etc.
Education: post-secondary education and vocational training
Restorative: medical/healthcare services plus AT
Maintenance: transportation, clothing, vehicle/home modifications, etc.
Other

7. Account for Variation in VR Consumers and Types of Services Provided
VR consumers by type of impairment
We estimate separate impacts by types of impairment (mental illness, cognitive impairments, physical impairments, learning disabilities)
VR services
We allow for different impacts of the VR service categories (DTERMO)
We can calculate Rate of Return by disability type or VR service category, as well as agency- wide

8. Model & Estimation
Service receipt equation: binary service receipt as a function of demographic characteristics and DRS office and counselor propensities (multivariate probit)
Employment equation: binary employment as a function of demographic characteristics, local labor market conditions, and service receipt (probit)

9. Model & Estimation
Quarterly Earnings: log quarterly earnings as a function of demographic characteristics, local labor market conditions, and service receipt
SSI/DI receipt: binary receipt as a function of demographic characteristics and service receipt
Division of time into 3 segments: before application; first two years after application (short-run); and more than two years after application (long-run)

10. Model & Estimation
Estimation by maximum simulated likelihood estimation: all parameters estimated jointly with allowance for rich covariance structure across equations

11. Measuring “Rate of Return” versus “Return on Investment”
ROR & ROI both use net earnings impacts and cost of service provision to calculate a measure of VR service efficacy
ROI requires the arbitrary selection of an interest rate, the choice of which becomes more important the longer the earnings time horizon
ROR for VR can be readily compared to rates of return such as the 10% annual ROR for long-term U.S. stock market performance

12. 2000 VA Results: ID

13. 2000 VA Results: MI

14. 2000 VA Results: PI

15. 2000 VA Work in Progress Working on LTESS Working on LD
Important to control for variation in vendor services
Working on LD
Race-based differentials in LD diagnosis and how it changes over time causes surprising results
Working on Autism
Data is constructed; estimation will begin soon

16. Service Aggregation for 2007
DTERMPS
Diagnosis & evaluation
Training
Education
Restorative services
Maintenance
Placement
Supported Employment

17. Maryland 2007 Data Description

18. Maryland 2007 Data Description

19. Maryland 2007 Data Description
MI Transition Probabilities
Initial Service
Employment
Not Employment
Subsequent Service
0.862
0.050
0.087
0.000
0.002
0.842
0.144
0.012
0.039
0.951
0.009
0.057
0.096
0.847

20. Maryland 2007 Data Description

21. Maryland 2007 Special Issues and Progress
Federal employees
Cross-border employment
Very high correlation of physical impairments and mental illness
MI estimation is proceeding, and we will have results soon

22. Oklahoma 2007 Data Issues and Progress
Enough Native Americans in sample to estimate effects of VR on that group
Data construction is mostly done, and estimation is ready to start

23. VA DBVI 2007 Modeling and Data Issues
The model that applies to MI, CI, and PI does not work as well for people who are blind because the nature of service provision, the aggregation of services, and (possibly) the measurement of outcomes are different. We are working with blind agencies to make progress on this issue.
There are some issues associated with observation of services in this data. But, we are relatively confident the service data exist, so we should be able to solve this problem.

24. 2007 Analysis Timeline
Maryland: estimation for MI group running now; other groups will follow; completion over next 6 months
Oklahoma: working through last data issues; estimation will start soon; completion over next 6 months
VA DBVI: significant modeling and data issues to resolve; completion over next year

25. 2007 Data and Beyond: Big Picture
Potential expansion to other states
Potential improvements in efficiency in collecting administrative data
Other useful data
SSA
Federal employment data
Other states’ UI data