6th Annual Summit on Vocational Rehabilitation Program Evaluation & Quality Assurance Providence, RI: September 16, 2013 Dr. David Dean, University of Richmond† Dr. Kirsten Rowe, Va. Dept. for Aging and Rehabilitative Services Estimating Return on Investment for State Vocational Rehabilitation Programs †Dr. Dean died on August 11, 2013, after confronting a very difficult year of illness with characteristic humor, courage, and strength.

Project Overview NIDRR-funded 3-year grant to University of Richmond VR agency partners include: – Virginia General (DARS) – Virginia Blind (DBVI) – Maryland Combined (DORS) – Oklahoma Combined (DRS) Purpose is to develop “state of the science” ROI estimates using readily available data

Key Features of Our Approach (1 of 2) Link readily-available longitudinal data from multiple systems to examine impacts Focus on individuals, not cases Use applicant cohorts, not closure cohorts Include everyone who applies for VR Start examining program impact with the first VR application

Key Features of Our Approach (2 of 2) “Crack the black box” of VR services Control for selection bias Develop individual-specific “Rate of Return” (ROR) estimates

Use Longitudinal Data on Employment, VR Services, and DI/SSI Receipt  Earnings & Employment data from state Unemployment Insurance program records o 3 years prior through 5 or 10 years post VR application date using quarterly data for all VR applicants in SFY 2000/2007  VR Service Provision o Longitudinal VR service provision (up to 10 years) to account for multiple cases over time; account for both purchased services and in-house costs  DI/SSI data from Social Security Administration o 3 years prior through 5 or 10 years post using monthly receipt & dollar amounts

Study Individuals, Use VR Applicant Cohorts, Evaluate Impact of Initial VR Case  VR is no longer a “one and done” program: Many individuals have multiple VR cases  Closure cohorts enter the VR program over a number of years, spanning VR program and economic climate changes  We separate multiple VR episodes into o A “base case” – the first application occurring in a given SFY o All prior VR applications o All subsequent VR episodes occurring within ten years of the base case  We evaluate all applicants whose initial base case was in a given fiscal year (SFY 2000 or 2007)

Account for Variation in VR Consumers and Types of Services Provided  VR consumers by type of impairment o We estimate separate impacts by types of impairment (mental illness, cognitive impairments, physical impairments, learning disabilities)  VR services o We allow for different labor market effects of seven categories of VR services (DTERMPS: diagnosis and evaluation, training, education, restoration, maintenance, placement and supported employment)  We can calculate ROR by disability type or VR service category as well as agency-wide

DTERMPS Across All Agencies: 25,765 Base Cases TotalDTERMPS % Receiving 68%38%15%9%24%29%10%14% Avg. cost (if any) $3,297$491$2,161$3,657$1,786$1,179$1,404$3,307 D:Diagnostic & Evaluation T :Training E :Education R :Restorative M:Maintenance P:Placement S:Supported Employment

DTERMPS by Agency AgencyDTERMPS 1 % Receiving 31%12%5%19%28%5%18% Avg. Cost $512$1,646$2,540$1,067$900$1,735$3,768 2 % Receiving 34%24%12%37%53%14%-- Avg. Cost $435$4,147$7,480$2,321$3,887$4, % Receiving 57%11% 18%24%18%15% Avg. Cost $543$2,481$2,597$2,190$1,097$1,130$1,886 4 % Rcvng 23%22%12%36%33%9%7% Avg. Cost $282$2,221$5,382$2,065$1,338$1,565$5,544

Use a State-of-the-Science Labor Economics Model to Identify Employment Impacts  We formalize and estimate a model of labor market outcomes (likelihood of employment and earnings increases) resulting from the choice of VR service mix  Features of our model control for “selection bias” (unobservable differences between those who receive services and those who do not) o "Instrumental variables" are variables correlated with service choice but not with unobservable influences on labor market outcomes o Pre-program labor market outcomes aid in controlling for differences between those who do and do not receive VR-paid services o Statistical model controls for interrelationships between service choices and labor market outcomes and aids in the interpretation of results

Measuring “Rate of Return” versus “Return on Investment”  ROR & ROI both use net earnings impacts and cost of service provision to calculate a measure of VR service efficacy  ROI requires the arbitrary selection of an interest rate, the choice of which becomes more important the longer the earnings time horizon  ROR for VR can be readily compared to rates of return such as the 10% annual ROR for long-term U.S. stock market performance

Some Preliminary Estimates of VR’s Impact  Our estimates differ dramatically across impairment groups  For people with mental illness: o Median annual rate of return is 17.5% o 88.5% have positive rates of return o 10% exceed a 50.7% annual ROR  For people with cognitive impairments o Median annual rate of return is 34.5%. o 78.7% have positive rates of return o 20% exceed a 101.9% annual ROR Note: These estimates are for SFY 2000 cohort from Virginia

 Our estimates also differ by type of service provided  For people with mental illness o Most effective: Training (includes supported employment)  $7,200 average present value for 10 years of earnings o Education  $1,700 average present value for 10 years of earnings  For people with cognitive impairments o Most effective: Education  $36,000 average present value for 10 years of earnings o Training (includes supported employment)  $10,000 average present value for 10 years of earnings Some Preliminary Estimates of VR’s Impact Note: These estimates are for SFY 2000 cohort from Virginia

Next Steps Develop population- and agency-specific ROR estimates using SFY 2007 cohorts Develop three-state estimate for individuals who are blind or vision impaired Work with all participating agencies to disseminate and use results

Contact Information Kirsten L. Rowe, Ph.D. VR-ROI Project Coordinator Va. Dept. for Aging and Rehabilitative Services 8004 Franklin Farms Dr. Richmond, VA

Acknowledgments This project is funded by Field Initiated Project grant #H133G from the National Institute on Disability and Rehabilitation Research to the University of Richmond. Dr. Rowe wishes to acknowledge the invaluable contributions of her friend and colleague Dr. Dean to both this presentation and the study of return on investment for vocational rehabilitation.