1 The Effects of Customer Choice: First Findings from the Individual Training Account (ITA) Experiment Mathematica Policy Research, Inc. Social Policy Research Associates Presenter: Irma Perez-Johnson DOL Workforce Innovations Conference Philadelphia, PA – July 11-13, 2005

2 Policy Background: The Workforce Investment Act Key objective was to promote informed customer choice Key objective was to promote informed customer choice Required use of ITAs to give customers choice of training program Required use of ITAs to give customers choice of training program Customer selections to be made from Eligible Training Provider lists Customer selections to be made from Eligible Training Provider lists Established Consumer Report Systems to provide information on programs Established Consumer Report Systems to provide information on programs

3 Overview of the ITA Experiment Goal was to assess important tradeoffs in the design of ITA approaches Goal was to assess important tradeoffs in the design of ITA approaches Tested 3 approaches side-by-side in 8 LWIAs Tested 3 approaches side-by-side in 8 LWIAs Customers randomly assigned to approaches Customers randomly assigned to approaches Approaches selected to: Approaches selected to: –Reflect spectrum of emerging ITA approaches –Promote innovation in ITA program design

4 Approaches Tested Approach 1: Structured Customer Choice Approach 2: GuidedCustomerChoice Approach 3: MaximumCustomerChoice CounselingMandatory; most intensive Mandatory; moderate intensity Voluntary Can Counselors Reject Choices? YesNoNo Award Amount Customized, High Cap Fixed,ModestFixed,Modest

5 Evaluation Data Sources Implementation Study* Implementation Study* Study Tracking System* Study Tracking System* –Analyzed data for 7,922 study participants Follow-Up Survey Follow-Up Survey Unemployment Insurance Records Unemployment Insurance Records * Data sources for interim report

6 Methodology Estimate impacts on intermediate and final training outcomes Estimate impacts on intermediate and final training outcomes Intermediate outcomes: Intermediate outcomes: –Participation in counseling and training –Training choices Final outcomes: Final outcomes: –Completion of training –Customer satisfaction –Employment and earnings –Receipt of UI, public assistance –Cost of counseling, training

7 Responses to the ITA Approaches Most customers came in with strong ideas about training Most customers came in with strong ideas about training When counseling was voluntary, few customers used it When counseling was voluntary, few customers used it Counselors tended to defer to customer preferences Counselors tended to defer to customer preferences Local staff preferred Approach 2 (Guided Choice) Local staff preferred Approach 2 (Guided Choice) Private vendors repackaged their programs in response to caps Private vendors repackaged their programs in response to caps Little response from community colleges, universities Little response from community colleges, universities

8 Mandatory Counseling Discouraged Use of ITAs *** Difference relative to A2 is statistically significant at 0.01 confidence level

9 Training Costs Highest Under Approach 1, Lowest Under Approach 2 A1: Structured Choice A2: Guided Choice A3: Maximum Choice Average price of selected programs $4,764***$3,430$3,133 Average ITA expenditures per trainee $4,731***$2,849$2,847 Training rate (percentage) *** Average training cost per ITA customer $2,791***$1,641$1,853*** NOTE: Cost estimates do not include WIA administrative costs or cost of ITA counseling. *** Difference relative to A2 is statistically significant at 0.01 confidence level

10 Limited Differences in Training Choices by Approach ITA customers chose similar occupations ITA customers chose similar occupations Private schools were the dominant provider regardless of approach Private schools were the dominant provider regardless of approach Approach 3 customers slightly more likely to choose community colleges Approach 3 customers slightly more likely to choose community colleges Approach 1 customers chose slightly longer programs Approach 1 customers chose slightly longer programs

11 Remaining Questions Do more counseling and/or higher caps lead to better customer outcomes? Do more counseling and/or higher caps lead to better customer outcomes? Does lack of counseling lead to poorer customer outcomes? Does lack of counseling lead to poorer customer outcomes? What is the most cost-effective approach to structuring ITAs? What is the most cost-effective approach to structuring ITAs?

12 Remainder of ITA Experiment Follow-up survey Follow-up survey Administrative records Administrative records Final report (summer 2006) Final report (summer 2006)