Recent Advances in Performance Measurement of Federal Workforce Development Programs Evaluation and Performance Management of Job Training Programs Organized by European Commission Directorate-General for Employment, Social Affairs and Equal Opportunity and University of Maryland School of Public Policy Randall W. Eberts W.E. Upjohn Institute for Employment Research November 7, 2009

2 Purpose Describes a Department of Labor project conducted by the Upjohn Institute that addresses some of the problems with current workforce system performance targets outlined in previous presentations: –Cream skimming –Inability to account for differences in local labor market conditions Accounts for factors outside the control of state and local administrators –By “outside the control,” we mean factors that affect performance outcomes but are not related to the services and other assistance provided by the programs –Local labor market conditions (unemployment rates) –Personal characteristics of participants (prior work history, educational attainment, barriers to employment) Offers a systematic, objective and transparent framework for: –Setting targets –Focusing on the value-added of WIA services –Diagnosing WIA performance –Leveling the playing field

3 Framework Procedure follows the basic regression-adjusted approach used to adjust JTPA performance targets Uses estimates of the effects of unemployment rates and personal characteristics on performance outcomes, based on all WIA exiters in all 50 states Estimates are based on the experience of individual participants within their local labor markets Procedure allows the aggregation of performance outcomes and factors from the individual to the WIB to the State to the Nation –By using the same weights for each level of jurisdiction, the differences add up –Thus the targets are consistent across jurisdictions Focus on three WIA programs and three common measures

4 Procedure Three step process: one for each jurisdictional level: national, state, WIB Step One: Adjust the national targets for assumed changes in unemployment rates Step Two: Use the national adjusted targets as the departure for setting state performance targets –State and national performance outcomes differ because of differences in unemployment rates and participant characteristics Step Three: Use each state’s adjusted targets as departure for setting targets of WIBs within the state –WIB and state performance outcomes differ because of differences in unemployment rates and participant characteristics

5 Basic Equation Use ordinary least squares regression to relate the performance measures to individual participant characteristics and local labor market conditions as measured by local unemployment rates Y isq = b o + b 1 *X isq + b 2 D s + b 3 D q + b 4 U sq + error Y isq : performance measure X isq personal characteristics and employment history D s : state or WIB dummy D q : quarter dummy U sq : quarterly unemployment rate by WIB or state Unemployment rates are entered in three ways depending on performance measure Personal characteristics and employment history are entered as categorical variables with one of the categories omitted from the equation as the reference group

6 Unemployment ratePersonal CharacteristicsOther WIB unemployment rateGenderWIB dummy variables Age (5 categories)Year-Quarter dummy School attainment (8 categories)Urban indicator Race/ethnicity (6 categories)Industrial structure Disabled Veteran Limited English Single parent TANF Other assistance Low income UI claimant/exhaustee Prior employment Variables included in estimation

7 Sample AdultDislocated Worker Youth Unit of observation Individual participants, quarterly Individual participant, quarterly Number of observations 480, ,000455, ,00060, ,000 States included All+PR+DC Demographic variables Employment history YES Fixed Effects WIBs Geographical unit of unemployment rates WIB (2000:q3-2007:q3) WIB (2000:q3-2007:q3) WIB (2000:q3-2007:q3)

8 Unemployment rates among counties with total employment of more than 100,000 ranged from 1.1 to 14.9 percent from 2000 through Significant Differences in Unemployment Rates Across States and Counties

9 Estimates of the Effect of Unemployment Rates on Performance Measures AdultDislocated Worker Older Youth Mean% Change Mean% Change Mean% Change Entered Employment %*** %*** ** Retention rate *** *** * Earnings11, **14, **6, *** (Percentage change of the performance measure associated with a one percentage point change in the unemployment rate; estimates are statistically significant at the (***), 0.01 (**) and 0.05 (*) confidence levels)

10 Step One: National Use estimates of the effect of unemployment rates on individual participants to adjust national performance targets based on President’s 2010 Budget

11 The adjusted targets take into account changes in the assumed unemployment rates, whereas GPRA targets remain flat during increases in unemployment rates.

12 Step Two: State Targets State targets differ from national targets: –Differences in unemployment rates –Differences in personal characteristics Add adjustment to the departure national target rate ABCDE WIA Adult Entered Employment State ANationalDifference (A-B) Effect on EEAdjustment: Weighted Difference (C * D) Unemployment rates12.6%8.3% High School drop out BA degree Disabled Work experience Adjusted Target Total adjustment (add column E) -11.5

13 Step Three: WIB Targets WIB targets differ from state targets: –Differences in unemployment rates –Differences in personal characteristics Add adjustment to departure state target rate ABCDE WIA Adult Entered Employment WIB A in State A State ADifference (A-B) Effect on EEAdjustment: Weighted Difference (C * D) Unemployment rates7.8%12.6% High School drop out BA degree Disabled Work experience Adjusted Target Total adjustment (add column E) +9.42

14 Adjustments Add Up Adjustments add up from WIB to State to Nation –Based on differences in characteristics –Weights are the same at all levels ABCDEF WIA Adult Entered Employment WIB A in State A State ANationalEffect on EE Difference in WIB and National (C – A) Adjustment: Weighted Difference (D * E) Unemployment rates7.8%12.6%8.3% High School drop out BA degree Disabled Work experience Adjusted Target Total adjustment (add column F) -2.12

15 Examples of Performance Adjustments State Adjustment ee ret earnings ee ret earnings place att lit Adult Dislocated Youth The adjusted targets, and their components, are shown for six states. It should be noted that the direction of the effect of the unemployment rate may be different for retention than for the other two performance measures since retention is estimated as the change in the unemployment. Differences in the changes in the unemployment rate between the state and the nation may be different from the differences in the levels.

16 WIA AdultWIA Dislocated Worker MeanMinimumMaximumMeanMinimumMaximum Entered Employment UR Labor Market Personal Characteristics Retention Rate UR Labor Market Personal Characteristics Earnings UR Labor Market Personal Characteristics Means of the Adjustment Components for WIA Adult and Dislocated Workers

17 State differences in personal characteristics contribute to a difference of as much as 12 percentage points in performance outcomes and the differences have increased in recent years. Significant Differences in Personal Characteristics Across States and WIBs Note: WIA Adult Entered Employment. Participant attributes are weighted by their estimated effect on performance outcomes. Mean Minimum Maximum

18 Summary Target adjustment procedure provides a systematic, transparent, and objective way to set national, state, and WIB performance targets for WIA programs Adjustment factors, since they are related to factors that are familiar to administrators, can be easily scrutinized to better understand and diagnose programs –Also familiar since state adjustment procedure is similar to the JTPA method National performance targets have already been adopted by USDOL and GAO Currently exploring adjustments at the state and WIB levels

19 Contact Information Randall Eberts W.E. Upjohn Institute for Employment Research 300 South Westnedge Ave. Kalamazoo, MI