Changing Perspectives on Workforce System Performance- Adjustment Models Workforce Innovations Conference July 21, 2004

2 Presenters USDOL Grant Project Overview Proposed Model Panel Response Q & A Amanda Ahlstrand, USDOL- ETA Marcia Black-Watson, MI Randall Eberts, WE Upjohn Institute Neil Ridley, Heldrich Center Dan O’Shea, Univ. of Texas Craig Schrader, Michigan Works! Janet Howard, MI

3 Background: US DOL-ETA Grant Performance Environment -- Focus on results for customers -- Better utilize available information to inform program decisions Grant awarded to Michigan: Develop strategies and guidance for state and local workforce investment system goal setting and performance adjustment on behalf of the states –Review information needed to establish a process for goal setting and managing programs under WIA –Confer with other states on suggested approaches –Discuss ways to frame and analyze issues, data, and mechanisms for system performance management –Develop proposals for comment and review –Implement pilot projects for testing

4 Project Overview Two-tier Project With a Focus on Performance Develop a framework to assist states and local workforce investment areas in performance goal setting Provide management tools to help local workforce investment areas performance management

5 Project Team Department of Labor & Economic Growth Corporation for a Skilled Workforce Public Policy Associates W.E. Upjohn Institute for Employment Research

6 Informed Decisions Meeting the needs of our customers Evaluating agency performance fairly and equitably Determining the effectiveness of the delivery of services Evaluating and improving programs using “real time” measures

7 Evidence-based Decision Tools Administrators are able to: Develop management tools Understand what factors contribute to success Adjust their course of action during the program year

8 Project Components: Tools to Identify the Factors That Contribute to Participant Success Distinguish between: –Factors outside the control of the local administrators –Factors that are within their control – “Value-added Performance” Customize the tools for local areas using individual administrative data and wage records

9 Project Components: Framework for Negotiating Performance Goals Adjusts for factors outside the local administrator’s control Tracks the progress of local areas in meeting their performance goals Offers evidence-based decision making in referring participants to services and in improving the quality of services

10 Performance Adjustment Model: Purposes To develop “fair” measures of local workforce program performance using the new “common measures” To develop “value added” measures of the program To develop “timely” predictors for local program managers on how well their local area will do on common measures

11 Performance Adjustment Model: Methodology Statistically adjust for how outcomes are influenced by customer characteristics and local economic conditions Use statistical model, data on local customers, and forecasts of local economic conditions to provide “real time” forecast of how a local area’s performance will be adjusted for performance standards Use data on intermediate outcomes to predict the local area’s performance on common measures Combine predictions of outcomes and adjustment to predict whether the MWA will exceed each performance standard

12 Adjustment Procedure Quantify the impact client and economic conditions have on individual outcomes Aggregate these impacts to determine statewide or agency-based “expected” outcomes Compare “expected” outcomes to actual outcomes when data become available

13 Requisites for a “Good” Model High predictive power Understandable by administrators Objective Useful as a management tool Generate performance measures in a timely fashion

14 Requisites for Explanatory Variables Make intuitive sense Significant explanatory power Available on real-time basis at the one-stop when person registers for program Parsimonious –For example, pared 84 variables to 41 in common measure 1 regression

15 DOL’s Common Measures Adult Measures Entered employment Employment retention Earnings increase 1 Earnings increase 2 Program efficiency Youth Measures Placement in employment and/or education Attain a degree or certificate Literacy and numeracy gains Program efficiency

16 Common Measure 1: Employment Rate Example is from WIA Adult Program Whether adult WIA participant who was not employed at registration is employed one quarter after exiting the program Employment in a quarter is measured by earnings greater than zero in wage record

17 Common Measure 1: Employment Rate RegistrationExit Q1 after exit Q2 after exit Q3 after exit Not Employed at registration AND Employed in Q1 after exit E=1 E=0

18 Estimate Impacts on Q1 Employment Y ij = B x X ij + W j + e ij Estimation based on individual observations of participants from 7/00-9/02 Wage records and administrative data Adult WIA Programs (10,000+ observations)

19 Predicting Common Measures Y ij = B x X ij + W j + e ij Variation in common performance measures (Y) explained by three sets of factors: 1)Personal characteristics (X) 2)Local labor market conditions (X) 3)The effect of program intervention (W)—value added measure B x : effects of customer characteristics on post- exit employment W j : Value added impact of program intervention

20 Explanatory Variables VariableEmpl. Rate RetentionEarnings Chg 1Earnings Chg 2 AgeXX GenderXXXX EducationXXXX RaceX WagesXXX BarriersXXXX IndustryXXX WIA areasXXXX Unemp rateXXXX

21 Selected Results: Adult CM1 Individuals, age 29 or less, are 8% more likely to become employed compared with those 50 or older Local areas that experience unemployment increases of 1% from one quarter before registration to one quarter after exit are expected to have 1% lower employment rate one quarter after exit Individuals with disability are 7% less likely to become employed one quarter after exit High school dropouts are 4% less likely to become employed Several others......

22 Adjustment Factors Adjustment factor –The difference between the weighted average characteristics of the individual MWA and the weighted average characteristics of the state Value Added Performance –Difference between Local Area Performance and Statewide Average Performance –Adjusted for the difference between Local Client & Economic Conditions and Statewide Client & Economic Conditions

23 Example of Adjusting Common Measure 1 for WIA Adults MWA Actual employment rate for adult WIA Participants, 1 quarter after exit Difference from state mean Portion of difference from state mean due client or local area adjustment factors Portion of difference from state mean not due to adjustment factors—e.g. Value Added A B C D E F G H I J

24 Example of Adjusting Common Measure 1 for WIA Adults

25 Real-time Forecasts Use the model to predict performance of each MWA using all estimated effects except for MWA area effects and effects of changes in unemployment –Use this to adjust the MWAs actual performance relative to state standard or to adjust state standard to a local standard Adjustment factor can be computed for each individual for individual characteristics at time of registration Local economic trends can be assumed or forecasted Allows local administrators to glean the direction of adjustments to performance standards

26 Final vs. Predicted Overall Adjustment (Correlation = 0.934)

27 Forecasting Actual Performance based on Intermediate Outcomes This model with real-time adjustment can be used to forecast actual performance for each MWA Common measure outcomes for each local area may be forecasted at any point in time by using a forecasting equation that includes intermediate outcomes in addition to the other explanatory variables Intermediate outcomes are often highly correlated with common measures; e.g. for CM1—someone employed at exit is 35% more likely to be employed at +Q1 These predictions have a 0.79 correlation with actual performance

28 Predicting Common Measure 1 for WIA Adults with Exit Variables MWAsActual employment rate 1 quarter after exit Predicted employment rate using data on employment at exit A B C D E F G H I J K L

29 Correlation of Actual and Predicted Performance =.789

30 Predicting whether MWA Meets Performance Standards Combine predictions using intermediate outcomes to forecast actual outcomes, and forecast adjustment using registration data to predict how each MWA will do on performance standard Correlation =.693

31 Other Common Measures, Programs and Groups Similar adjustment and forecasts using intermediate outcomes are developed for all common measures based on employment, earnings, or educational attainment: –WIA youth and displaced worker programs; –ES; TAA; WorkFirst Models so far have been estimated for 4 common measures for WIA adult programs and 2 common measures for WIA youth

32 Selected Results Many good individual predictors for: –Adult CM2 (job retention) –Adult CM3 (earnings gain - Q1 to + Q1) –Youth CM1 (job entry) –Youth CM2 (attainment of degree or certificate) Adult CM4 is problematic; relatively few good predictors for CM4 (earnings gain – Q1 to + Q3) Possible to construct good real time predictors for all common measures but adult CM4 Possible to construct good predictors using intermediate outcomes for adult CM1 and CM3

33 Timeline _______________________ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ RegistrationExit 1 Quarter after Exit Unemployment data 1 quarter after exit available Wage record data on one quarter after exit available Estimate model Using Historical data, integrate into local data management system Collect participant data, including prior earnings from wage records. Combine with model βs to predict adjustment Collect exit data. Use to predict actual common measure and to predict success or failure on performance standards With unemployment and wage record data, calculate final performance adjustments and final success or failure on performance standards

34 What’s the Payoff? Timely data –enhances accountability –improves program management and planning –greater satisfaction for clients and business Accurate and fair performance evaluation –prevents “creaming” –gives programs “bonus points” for working with the hard-to-place Better and more targeted use of resources –identifies what works best –tracks improvements over time

35 What is needed to succeed? Administrative data at the local program level –Including Social Security Numbers Ability to match wage-record and administrative data using SSN as the link Opportunity to track wage-record data for individuals who work in or move to other states Assistance from DLEG’s Bureau of Labor Market Information and Strategic Initiatives Information system enhancement to handle increased data reporting and analysis

36 Panel Neil Ridley, Heldrich Center, Rutgers University Dan O’Shea, University of Texas Janet Howard, State of Michigan: Moderator

37

38 For Further Information Marcia Black-Watson, State of Michigan, Martha Reesman, Corporation for a Skilled Workforce,