Challenges of Measuring Employment Program Performance William S. Borden November, 2009

Mathematica Proprietary & Confidential  Effective performance management  Goals and definitions of measurement and measures  Impact of performance system on behavior  Methods for obtaining reliable data  Stakeholder input  Fear and burden  Accountability and complexity  WIA performance measures Topics 2

Mathematica Proprietary & Confidential  Designing and implementing national performance systems involves different set of tools than research or policy  Effective government performance management based on software development methods  High value data requires precise and objective definitions, detailed documentation, sound software development and testing practices  Highly fragmented national management information systems, imprecise definitions and lack of motivation to increase performance outcomes poses risk to data quality Operational Challenges of Performance Management 3

Mathematica Proprietary & Confidential  Legitimate discussion on value of specialized service delivery programs for special populations –Elderly poor –Disadvantaged youth –People with disabilities –Veterans  Overlapping programs present comparability challenge –Assessing relative effectiveness versus mainline programs  Service delivery fragmentation leads to reduced management and data capacity and resistance to increased burden –Economies of scale reduce management capacity Comprehensive View of Employment Programs 4

Mathematica Proprietary & Confidential  Performance data can provide essential management information for all program levels –Good performance management process is necessary foundation for research evaluations (otherwise data will be unreliable)  Very involved technical process  Information is not useful without –Precisely defined and objective measures and data elements –Extensive technical documentation –Standardized automated edits and calculations –Extensive software testing Effective Performance Management 5

Mathematica Proprietary & Confidential  Upfront investment in well-defined measures, data elements, measure calculations and standardized tools  Investments are leveraged across all levels of system  Much more accurate, timely and useful data  Careful initial planning reduces the need to redesign and rebuild systems – fewer rounds of stakeholder input  Inconsistent and unreliable data are not cost effective Effective Performance Management Lowers Costs 6

Mathematica Proprietary & Confidential  Determine program effectiveness, return on public investment  De-fund ineffective programs  Provide incentives for high performance Market Related Goals of Performance Management 7

Mathematica Proprietary & Confidential  Competition, profit and loss translate poorly to government program evaluation  Defining goals is difficult  Performance-based budgeting is ultimate market mechanism –Requires very precise and accurate data –Provides maximum incentive for inappropriate behaviors (creaming, manipulating enrollment, exit and exclusion data)  Public programs have natural geographic and political monopolies (hard to defund Ohio and send customers to Michigan) Limitations of Market Motives 8

Mathematica Proprietary & Confidential  Understand basic facts about programs –Customers served –Services provided –Results  Detect superior and inferior performance and associated service delivery approaches –Act on findings by implementing remedial steps –Identify and assimilate best practices –Analyze performance trends Goals of Performance as a Management Tool 9

Mathematica Proprietary & Confidential  Measures must generate rates of success and not counts –Must be able to track performance trends over time –Compare performance across operating units  Outcome measures better than process measures  Intermediate measures of progress needed if customers are in services for a long time  Standards needed to identify acceptable and unacceptable performance –Must be adjusted to account for differences in customers and labor markets Defining Measures 10

Mathematica Proprietary & Confidential  ETA has strong data validation system – WIA, NFJP, TAA, ES, UI –Based on long history of performance measurement and data validation in Unemployment Insurance program  Uniform national standards and software to edit, calculate and validate data  Hard to define and document what makes data valid – how to document homeless youth?  UI has standard for data quality based on review of sample cases (and incorporating standard error)  No data quality standards for employment training programs and no calculation of standard error Obtaining Reliable Data 11

Mathematica Proprietary & Confidential  Difficult to define enrollment, exit, employment and earnings –These data elements drive the calculations  Some states cut enrollment in response to WIA to manage flow of customers into performance measures –Issue of responsibility for self-service customers –How valid to measure impact of such a small intervention, but there were large infrastructure costs  Many customers never exited from JTPA –WIA created “soft exit” – no services for 90 days so that everyone would be counted  Try to negotiate lowest possible goals to allow for improvement Manipulating Performance 12

Mathematica Proprietary & Confidential  Stakeholders do not want to be accountable for circumstances beyond their control  Customers “disappear” and become negative outcomes –These situations should occur randomly and evenly across states or grantees –If one state had a significantly higher percentage – might indicate flaws  Exclusions from performance – death, illness, incarceration –Death is the most simple– exclude record from performance –Illness and family member illness is very subjective – documentation is difficult – more prevalent and problematic in older worker program  All of these factors greatly increase complexity of measures  Stakeholders then complain that measures are too complex Accountability and Complexity 13

Mathematica Proprietary & Confidential  Almost all measures derive from legislation  Agencies must develop operational definitions, calculations  Inputs from states, grantees and local areas is valuable –They have strong knowledge of issues with the data –Their buy-in is critical for acceptance of rewards and sanctions For them to use performance data as a management tool  Resistance to measures, especially where management capacity is deficient  Strong centralized leadership and effective communication of goals and methods is essential Stakeholder Involvement 14

Mathematica Proprietary & Confidential  Considerable fear of performance measures  First reaction is to complain about the burden  Reporting burden is exaggerated; performance reporting uses data agencies already track for program management –Follow up data is largest burden; can replace with wage records  Data validation is large burden for family income, homelessness, health performance exclusions  Shifting focus from service delivery to making the numbers Fear and Burden 15

Mathematica Proprietary & Confidential  UI wage records are key to objective measurement of program outcomes –Long lags are a problem for prompt feedback to program operators –Effort involved to get national wage file including federal and military employment  Measuring earnings gain has been problematic –Pre-to-post program ratio distorted by pre-enrollment earnings gaps  Skill and credential attainment rates were ill-defined –Reluctance to develop precise definitions –No usable data  New measures much better –Diploma or certificate and literacy and numeracy –Standardized, well-defined, very complex to calculate and test WIA Performance Measures 16

Mathematica Proprietary & Confidential  Measures and data elements are hard to define and validate  Risky to draw strong conclusions from performance data  Emphasis on sanctions and defunding may promote inappropriate behavior  Emphasis on management information and detection of problem areas promotes improvement and cooperation  Need to invest in technical infrastructure, standardization to achieve reliable and comparable results Conclusion 17