Lessons from the United States: Evaluating Employment Services is Neither Easy Nor Cheap November 7, 2009 Sheena McConnell Peter Schochet Alberto Martini
Evaluations have led to new programs –Worker Profiling and Reemployment Services system –Self-Employment Assistance program Expanded funded for programs found effective –Job Corps Reduced funding for programs found ineffective –JTPA youth program Provided administrators useful information –Individual Training Account experiment Evaluation Has Informed U.S. Policy 2
What to evaluate? –Whole program or component? For whom is the program effective? –Decide on populations of interest prior to evaluation What is the counterfactual? –“No service” environment does not exist Step 1: Develop Research Questions 3
Consider experiments –Provide credible estimates –Widespread in US and developing countries Nonexperimental approaches –Need good quality data –Careful analysis –Not as credible Step 2: Choose Evaluation Design 4
More politically acceptable if: –Pilot of new program –Excess demand for program’s services –Treatments differ, no control Cooperation from program staff requires: –Discussing the importance of the experiments –Reduce burden on program: small control groups, quick random assignment Political and Program Support Necessary for Experiments 5
Experiments often take years –Sample intake period –Follow-up period Costs incurred only by experiments: –Site recruitment –Staff training –Conducting and monitoring random assignment But experiments require smaller sample sizes than nonexperimental evaluations Experiments Take More Time and May Cost More 6
Outcomes –Survey versus administrative data Baseline –Study form, surveys, administrative data Service receipt –Both treatment and control/comparison group Program implementation –Understand what is being evaluated Program costs –To determine cost effectiveness Step 3: Collect Data 7
Collect high quality data Consider conducting experiments Invest in rigorous evaluations Recommendations 8