1. Lessons from the United States: Evaluating Employment Services is Neither Easy Nor Cheap November 7, 2009 Sheena McConnell Peter Schochet Alberto Martini

2.  Evaluations have led to new programs –Worker Profiling and Reemployment Services system –Self-Employment Assistance program  Expanded funded for programs found effective –Job Corps  Reduced funding for programs found ineffective –JTPA youth program  Provided administrators useful information –Individual Training Account experiment Evaluation Has Informed U.S. Policy 2

3.  What to evaluate? –Whole program or component?  For whom is the program effective? –Decide on populations of interest prior to evaluation  What is the counterfactual? –“No service” environment does not exist Step 1: Develop Research Questions 3

4.  Consider experiments –Provide credible estimates –Widespread in US and developing countries  Nonexperimental approaches –Need good quality data –Careful analysis –Not as credible Step 2: Choose Evaluation Design 4

5.  More politically acceptable if: –Pilot of new program –Excess demand for program’s services –Treatments differ, no control  Cooperation from program staff requires: –Discussing the importance of the experiments –Reduce burden on program: small control groups, quick random assignment Political and Program Support Necessary for Experiments 5

6.  Experiments often take years –Sample intake period –Follow-up period  Costs incurred only by experiments: –Site recruitment –Staff training –Conducting and monitoring random assignment  But experiments require smaller sample sizes than nonexperimental evaluations Experiments Take More Time and May Cost More 6

7.  Outcomes –Survey versus administrative data  Baseline –Study form, surveys, administrative data  Service receipt –Both treatment and control/comparison group  Program implementation –Understand what is being evaluated  Program costs –To determine cost effectiveness Step 3: Collect Data 7

8.  Collect high quality data  Consider conducting experiments  Invest in rigorous evaluations Recommendations 8