1. Jonathan D. Breul Evaluation as an instrument to improve the quality of public spending - a look at the global financial crisis V Konferencja Ewaluacyjna Warszawa, 23.10.2009

2. © Copyright IBM Corporation 2009 Ministry of Regional Development & Polish Agency for Enterprise Development 5 th Evaluation Conference Warsaw, Poland October 23, 2009 Evaluation as an instrument to improve the quality of public spending - a look at the global financial crisis

3. © Copyright IBM Corporation 2009 3 Role of evaluation with the economic crisis  Imperative to measure the impact of investments and help make adjustments.  While it may be too late to shape how governments respond to this economic crisis, we have a responsibility – at a minimum – to capture lessons that can inform responses in the future.

4. © Copyright IBM Corporation 2009 4 Defining success  Confusion whether the programs represent spending or investment  In US, the Recovery Act has multiple purposes: 1.Government spending 2.Tax cuts 3.State fiscal relief  Not only intended to quickly create jobs – but also to make longer term investments the will benefit the growth of the economy in the longer term.

5. © Copyright IBM Corporation 2009 5 Measurable criteria  Achieve economic stimulus: Jobs created or job losses avoided  Achieve long-term public benefits by investments in transportation, environmental protection or technology  Foster energy independence or improved educational quality  Deliver program results  Satisfy transparency and accountability requirements

6. © Copyright IBM Corporation 2009 6 Many challenges  Measure outputs or outcomes?  It might be impossible to measure the impact or results – since impact often times lags years behind the actual investment  Often other factors will influence any particular impact  Hugely difficult and probably impossible to make a causal connection without a counterfactual.  We don’t know what would have happened in the absence of the Recovery Act  Compound effect of Recovery Act, and other measures

7. © Copyright IBM Corporation 2009 7 Stimulus job creation  The models are based on historical experience  But, we are outside anything we have experienced historically.  We are completely in a world we do not understand and know.  High level of uncertainty.  White House: plan will create 3.3 million to 4.1 million jobs by late 2010  Congressional Budget Office: estimates 1.2 million to 3.6 million additional jobs

8. © Copyright IBM Corporation 2009 8 Economists use a two-step process  First, they project how much the legislation will make the economy grow  Then they predict how many jobs that growth will create.  The second part is less complicated – because economists tend to rely on rules of thumb.  The White House assumed that each 1 percent increase in the economy’s size would produce 1 million jobs  The initial calculation is tougher.  To make it, economists rely on a mix of facts and assumptions.  Understandably, they come to different conclusions.

9. © Copyright IBM Corporation 2009 9  The White House method assumes that things were getting worse and that the stimulus is the sole factor responsible for stopping the bleeding.  Procedure is somewhat crude and does not take into account the obvious differences in wages and other costs across different types of projects and across different parts of the country  It does take into account key differences between tax changes or state fiscal relief, an direct government investment spending. Council of Economic Advisors estimates

10. © Copyright IBM Corporation 2009 10 OMB reporting guidelines  Direct reporting requirements cover one-third ($271 billion) of stimulus funds.  Recipients and sub-recipients required to report how many jobs are created or saved.  Total hours of work created divided by the number of hours worked by a full-time worker, along with a brief description of the types of work involved.  This data will later be used to evaluate the quality of jobs created by stimulus funding.

11. © Copyright IBM Corporation 2009 11 Obama Administration program evaluation effort  October 7, 2009, the Office of Management and Budget has issued new guidance and funding to agencies that develop and perform high-priority evaluation activities  Step toward making program reviews an essential function of government  Most rigorous study designs appropriate for different programs, given size, stage of development, etc.  Focus on “impact evaluations” to determine causal effects http://www.whitehouse.gov/omb/assets/memoranda_2010/m10- 01.pdf

12. © Copyright IBM Corporation 2009 12  We may never know for sure.  But - the need to enhance public sector performance has become more urgent as governments face mounting demands on public expenditure and calls for higher quality services.  The benefits of being clearer inside and outside government about purposes and results are undeniable. Is stimulus stimulating jobs?

13. © Copyright IBM Corporation 2009 13 Contact information Jonathan D. Breul Executive Director IBM Center for The Business of Government 1301 K Street NW, 4 th floor, West Tower Washington, DC 200045 (202) 515 -4382 jonathan.d.breul@us.ibm.com www.businessofgovernment.org