Rio Grande do Sul SWAp Brazil Impact Evaluation Concept Note Joint work with colleagues from Rio Grande do Sul.

Intervention Summary Multi-sector US$ 480m SWAp with Brazilian state of Rio Grande do Sul. Theme: improve growth impact of public investment through support for public sector M&E and consultative mechanisms. Four sectors: education, disaster risk & environmental management, transport, and private sector development PSD: evaluate government support for diffusion of management practices in industrial regions WB engagement increasingly in Brazil at state level; planning loan operation with state of Rio Grande do Sul.

Evaluation Questions Evaluation question: impact of receiving ‘soft upgrading’ services/strategy consulting on operating performance? Policy objective: induce other firms to upgrade, encourage diffusion of modern management practices. PSD: tradition of support for clusters (APLs) – horizontal linkages for collective efficiency, public upgrading services; but need for taking stock – WB involvement will contribute to evaluation and redesign; assessment is similar to David’s work in India.

Evaluation Design Treatment (av. 200 per region) Interested Population Randomization Population Control (av. 200 per region) Invitation Participation voluntary – so issue invitation, and then randomize treatment among interested firms. Each region independent, but can pool data ex post to increase power. Not interested Unit of analysis is firm: but services delivered regionally (‘nucleos produtivos regionais’); so effectively 15 independent evaluations; one-shot grant per region.

Sampling and Data Benchmark Data Cost measure Costs as % of turnover Firm 1 Firm 2 Firm 3 Firm 4 Firm 5 Firm 6 Finished goods 0.21 3.97 0.84 0.75 0.43 0.28 Work in progress 0.50 0.06 0.58 0.65 0.64 Raw materials 0.53 10.91 2.79 0.88 1.62 0.49 Customer return rate 6.00 0.10 0.00 – Internal rework rate 0.22 1.76 0.89 Internal scrap rate 6.34 1.15 13.46 0.29 4.92 Downtime due to changeovers 2.51 1.67 2.25 7.24 4.72 2.67 Downtime due to machine breakdowns 4.99 1.40 0.40 1.03 3.15 4.55 Downtime due to tooling breakdowns 2.75 1.64 3.10 6.29 0.78 Downtime due to int. material unavailability 0.13 Downtime due to ext. material unavailability 6.20 Absenteeism rate 2.57 0.36 1.00 1.18 TOTAL 28.20 23.10 20.77 20.14 19.05 17.10 Use independent benchmarking firm to collect baseline and follow up data – separate responsibility for data collection from intervention. Two advantages: avoid conflict of interest; induce control group to provide sensitive information (give them international BM data). But be aware that benchmark itself may have treatment effect!

Timeline July 2011: approve concept note, begin developing methodology March 2012: first disbursement; collect baseline data; begin firm training June 2013: evaluate first 5 regions (1,000 treated firms)

Team and staffing State of Rio Grande do Sul: Sergio Kapron (AGDI), Julia Ambros (AGDI) World Bank: Tom Kenyon (TTL/PSD specialist); Miriam Bruhn and David McKenzie (evaluation specialists)

Budget ?