1. Rio Grande do Sul SWAp Brazil Impact Evaluation Concept Note
Joint work with colleagues from Rio Grande do Sul.

2. Intervention Summary
Multi-sector US$ 480m SWAp with Brazilian state of Rio Grande do Sul.
Theme: improve growth impact of public investment through support for public sector M&E and consultative mechanisms.
Four sectors: education, disaster risk & environmental management, transport, and private sector development
PSD: evaluate government support for diffusion of management practices in industrial regions
WB engagement increasingly in Brazil at state level; planning loan operation with state of Rio Grande do Sul.

3. Evaluation Questions
Evaluation question: impact of receiving ‘soft upgrading’ services/strategy consulting on operating performance?
Policy objective: induce other firms to upgrade, encourage diffusion of modern management practices.
PSD: tradition of support for clusters (APLs) – horizontal linkages for collective efficiency, public upgrading services; but need for taking stock – WB involvement will contribute to evaluation and redesign; assessment is similar to David’s work in India.

4. Evaluation Design Treatment (av. 200 per region) Interested Population
Randomization
Population
Control (av. 200 per region)
Invitation
Participation voluntary – so issue invitation, and then randomize treatment among interested firms. Each region independent, but can pool data ex post to increase power.
Not interested
Unit of analysis is firm: but services delivered regionally (‘nucleos produtivos regionais’); so effectively 15 independent evaluations; one-shot grant per region.

5. Sampling and Data Benchmark Data Cost measure Costs as % of turnover
Firm 1
Firm 2
Firm 3
Firm 4
Firm 5
Firm 6
Finished goods
0.21
3.97
0.84
0.75
0.43
0.28
Work in progress
0.50
0.06
0.58
0.65
0.64
Raw materials
0.53
10.91
2.79
0.88
1.62
0.49
Customer return rate
6.00
0.10
0.00 –
Internal rework rate
0.22
1.76
0.89
Internal scrap rate
6.34
1.15
13.46
0.29
4.92
Downtime due to changeovers
2.51
1.67
2.25
7.24
4.72
2.67
Downtime due to machine breakdowns
4.99
1.40
0.40
1.03
3.15
4.55
Downtime due to tooling breakdowns
2.75
1.64
3.10
6.29
0.78
Downtime due to int. material unavailability
0.13
Downtime due to ext. material unavailability
6.20
Absenteeism rate
2.57
0.36
1.00
1.18
TOTAL
28.20
23.10
20.77
20.14
19.05
17.10
Use independent benchmarking firm to collect baseline and follow up data – separate responsibility for data collection from intervention. Two advantages: avoid conflict of interest; induce control group to provide sensitive information (give them international BM data). But be aware that benchmark itself may have treatment effect!

6. Timeline July 2011: approve concept note, begin developing methodology
March 2012: first disbursement; collect baseline data; begin firm training
June 2013: evaluate first 5 regions (1,000 treated firms)

7. Team and staffing
State of Rio Grande do Sul: Sergio Kapron (AGDI), Julia Ambros (AGDI)
World Bank: Tom Kenyon (TTL/PSD specialist); Miriam Bruhn and David McKenzie (evaluation specialists)

8. Budget ?