1. For Peace and Goodwill? Using an Experimental Game to Analyse the Effect of the Desarollo y Paz Programmes in Colombia. David PHILLIPS Institute for Fiscal Studies, London (also Orazio Attanasio and Luca Pellerano) LACEA Annual Conference Buenos Aires, 1st October 2009

2. Introduction Increasing emphasis on evaluating programmes with socio-cultural aims: – values, aptitudes, perceptions, motivations, capabilities, empowerment – relationships, networks, social capital Rigorous impact evaluation in this domain is extremely challenging as: – Expected socio-cultural changes are often not clearly specified (Theory of Change) – They are difficult to measure – As the analysis moves in the domain of the intangibles, the evaluation must be driven my a multi method and multidisciplinary approach. Attanasio, Pellerano and Phillips (2009): – We evaluate the effect of a Social Fund type intervention in Colombia on social capital and conflict management using a Public Goods game.

3. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

4. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

5. The Setting. Conflict in Colombia. Colombia has been affected by a complex civil conflict along the whole course of its recent history. Two main illegal factions: Guerrilla(s) and Paramilitary groups (+ new emerging groups after process of desmovilización) Despite progress, violence still affects many rural areas of the country in a very clustered way. Strong association between the presence of illegal groups and violence levels, local political instability, inequality (Sanchez et al, 2003)

6. The Peace and Development Regional Programmes (PRDPs) Community-driven development / peace-building initiative Networks of grassroots organizations formed from existing civil society (labour unions, the church, private foundations) in poor and violent regions Claim peace is bottom up and based on development and community action. Support a wide range of development processes informed by the principles of protecting life (Primero la vida), equity, solidarity and participatory democracy The PRDPs work as an umbrella fund, financing a series of projects along three main axes: 1.Productive Development 2.Institutional Development and Governance 3.Human Rights and Culture of Peace

7. The Peace and Development Regional Programmes Started in Magdalena Medio in the early 1990s (Jesuits) 6 PRDPs have been supported by the UNDP, the World Bank (LIL1 and LIL2 grants, 1998-2003; Paz y Desarrollo Loan, 2004-2008) and the European Commission (Laboratorios de Paz I, II and III, 2002-2010) Total investment of more than 180 million USD

8. 6 Regions 9 Departments 125 Municipalities 1363 projects 800 grassroots organizations 180,000 estimated beneficiaries On average: 300 beneficiaries per project 2 years project duration 120,000 USD project value The Peace and Development Regional Programmes

9. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

10. Experimental Games Social capital Competing definitions and competing measuring tools Standard survey techniques – Structural approach (membership) versus motivational/value based approach (trust, reciprocity) – People tend to respond to value based survey questions according to social expectations and social norms (Glaeser et al., 2000). Our definition: social cooperation (act collectively for mutual gain) Our measure: a behavioural measure of contribution to a local public good in a field experimental setting

11. Experimental Games Based on the theoretical framework of the experimental economics: players decisions should reveal their preferences/motivations The controlled experiments are designed in such a way that decisions only have monetary consequences (no reputation effects) As the decisions imply a real monetary payoff (on average the equivalent of 3 USD) participants are expected to reveal their preferences more similarly to how they would behave in a real life situation

12. Experimental Games Voluntary Contribution Mechanism – VCM (Marwell and Ames, 1979) The experiment is a simple public good game in which subjects decide to invest in a public or a private account Played in group (40 participants on average) Every token invested in the public account yields benefits for the whole group, while the token invested in the private account only yields benefits to the owner of the account Investing in the public pot is socially optimal However an individual is always better off by investing in the private account (Dominant Strategy)

13. The game is played in Two Rounds: First Round. Anonymous decision Second Round. 10 minutes communication contribution to local public good communication

14. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

15. The Data Two main data sources 1.(Late) Baseline for the evaluation of PyD and LP (Nov 2006 – Feb 2007). Treated Locations. Beneficiary and Non-Beneficiary Households 18 months of exposure to the activities of the PRDPs on average High variability in exposure because of the roll out of the PRDPs both within and across municipalities 2.Second follow up for the evaluation of Familias en Acción (Nov 2005 – Apr 2006) Control Locations Sample for the evaluation of the famous CCT programme. Targeted to the poor (SISBEN1), mainly women

16. PRDP Project Beneficiaries Members of Other Civic Organizations Sample of Treated Locations 904 Non-Beneficiaries (Control Group 2) 782 Beneficiaries (Treatment Group) Beneficiaries with higher exposure Beneficiaries with lower exposure The Data Sample of Control Locations Sample of 2472 (Control Group 1) PRDP Municipalities (37) FA Municipalities (67) FA DataPRDP Data

17. The Data. Experimental Game

18. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

19. Descriptive Evidence

20. High degree of heterogeneity in beneficiary characteristics according to project types

21. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

22. Evaluation Strategy No random assignment -> Potential endogenous programme placement (selection and self selection) – Across locations – Within locations No pre-programme information (apart from municipality characteristics) Identification strategy a) within locations (beneficiaries vs. non-beneficiaries in treated locations) b) across locations (treated vs. control matched municipalities) c) within treated households (based on the intensity of exposure) May provide insights on the presence and magnitude of spill-over effects Multivariate regression models. All models are based on the selection on observables assumption

23. Evaluation Strategy. Across Municipalities We match treatment and control locations on a complete set of pre- programme municipality characteristics 2 control locations dropped because of missing information 20 control locations and 16 treated locations are dropped because they fall outside the common support 2,427 households (58% of the original sample) are on the common support)

24. Evaluation Strategy. Across Municipalities

25. Evaluation Strategy. Within Treated Municipalities

26. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

27. Results. Experimental Game There is no significant difference in the contribution rates between treated and control households within treated locations. Effects within treated municipalities Round 1 Round2 Treated Household-0.004-0.003-0.012 0.0300.0230.022 [0.037] [0.036] [0.043][0.046] Round 1 Decision YES Location ControlsYES Household ControlsYES Session Composition ControlsYES Potentially Endogenous Vars.YES Observations1485 Marginal Probit. Errors are Clustered at the Session level. * significant at 10%; ** significant at 5%; *** significant at 1%

28. Results. Experimental Game The level of contribution to the public good is much higher in municipalities where the PRDPs operate, than in other municipalities in the country with comparable characteristics. Results hold if CS is imposed based on matching on individual characteristics Effects across municipalities Round 1 All Locations Matched Locations All Locations Matched Locations All Locations Matched Locations Treated Location0.256***0.221***0.150*0.185*0.135*0.141 [0.059][0.071][0.083][0.102][0.082][0.098] Location ControlsYES Household ControlsYES Session Composition ControlsYES Potentially Endogenous Vars.YES Observations378222433782216237782162 Marginal Probit. Cluster at the Session level. * significant at 10%; ** significant at 5%; *** significant at 1% Round 2 All Locations Matched Locations All Locations Matched Locations All Locations Matched Locations Treated Location0.332***0.313***0.305***0.288***0.300***0.263** [0.065][0.075][0.099][0.110][0.100][0.104] Round 1 DecisionYES Location ControlsYES Household ControlsYES Session Composition ControlsYES Potentially Endogenous Vars.YES Observations378222433674216236742162 Marginal Probit. Cluster at the Session level. * significant at 10%; ** significant at 5%; *** significant at 1%

29. Results. Experimental Game We find some light evidence that, ceteris paribus, increased exposure to the PRDPs lead to higher contribution to the public good, particularly in the long run. Exposure effects Round 1 Round 2 Exposure (7-14 months)0.0450.010 -0.047-0.058 [0.048][0.047] [0.043] Exposure (>14 months)0.085**0.064 -0.014-0.032 [0.043][0.042] [0.048][0.043] Months of Exposure0.0030.002 -0.001 [0.003] Months of Exposure (Municipality Max) 0.003**0.002* -0.001 [0.001] Round 1 Decision YES Location ControlsYES Household ControlsYES Session Composition ControlsYES Observations684 Errors are Clustered at the Session level. * significant at 10%; ** significant at 5%; *** significant at 1%

30. Results. Experimental Game Exploiting knowledge networks, social capital effects seem to propagate within treated municipality through a spill-over mechanism. The fact of knowing the PRDP could however be endogenous to social capital. Examining the case for Spill Over effects. The previous findings suggest that there might be some contamination effect in social capital formation at the community level between treated and control households within the same municipality. Round 1 Round 2 Knows PRDP0.078**0.094*** 0.0010.013 [0.033][0.034] [0.031][0.032] Treated Household-0.044-0.048 0.030.016 [0.038][0.037] [0.046][0.048] Round 1 Decision YES Location ControlsYES Household ControlsYES Session Composition ControlsYES Observations1472 Marginal Probit. Errors are Clustered at the Session level. * significant at 10%; ** significant at 5%; *** significant at 1%

31. Results. Experimental Game Because of the sampling procedures, the proportion of beneficiaries in the session doesnt reflect coverage rates in the municipality In the second round, the group effect seems to be operative for control households only This suggests that it is mainly driven by reputation factors Complementary evidence suggests that PRPD beneficiaries are actually considered more trustworthy than the rest of the population Critical Mass or Conditional Cooperation? Round 1 Round 2 Proportion of Treated in the Session0.521**0.614*** 0.584**0.634** [0.214][0.223] [0.264][0.256] Treated Household0.0250.060 0.248*0.265** [0.145][0.153] [0.129][0.124] Interacted (proportion * treated)-0.082-0.143 -0.518*-0.564* [0.316][0.328] [0.314][0.301] Decision in Round 1 YES Location ControlsYES Household ControlsYES Session Composition ControlsYES Observations1485 Errors are Clustered at the Session level. * significant at 10%; ** significant at 5%; *** significant at 1%

32. Results. Experimental Game Exploring the critical mass hypothesis

33. 1.The Setting and the Programme 2.The Experimental Games 3.The Data 4.Descriptive Statistics 5.The Evaluation Strategy 6.Results 7.Conclusions

34. Conclusions PRDPs are grassroots-driven programmes designed to improve social capital and hence entrench peace and drive development. Survey-based evaluations has problems so an experimental Public Goods game is used instead: contribution as measure of social capital Difficult to identify an effect at individual level. Much more contribution in treatment versus control municipalities. Increased duration of exposure, knowledge of programme increase contribution. Proportion of beneficiaries in game increase contribution Suggest reputation is important and critical mass and intensity and coverage are key

35. For Peace and Goodwill? Using an Experimental Game and a Social Dilemma to Analyse the Effect of the Desarollo y Paz Programmes in Colombia. more information on the PRDPs can be found at: http://www.redprodepaz.org/ http://www.accionsocial.gov.co/ THANKS LACEA Conference Buenos Aires, 1 st October 2009