Using non-experimental methods to evaluate a Conditional Cash Transfer program: The case of Bolsa Alimentação

International Food Policy Research Institute –Pedro Olinto Bolsa Alimentação (Ministry of Health, Brazil) –Eduardo Nilson London School of Hygiene & Tropical Medicine –Saul Morris Emory University –Rafael Flores Instituto Materno-Infantil de Pernambuco –Ana Claudia Figueró Instituto de Estudos do Trabalho e Sociedade –Alinne Veiga

The program Bolsa Alimentação is a Brazilian federal government program designed to reduce nutritional deficiencies and infant mortality Beneficiaries –(i) pregnant women, –(ii) mothers breastfeeding a child up to 6 mo –(iii) children of age six months up to seven years Beneficiary selection: –Municipal quota based on projected number malnourished children –Municipal authorities select beneficiary households R$15 (US$6.25) /beneficiary /month (up to R$45) Mother signs “Charter of responsibilities”

Evaluation design: main features Non-experimental Ex-post only comparison of beneficiary and non-beneficiary households Non-beneficiary households were registered by local authorities as beneficiaries, but excluded due to: –Data transmission error –“Special characters” in the name –Discrepancy family records Bolsa Escola

Evaluation design: main features Evaluation limited to municipalities –In the Northeast region of the country –With at least 40 excluded households –Where transfers had been made for 6 months Fieldwork undertaken exactly 6 mo after first transfer in each of the 4 survey municipalities

Pairing of beneficiaries and excluded Pairing separate for each class of beneficiary –Pregnant women –“Lactating mothers” (up to 6 mo post-partum) –Children For each “would-be beneficiary”, the most similar similar actual beneficiary was identified, based on –Municipality of residence* –Sex* –Socio-economic status (based on reported income; rent; water, electricity & gas bills, family size) –Age

Pairing (2) Pairing algorithm: “nearest neighbor” with calipers –Pairwise distance calculated based on Euclidean distances –Relatively importance of age and SES chosen subjectively –No replacement of beneficiaries –“Excludeds” matched in a random order –Two beneficiaries per “excluded”, to maximize power

Problems with the pairing Turned out that program registers had not recorded all the young children in the hhs –Expected no. under-7s: 1.3 per hh –Revealed no. under-7s: 1.9 per hh Individual-level pairing not so good for household-level analyses, since same beneficiary hh paired to more than one excluded hh

Bolsa Alimentação beneficiary households Excluded households P-value Educational level of women aged 15-49: Incomplete primary 79.0% (646/818) 79.3% (252/318) 0.92 Flooring material: Earth Floor tiles Other (mostly cement) 8.0% (57/716) 10.8% (77/716) 81.3% (582/716) 9.6% (27/282) 11.0% (31/282) 79.4% (224/282) 0.70 Water source: public network50.5% (360/713)50.4% (142/282)0.97 Telephone9.2% (66/717)11.0% (31/282)0.39 Family size, mean (s.d.): Total 0.0 – 6.9 y 7.0 – 13.9 y 5.4 (2.1) 1.9 (0.9) 0.8 (1.0) 5.8 (2.3) 1.8 (0.9) 1.1 (1.1) <0.001 Bolsa Escola beneficiary26.8% (192/717)55.7% (157/282)<0.001 Duration of exposure to Bolsa Alimentação at the time of the survey, months, mean (s.d.): 5.9 (0.3) N/A N/A Socio-economic and demographic characteristics of Bolsa Alimentação beneficiary and excluded households in the evaluation sample

Marginal propensity to consume food out of Bolsa Alimentação transfers OLS model: –R$0.70 for every R$1.00 transferred IV model (controlling for possible endogeneity of number of beneficiaries): –R$0.57 for every R$1.00 transferred

Impact on dietary diversity

Longitudinal analysis

Associations between length of exposure to Bolsa Alimentação and weight gain in children aged 0–36 months. Panel analysis based on routinely recorded weight data: random effects model WEIGHT | Coef. Std. Err. z P>|z| [95% Conf. Interval] AGE | ln(AGE) | MALE | BA_BASELINE | BA_DURATION | BE | _cons | sigma_u | sigma_e | rho | (fraction of variance due to u_i)

Differential weight gain associated with each month of exposure to Bolsa Alimentação, estimated for children of ages 6, 12, 18, 24, 30, and 36 mo, in a random effects model that allows program effect to interact both with age and the natural log of age weight | Coef. Std. Err. z P>|z| [95% Conf. Interval] months | months | months | months | months | months |

Lessons learnt: methodology Very detailed understanding of beneficiary selection process resulted in robust non- experimental identification Success critically dependent on getting into the field at the right time Individual matching ensured that beneficiary and non-beneficiary groups were very similar Routinely collected weight data permitted longitudinal analysis, quality much better than expected However, lack of conventional baseline allows controversy over results to rage indefinitely!

Lessons learnt: results Strikingly high proportion of transfer spent on food, suggesting good targeting Diversity of family diet improved markedly, as in similar programs Vaccination coverage not improved because already excellent Weight gain effect probably illustrates dangers of linking targeting and/or program graduation to nutritional status outcomes For nutrition, too many beneficiaries over 2 yo, damage likely to be irreversible