Health Programme Evaluation by Propensity Score Matching: Accounting for Treatment Intensity and Health Externalities with an Application to Brazil (HEDG Working Paper 09/05, March, 2009) Rodrigo Moreno-Serra Centre for Health Economics & Department of Economics University of York AfrEA-NONIE-3ie Conference, Cairo, 2009

Introduction & Motivation Main programme evaluation challenge: ex-post construction of an adequate comparison group, often within a non- experimental setting, to obtain average treatment effects Key assumption: Values of treated and untreated outcomes for a given individual are not influenced by the treatment status of other individuals Usually unrealistic for health programmes: externalities can lead to underestimation of total programme impacts (cf. e.g., Miguel and Kremer, 2004)

Introduction & Motivation Possible solution in non-experimental settings: use availability of a health programme in a given area as the treatment variable of interest Treated individuals defined as those who live in areas where programme is in place (treated areas) Methodology already used, normally through an indicator variable for presence/absence of the programme One mean programme impact that accounts for health externalities to individuals in the treatment areas who did not directly receive the intervention Yet magnitude of externalities within each locality (and thus the associated average treatment effect) is likely to depend on the number of individuals who actually receive the programme’s services there: intensity of treatment

Suggested empirical methodology at a glance Use a measure of the programme’s population coverage across areas as the treatment variable of interest Estimate average treatment effects through comparisons between the health impacts of alternative coverage levels vs. reference level (e.g., zero) Need panel-data or repeated cross-sections (before and after) on coverage levels (phased-in programme) and individual variables Compare change in outcomes for individuals living in an area with coverage level l (a treatment area) to the change in health outcomes for similar individuals living in the area with coverage level 0 (the comparison area), for a number of l > 0 Implementation: propensity score matching estimators adapted to the case of multiple treatments (Imbens, 2000; Lechner, 2000), coupled with a difference-in-differences approach (PSDD)

Suggested empirical methodology at a glance PSDD estimator of the average treatment effect on the treated (ATT) with repeated cross-sections (Blundell and Costa-Dias, 2000) : Construction of comparison groups through propensity score matching for multiple treatments (generalized propensity score): more than one active treatment, i.e. coverage levels ATT computed by difference-in-differences Key assumption: bias stability ATT of living in area l vs. living in area 0 takes into account health benefits to individuals living in l who did not receive programme services themselves

An example: the Brazilian PSF Family Health Programme (PSF): MoH initiative with the stated aim of “improving the health status of covered families” Family Health Teams have to be formed by family doctor, nurse, assistant-nurse and 4-6 community health agents Monthly household visits: preventive and health promotion actions for all the individuals in a family (adults, children, seniors) Municipalities make PSF adoption decision: individuals mandatorily covered (visited) Broadest health programme ever launched in Brazil: 80 million people (2006), yet important variations across regions Average health gain for resident of a given region likely to increase (non-linearly?) according to coverage, also due to externalities

Data Porto Alegre is the comparison region (~ zero coverage, 98-03) Household survey data (1998 & 2003): >127,000 individuals (adults and children) and 34,000 households per wave Individual matching variables include household living conditions, demographics, education, labour and income characteristics

Evaluation question What are the average health impacts of being exposed to each of the eight observed PSF coverage levels from 1998 to 2003, compared to living in the comparison region during the same period (the “no-programme” benchmark)? Health outcomes: (1) self-assessed health; (2) bed due to illness; and (3) inability to perform usual activities due to illness One ATT is estimated for each of the eight relevant pairwise comparisons: being exposed to the PSF coverage level observed in region 1, 2, 3… vs. not exposed to the PSF in Porto Alegre Specification tests: matching successful for construction of similar comparison groups (both samples, adults and children)

Main results Overall, positive levels of PSF coverage in a region tend to lead to improvements in individual health outcomes (small effects for adults, larger estimated impacts for children) Largest ATT tend to be found for residents of the regions with the three highest median PSF coverage levels during (Belo Horizonte—16%, Recife—23%, Fortaleza—24%) E.g., children in Fortaleza vs. Porto Alegre: (1) 5-8p.p. higher prob of good SAH; (2) 3p.p. lower prob of bed episode; (3) 4-5p.p. lower prob of inability to perform usual activities due to illness But no clear pattern of increasing health benefits according to higher coverage levels: too few/low coverage levels available

Concluding remarks Health programme evaluation strategy that can be applied when information on actual impacts is needed to guide resource allocation and roll-out strategies, but only limited (routine, non-experimental) data are available Of course, validity of assumptions of the PSDD estimator with multiple treatments needs to be assessed case-by-case Impact estimates account for (i) different health endowments, and the potentially substantial (ii) treatment non-linearities and (iii) externalities from different levels of population coverage Comprehensive account of the health benefits generated by an intervention—not only its effects on actually “treated” individuals: relevance for policy-makers