Evaluating HIV Prevention and Treatment Programs Damien de Walque Markus Goldstein

2 (Impact) Evaluating Health Programs is different Norm in medicine to use randomized control trials to figure out what works What we know less about (for example): –Socioeconomic effects of health interventions –How to get people to utilize treatment (e.g. vaccines) –What is the most cost effective mode of treatment

3 This is an issue for HIV/AIDS Examples: –ART works biologically, but what impact does it have on patients socio-economic status? –Circumcision is effective in reducing HIV risk, how do we get people to volunteer? –We know different prevention interventions have some effect – which is the most cost effective?

4 This presentation We will not focus on the medical findings We know something about non-medical outcomes through evaluation, but actually not that much rigorous (in a comparison group sense) evidence But, a lot in progress, and we will talk about some of that

5 Example 1 Duflo, Dupas, Kremer & Sinei: Education and HIV AIDS Prevention The interventions: 1.training teachers in the Kenyan government’s HIV/AIDS-education curriculum 2.organized debate and essay contest on the role of condoms in protecting teens against HIV/AIDS 3.reduced cost of education through the supply of school uniforms 4.information campaign for Kenyan teenagers to spread the awareness of high HIV prevalence among adult men (uses earlier evaluation work)

6 Evaluation design –Schools randomly assigned to treatment groups and control –Baseline and endline data collection to use a difference in difference approach –Measured the effect of the program on: Teacher coverage of HIV/AIDS Student HIV knowledge Student attitudes Self reported behavior Child bearing (girls) and drop out rates

7 Main findings –Teacher training led to no significant reduction in teen pregnancy, but increased likelihood that pregnancies occur within marriage –Debate over condoms led to (some) increased report of condom use –Reductions in the cost of schooling led to reduction in drop out rates and reduction in teen pregnancies

8 Prevention of HIV/AIDS Source: Duflo, Dupas, Kremer, and Sinei (2006)

9 Example 2 Thirumurthy, Graff Zivin, and Goldstein: The Economics of AIDS Treatment: Labor Supply in Western Kenya The intervention –Provide ART to patients, rule is for a CD4<200, but initial rationing –Treatment for opportunistic infections –Nutrition supplementation, but not much until later Data – two rounds of hh survey – random sample & patients

10 Evaluation Design We know what happens to counterfactual group Medical evidence: continued decline in health and death Allows estimation of upper bound of treatment impact Zero labor supply in round Mean hours worked in past week Days on ARVs treatedcounterfactual

11 BMI Before and After ARV Therapy Source: AMPATH Medical Records System – data as of March LFP before and after ARV Therapy Source (right): Household survey data.

12 Main Findings Large and rapid labor supply response in patients Spillover benefits to other members of the household – young boys & women reduce their labor supply Patient earnings (relative to zero counterfactual) are close to the cost of treatment

Some work in progress

14 Question 1: Socio-economic impact of reducing premature adult mortality: the case of ART In addition to labor supply: Schooling and welfare of children How household coping mechanisms change with ART Effects of ART on farming productivity Using evaluation results for macro models of the economic effects of AIDS & treatment

15 Question 2: Possible effects of ART on HIV transmission and prevention Direction of effect Beneficial (Slow transmission) Adverse (Speed transmission) Type of effect Biological Reduce infectiousnessSelect for resistance. Longer duration of infectivity Behavioral Encourage prevention, especially testing Off-setting behavior, “disinhibition”

16 Other questions (more on the supply- facility side) 3) Adherence to treatment 4) How to avoid the development and spread of resistance? 5) How are ART beneficiaries identified? How to encourage timely uptake? 6) How to assure the quality of HIV/AIDS service delivery? 7) How to encourage capacity building to reinforce the sustainability of ART delivery?

17 Methodology and data collection (Longitudinally) Biomedical follow-up including data on treatment regimen and treatment success (CD4 counts) Household surveys (HIV patients and general population) including health, schooling, labor force. 7 countries: Burkina Faso, Ghana, Kenya, India, Mozambique, Rwanda and, South Africa Surveys ongoing and scheduled. Data and preliminary analysis will be available by the end of 2008.

18 Determinants Patient and Provider Behavior Outcomes Impact on the entire country Treatment Outcome, Resistance Development Socio- Economic benefits for households Prevention Patient Adherence Information socio-economic variables Community variables Stigmatization Training Selection / Recruiting Impact On health system Quality of service delivery Equipment Staffing and Incentives Associations Employers Socio- Economic benefits for firms Framework for Learning Agenda

19 Methodological challenges: endogeneity Given issues with randomization, in some countries, we will evaluate some experiments on the conditions of ARV delivery. Rwanda: performance-based contracting for HIV/AIDS services in health facilities South Africa: food and counseling intervention as adherence support. Kenya: text messaging intervention as reminders for adherence

20 Some other common issues for HIV evaluations Measurement. Dupas & co. raise the issue of what should be the impact variable, particularly in prevention –Issue of self reported behavior (Gersovitz) –Think about using biomarkers – HIV or STI?

21 Comdom use at the last intercourse with the spouse: discordant reports Source: DHS 2003 and 2004 N = Burkina Faso 1630 Cameroon 1764 Ghana 1830 Kenya 1361 Tanzania 2497 Man no, Woman no 88.9%90.8%91.9%97.0%90.9% Man yes, Woman yes 2.0%2.2%1.9%1.0%1.6% Man yes, woman no 6.9%4.2%5.0%1.8%4.3% Man no, Woman yes 2.2%2.8%1.2%0.3%3.2%

22 Issues in HIV evaluation, cont. Control Group: Thirumurthy et. al. point to the problem of a medical intervention known to be effective and the problem of generating a counterfactual –Sequel paper looks at children, here the authors use a range of control group techniques The shape & determinants of the epidemic vary across time and across countries, so results in one country may not apply to another…need to do multiple evaluations of the same intervention