1. Supporting community action on AIDS in developing countries Evaluating Impact The Alliance experience with the Frontiers Prevention Programme Presentation to IMPACT EVALUATION WORKSHOP 24th May 2011

2. Supporting community action on AIDS in developing countries Contents Introduction to the Alliance The Frontiers Prevention Programme The FPP evaluation Challenges to the Evaluation Evaluation Results Conclusions

3. Supporting community action on AIDS in developing countries Alliance Mission: To reduce the spread of HIV and meet the challenges of AIDS Alliance Vision: A world in which people do not die of AIDS

4. Supporting community action on AIDS in developing countries A Global Alliance Currently working in over 33 countries Supporting community- based organisations Building national capacity Spreading good practice and lessons learned Emphasising the importance of working with marginalised and key populations

5. Supporting community action on AIDS in developing countries Evaluating Impact: A case study on the Frontiers Prevention Program INSTITUTO NACIONAL DE SALUD PÚBLICA (INSP)

6. Supporting community action on AIDS in developing countries Overview of Frontiers Prevention Project Aim: To make a significant contribution to reducing HIV infections in relatively low- prevalence countries BMGF-funded: US$25m for three years Comprehensive packages of interventions targeting ‘key populations’ (KPs) in Ecuador, India, Cambodia and Morocco KPs include: sex workers and clients; PLHIV; MSM; IDUs; and STI service users

7. Supporting community action on AIDS in developing countries 28 interventions in 8 clusters: 1.Strengthening (NGO & others) capacity to work with KPs 2.Implementing peer outreach activities 3.KP collective mobilisation for advocacy 4.Implementing KP mutual support, cultural and solidarity building 5.Implementing KP risk reduction skills building activities 6.Developing & disseminating IEC for KPs 7.Strengthening clinical capacity & quality of services 8.Implementing anti-stigma and HIV prevention with the general public FPP interventions

8. Supporting community action on AIDS in developing countries Decrease in HIV Incidence in site GOAL FPP conceptual framework Decrease In HIV incidence amongst KPs Decrease in HIV Incidence in site

9. Supporting community action on AIDS in developing countries Decrease in KP risky behaviour Decrease in KP STI Prevalence Decrease in HIV Incidence in site Decrease in HIV Incidence amongst KPs GOALPURPOSE FPP conceptual framework

10. Supporting community action on AIDS in developing countries Service and Commodity provision for KPs Decrease in HIV Incidence amongst KPs Decrease in KP risky behaviour Decrease in KP STI Prevalence Decrease in HIV Incidence in site Empowerment for prevention for KPs Enabling Environment GOALPURPOSE INTERMEDIATE OUTCOMES Presented by H. Gayle at BNA IAC (similar to Avahan and APSACS) FPP conceptual framework

11. Supporting community action on AIDS in developing countries Will the implementation of the FPP program have an impact in terms of in changes in behaviour and STI/HIV prevalence in sites? Service and Commodity provision for KPs Decrease in HIV Incidence amongst KPs Decrease in KP risky behaviour Decrease in KP STI Prevalence Decrease in HIV Incidence in site Empowerment for prevention for KPs Enabling Environment

12. Supporting community action on AIDS in developing countries Qualitative study (India) India (Horizons, IHS Hyderabad) Cross section longitudinal design with two observations: baseline (2004) and follow-up (2006). IDIs, FGDs and SPSS to compare main sub-categories of SWs (brothel-based, street-based, and home-based) and MSM. Baseline: 4 intervention and 4 comparison sites while for follow-up: 3 + 2. FPP evaluation model

13. Supporting community action on AIDS in developing countries FPP evaluation model Quantitative study (India & Ecuador) Survey instrument developed by a multidisciplinary team of researchers & validated with KPs Socioeconomic, demographic, behavioral, knowledge and HSV2 & syphilis biomarkers Baseline (Q3-Q4 2003): India: 2,182 FSW & 2,929 MSM in 24 evaluation sites Ecuador: 2,026 FSW & 2,093 MSM in 6 evaluation sites. Follow-up (Q3-Q4 2007): Revised to include data on exposure to interventions India: FSW: 2,374 & MSM: 2,014 Ecuador: FSW: 1,760 & MSM: 1,676

14. Supporting community action on AIDS in developing countries INDIA (AP) SW MSM IDU PLHIV ECUADOR Design: intervention & comparison sites

15. Supporting community action on AIDS in developing countries Adilabad Nizamabad Karimnagar MedakWarangal Rangareddy Hyderabad Nalgonda Mahabubnagar Guntur Kurnool Prakasam Khammam Krishna Ananthapur CuddapahNellore Chittoor West Godavari Visakhapatnam East Godavari Vizianagaram Srikakulam APSACS + HLFPPT APSACS + ALLIANCE ALLIANCE HLFPPT 8 8 6 1 No district uncovered No overlapping of mandals Information shared across partners Avahan in AP: disrupting the counterfactual

16. Supporting community action on AIDS in developing countries 74% 70% 99%98% 7 6 23 22 17 23 5 10 40 50 17 23 0% 20% 40% 60% 80% 100% NFPPFPPNFPPFPP Baseline Follow-up Condom last clientCondom regularSyphilisHSV-2 Results 1: India FSW

17. Supporting community action on AIDS in developing countries 45% 58% 96% 91% 12 15 29 41 22 26 8 9 27 37 14 13 0% 20% 40% 60% 80% 100% NFPPFPPNFPPFPP Baseline Follow-up Condom lastCondom femaleSyphilisHSV-2 Results 2: India MSM

18. Supporting community action on AIDS in developing countries Key challenges for the evaluation (1) Ethical concerns Control sites: ‘withholding’ interventions from those who need them Response: Limited funding only allowed saturation of 20 sites. Biomarkers: biomarker samples and HIV Response: Respondents provided with health benefit. HIV testing was not done. (A mistake?)

19. Supporting community action on AIDS in developing countries Key challenges for the evaluation (2) Consent and confidentiality with KPs Over-sensitisation towards KP-related issues led to researchers being overly guarded ‘Over-dependence’ on KPs made the process stilted and artificial. In some instances, KPs used their ‘KP status’ to complicate an otherwise simpler process which could have been more effective.

20. Supporting community action on AIDS in developing countries Key challenges for the evaluation (3) Striving for inclusiveness Challenge of accommodating differing opinions and approaches within the community Time-consuming Questions around the reliability of number of KPs generated through the PSA

21. Supporting community action on AIDS in developing countries Key challenges for the evaluation (4) Management issues Multiple stakeholders with different expectations, opinions and agendas High staff turnover Coordination and leadership Geographical lack of proximity of research partners Implementers as researchers created bias and lack of objectivity

22. Supporting community action on AIDS in developing countries Key challenges for the evaluation (5) Maintaining control sites Avahan (2003) initiated comparable interventions with similar set of organisations (IHAA included) Control sites were lost as a result. Lack of adequate coordination The research design was not flexible or sufficiently nimble to accommodate the changes.

23. Supporting community action on AIDS in developing countries SITE B SITE A SITE C FPP Interventions Adapting the evaluation: dose response

24. Supporting community action on AIDS in developing countries ‘Continuum of interventions’ ‘Comparison sites’ in Ecuador FPP full sites In India & Ecuador IAI sites in India Increased site level: a) ‘social capital’ b) +ve behaviour change c) +ve biomarker Measurements of program intensity and population coverage at site Dosage evaluation for community interventions

25. Supporting community action on AIDS in developing countries “These positive results, even considering the limitations of the evaluation design, suggest that a strong community component may significantly potentiate prevention impact. It is all the more convincing because over time Avahan became increasingly focused on community mobilization. While this supports the arguments in favour of community participation, it also calls for more robust evaluation in the future to characterize and quantify the benefits and costs of different approaches for community engagement and mobilization to accompany the provision of prevention services” Gutierrez JP., McPherson et al, (2010), Community-based prevention leads to an increase in condom use and a reduction in sexually transmitted infections (STIs) among men who have sex with men (MSM) and female sex workers (FSW): the Frontiers Prevention Project (FPP) evaluation results, BMC Public Health (Page 10)

26. Supporting community action on AIDS in developing countries Conclusions & discussion points Maintaining a counterfactual is not easy –Contamination of sites –Understand the response-scape will evolve—be flexible For complex programs defining a generic intervention ‘logic model’ that is evaluable is hard –Multiple causal factors, multiple attributions Significant challenges in assessing multi-component interventions with modest effect. –Ability to distinguish the marginal effect of the ‘intervention’ –Cost of these types of evaluations –Big investment in monitoring and surveys –Maintaining objectivity –Coordination and leadership

27. Supporting community action on AIDS in developing countries Conclusions & discussion points (continued) Be very careful when undertaking ‘impact’ evaluations Look at ‘dosage evaluation’ approaches Work in synergy with other NGOs and donors to evaluate ‘key development question’ evaluations (don’t try to ‘prove’ attribution) Measure output/process data that we KNOW has proven impact Focus on measuring efficiency and quality of interventions

28. Supporting community action on AIDS in developing countries Thank you!

29. Supporting community action on AIDS in developing countries Acknowledgements Fiona Samuels (ODI) JP Gutierrez (INSP) Stef Bertozi (INSP/Gates) Lalit Donanda (ASCI) AP Office staff (Alliance) Key Population groups Jeff O’Malley and Jerker Edstrom (ex Alliance)