Impact Evaluation Conference Wageningen March 25, 2013 “Are the children better off”, a large scale concurrent evaluation in India Pam Baatsen, Senior Advisor SRHR and HIV, KIT

Amsterdam, The Netherlands Presentation overview 1.Purpose of the IE? 2.The intervention being evaluated? 3.Context? 4.Impact evaluated/findings? 5.Design IE and its validity? 6.Communication around findings? 7.Utilization of findings?

1. Purpose of the IE? Evaluation framework Amsterdam, The Netherlands

2. The intervention? Districts using a phased approach 40,000 households with over 73,000 children Implementation by a consortium of 3 partners, coordination by government Overall coordination Community- based HIV care and support services Government facility-based clinical services Food Security/ Safety Net KIT/Swasti Evaluation

Amsterdam, The Netherlands The Programme and Approach Children and families infected & affected Psychosocial Safe ty net NutritionEducation Health To improve the quality of life of children and families infected with and affected by HIV IMPACT: Decreased mortality of children living with HIV Decreased morbidity among children and parents with AIDS Decreased number of children orphaned by AIDS Decreased number of children infected by HIV FAMILYCASEMANAGEMENTFAMILYCASEMANAGEMENT

Amsterdam, The Netherlands 3. The context HIV highly stigmatized Regular unrest and insecurity Geographical and cultural differences Numerous changes in leadership within the State Absence of size estimates on # of children infected with or affected by HIV ½ state covered by this intervention and other 1/2 covered by another intervention Numerous other smaller interventions in all districts New antiretroviral therapy treatment guidelines over the course of the project Transitioning to government (after 5 yrs) donor requirement

Amsterdam, The Netherlands 4. Impact evaluated? Has the quality of life (QoL) of children infected and affected by HIV, and their households been improved? Measured through: Changes in QoL scores; Body Mass Index (BMI), Mid Upper arm Circumference (MUAC); trait hope scale; morbidity (outcome HIV tests, CD-4 count levels) and mortality (of children and or their parents) And comparison between districts of this intervention versus other intervention

Amsterdam, The Netherlands 4. Impact evaluated and some findings…

Amsterdam, The Netherlands Mortality rate CLHIV higher than all children in the programme (4.6% versus 1%), However life of children living with HIV prolonged by the programme, including through successful earlier testing for HIV 4. Impact evaluated and some findings…

5. Design IE Intensified Household tracking survey

Amsterdam, The Netherlands Validity design? Assumption 50% difference in the QoL of those reached versus those not Generalization of survey findings to all the districts due to change in intervention protocol MUAC and BMI measurement not possible MIS important pillar for info (including mortality and morbidity) The decision to collect QoL data on the basis of different measurements, sometime contrasting findings but able to get overall picture Comparison state level data only possible for one (process) indicator

Amsterdam, The Netherlands 6. Communication around findings? Great effort to establish an evaluation core group at the onset of the evaluation to facilitate communication, and increase buy-in to the evaluation and its results Some guiding principles of evaluation protocol: Credit to all those involved Sharing of findings for action learning Sharing of the results through regular meetings; reflection action learning workshops; and evaluation core group meetings Publication protocol: representatives of all parties to be involved. Reasons: 1) HIV data sensitive; 2) data could have a (negative) impact on the intervention and 3) contractual arrangements.

Amsterdam, The Netherlands 6. Communication around findings? Policy briefs prepared by evaluators – first round, however, these were not shared, as the INGOs were disappointed with the nil-finding. Also the end line findings, did not show the - by the implementing agencies - desired results: sharing of the final results sensitive Findings shared with everyone involved in intervention, including at district level through a Reflection, learning and action workshop. However, it was left to the project implementers to share the findings with other key stakeholders in the state.

Amsterdam, The Netherlands 7. Utilization of findings? The IE has influenced the programme itself, to for instance better define the role of the outreach workers. Final results not shared with a large audience, due to the sensitivity of the findings. Efforts made to reach agreement on the publishing of an article with the main findings, but to date this has not yet been successful. KIT has advocated for the need to give more focus to younger children infected and affected by HIV on different occasions