Humanitarian aid evaluation at Medecins sans Frontieres Sabine Kampmüller, MIH MSF Vienna Evaluation Unit http://evaluation.msf.at

Dimensions of evaluation: Tool to describe the effectiveness of a program, measure etc. The goal of a logic model is to give an explicit description of the attended link between the inputs (what we invest), the performances/ outputs (what we do), the reactions of the target groups and the consequences for the affected people Such a description is helpful, because a lot of assumptions concerning the efficiency of performance are implicit External influences (positive and negative) can be taken in consideration A logic model allows the evaluator to identify goals and indicators to measure the effects of a programme Source: SDC (2002)

Evaluation post-disaster: Haiti earthquake response Dramatic and chaotic situation Massive mobilisation of aid, media, military Heath sector response largely uncoordinated during the first weeks Needs overwhelming: Dead, Injured and traumatised One million and a half homeless Extreme conditions in IDP camps and streets Vulnerability for disases, outbreaks, malnutrition and violence Evaluation conducted 5 – 10/2010

Evaluation process 1st time (!) all section Review 6 specific reviews: Global/operational, medical/surg, log/supply, Comms, FR, HR Quantitative & qualitative methods data, field visits, interviews, qualitative research - beneficiaries, web survey -staff Limitations: Incomplete and non-uniform data Recall problems Sabine ToR f specific reviews by teh specific platforms

First line emergency and outpatient services Consultations 123,108 Dressings 34,044 Antenatal care consultations 8,353 Victims of sexual violence 38 (?) Total 165,543 Surgery Major surgical interventions 6,259 Other surgical interventions5,903 Total12,162 Inpatient care Admissions for Surgery1,243 Maternity3,425 Medical1,982 Paediatrics1,132 Total 7,782 Postoperative care 2,604 Physiotherapy sessions 10,241 Mental Health Individual consultations 14,765 Group sessions 4,310

Patients / Community’s Perception Very positive perception of MSF´s interventions and services Fear/experience of stigmatisation, increase of violence in the camps  Lack of space for occupational and educational activities.   More attention to socio-cultural and spiritual needs Others actors Sabine

Haiti evaluation findings: MSF was one of the biggest emergency health actors Operational choices in line with emergency needs; less coherent over time Common data collection difficult to impossible International platforms / working groups are underused / undervalued Successful adocacy on some issues, complicated decision making prevented more Even if all the surgical referents have been in Haïti Assessment: done at the beginning nothing or near nothing apart OCP and OCG later for needs population

Main recommendations: Organize inter-section capacity for assessment and monitoring of evolving needs and assistance Define strategy on mass casualty Revise emergency supply strategies Ensure uniform data collection Strengthen technical working groups Focus on organisational learning Engage with national and international actors

Thematic Evaluation: Response to displacement Dramatic and chaotic situation Massive mobilisation of aid, media, military Heath sector response largely uncoordinated during the first weeks Needs overwhelming: Dead, Injured and traumatised One million and a half homeless Extreme conditions in IDP camps and streets Vulnerability for disases, outbreaks, malnutrition and violence Evaluation conducted 2009/2010

Evaluation process Comparison of 6 case studies (urban/rural, low/middle income, etc) Quantitative & qualitative methods Literature research, document/tools & data review, field visits, interviews, Limitations: Availability of data and key people Poor documentation Sabine ToR f specific reviews by teh specific platforms

Particularity of open settings Lack of clear boundaries : Geographical spread, invisibility, needs difficult to identify and measure, protection issues Displaced settle in environment with available resources and existing health system Better survival capacities, but deterioration likely Protracted, chronic or intermittent character: Mortality near-normal levels (pre-emergency), might rise slowly over time Invisibility due to legal status or mixity, as result of progressive break down of health infrastructure and exhausted coping

Findings on assessment Complexity of open settings requires more attention to assessment Critical aspects of information missing Quantitative information difficult to obtain Health system issues, access barriers little addressed Concerns on the use of surveys Views of displaced and host often omitted Capacities and vulnerabilities not assessed Same reflexes than in camp like situations: look for quantitative data, mortality measured in 3 of the vcountries, surveillance only in cameroon, not very effective, no validity of quaitative methods without triangulation

Recommendations on assessment Develop innovative assessment approaches for inaccessible areas Distance assessment“ Promote systematic use of qualitative methods Adopt concept of “continual” assessment Develop a frame to assess vulnerabilities, capacities and coping Provide better support and guidance assessment toolbox, experts, training

Findings on intervention Engagement with the health system a main challenge Outreach workers invaluable, set ups improvable Non-medical assistance is marginal Overambitious targets of coverage Strategy adapted to the level of emergency (top ten priorities to reduce excess mortality), parallel health system

Recommendations on intervention Need for new intervention frameworks Adopt existing models Generalize the practice to cover both displaced and host where appropriate Define the criteria / scope of “light support” Develop community based strategies

Evaluation criteria (donors) Appropriateness (Connectedness) (Coherence) Timeliness Coordination Coverage Relevance Effectiveness Efficiency Impact Sustainability Adopted from: Hallam, A. Good practice review (ODI), 1998

Ways to ensure UTILISATION Get original purpose very clear Ensure ownership, participatory process Get key stakeholders behind Share findings in the field Allow debate with evaluators Ensure good readibility of report Disseminate info widely / on all levels Promote credibility of evaluators Use in training Formulate recommendations clear and realistically

10 Steps in the process Defining purpose and scope Writing the terms of reference Analysing stakeholders Choosing the methodology Deciding the budget for evalution Selecting the evaluation team Preparing the field Research phase Reporting and dissemination Managment follow-up 18