1. Humanitarian aid evaluation at Medecins sans Frontieres
Sabine Kampmüller, MIH
MSF Vienna Evaluation Unit

2. Dimensions of evaluation:
Tool to describe the effectiveness of a program, measure etc.
The goal of a logic model is to give an explicit description of the attended link between the inputs (what we invest), the performances/ outputs (what we do), the reactions of the target groups and the consequences for the affected people
Such a description is helpful, because a lot of assumptions concerning the efficiency of performance are implicit
External influences (positive and negative) can be taken in consideration
A logic model allows the evaluator to identify goals and indicators to measure the effects of a programme
Source: SDC (2002)

3. Evaluation post-disaster: Haiti earthquake response
Dramatic and chaotic situation
Massive mobilisation of aid, media, military
Heath sector response largely uncoordinated during the first weeks
Needs overwhelming:
Dead, Injured and traumatised
One million and a half homeless
Extreme conditions in IDP camps and streets
Vulnerability for disases, outbreaks, malnutrition and violence
Evaluation conducted
5 – 10/2010

4. Evaluation process 1st time (!) all section Review
6 specific reviews: Global/operational, medical/surg, log/supply, Comms, FR, HR
Quantitative & qualitative methods
data, field visits, interviews, qualitative research - beneficiaries, web survey -staff
Limitations:
Incomplete and non-uniform data
Recall problems
Sabine
ToR f specific reviews by teh specific platforms

5. First line emergency and outpatient services
Consultations 123,108
Dressings 34,044
Antenatal care consultations 8,353
Victims of sexual violence 38 (?)
Total 165,543
Surgery
Major surgical interventions 6,259
Other surgical interventions5,903
Total12,162
Inpatient care
Admissions for Surgery1,243
Maternity3,425
Medical1,982
Paediatrics1,132
Total 7,782
Postoperative care 2,604
Physiotherapy sessions 10,241
Mental Health
Individual consultations 14,765
Group sessions 4,310

6. Patients / Community’s Perception
Very positive perception of MSF´s interventions and services
Fear/experience of stigmatisation, increase of violence in the camps 
Lack of space for occupational and educational activities.  
More attention to socio-cultural and spiritual needs
Others actors
Sabine

7. Haiti evaluation findings:
MSF was one of the biggest emergency health actors
Operational choices in line with emergency needs; less coherent over time
Common data collection difficult to impossible
International platforms / working groups are underused / undervalued
Successful adocacy on some issues, complicated decision making prevented more
Even if all the surgical referents have been in Haïti
Assessment: done at the beginning nothing or near nothing apart OCP and OCG later for needs population

8. Main recommendations:
Organize inter-section capacity for assessment and monitoring of evolving needs and assistance
Define strategy on mass casualty
Revise emergency supply strategies
Ensure uniform data collection
Strengthen technical working groups
Focus on organisational learning
Engage with national and international actors

9. Thematic Evaluation: Response to displacement
Dramatic and chaotic situation
Massive mobilisation of aid, media, military
Heath sector response largely uncoordinated during the first weeks
Needs overwhelming:
Dead, Injured and traumatised
One million and a half homeless
Extreme conditions in IDP camps and streets
Vulnerability for disases, outbreaks, malnutrition and violence
Evaluation conducted
2009/2010

10. Evaluation process
Comparison of 6 case studies (urban/rural, low/middle income, etc)
Quantitative & qualitative methods
Literature research, document/tools & data review, field visits, interviews,
Limitations:
Availability of data and key people
Poor documentation
Sabine
ToR f specific reviews by teh specific platforms

11. Particularity of open settings
Lack of clear boundaries :
Geographical spread, invisibility, needs difficult to identify and measure, protection issues
Displaced settle in environment with available resources and existing health system
Better survival capacities, but deterioration likely
Protracted, chronic or intermittent character:
Mortality near-normal levels (pre-emergency), might rise slowly over time
Invisibility due to legal status or mixity, as result of progressive break down of health infrastructure and exhausted coping

12. Findings on assessment
Complexity of open settings requires more attention to assessment
Critical aspects of information missing
Quantitative information difficult to obtain
Health system issues, access barriers little addressed
Concerns on the use of surveys
Views of displaced and host often omitted
Capacities and vulnerabilities not assessed
Same reflexes than in camp like situations: look for quantitative data, mortality measured in 3 of the vcountries, surveillance only in cameroon, not very effective, no validity of quaitative methods without triangulation

13. Recommendations on assessment
Develop innovative assessment approaches for inaccessible areas
Distance assessment“
Promote systematic use of qualitative methods
Adopt concept of “continual” assessment
Develop a frame to assess vulnerabilities, capacities and coping
Provide better support and guidance
assessment toolbox, experts, training

14. Findings on intervention
Engagement with the health system a main challenge
Outreach workers invaluable, set ups improvable
Non-medical assistance is marginal
Overambitious targets of coverage
Strategy adapted to the level of emergency
(top ten priorities to reduce excess mortality), parallel health system

15. Recommendations on intervention
Need for new intervention frameworks
Adopt existing models
Generalize the practice to cover both displaced and host where appropriate
Define the criteria / scope of “light support”
Develop community based strategies

16. Evaluation criteria (donors)
Appropriateness
(Connectedness)
(Coherence)
Timeliness
Coordination
Coverage
Relevance
Effectiveness
Efficiency
Impact
Sustainability
Adopted from: Hallam, A. Good practice review (ODI), 1998

17. Ways to ensure UTILISATION
Get original purpose very clear
Ensure ownership, participatory process
Get key stakeholders behind
Share findings in the field
Allow debate with evaluators
Ensure good readibility of report
Disseminate info widely / on all levels
Promote credibility of evaluators
Use in training
Formulate recommendations clear and realistically

18. 10 Steps in the process Defining purpose and scope
Writing the terms of reference
Analysing stakeholders
Choosing the methodology
Deciding the budget for evalution
Selecting the evaluation team
Preparing the field
Research phase
Reporting and dissemination
Managment follow-up 18