Assessing Humanitarian Performance: Where are we now? 24 th Biannual Meeting Berlin, 3 rd December 2008

24th ALNAP Biannual Meeting, December Various strands of ALNAP work are working towards assessing system- wide performance  Various components of RHA - evaluation synthesis, meta-evaluation (especially on joint evaluations) and themed chapters  Facilitation of TEC and discussions on recommendations  HPP - data mapping and exploratory analysis of how to assess system-wide performance

24th ALNAP Biannual Meeting, December What have we learned from the RHA?  Evaluation synthesis useful but, on its own, not able to assess performance  Component parts of RHA are good but the final product is probably less than the sum of its parts  Need to strengthen methodology and produce a more coherent whole

24th ALNAP Biannual Meeting, December What have we learned from the TEC?  joint evaluations better than single-agency evaluations in providing system-wide snapshot  system-wide joint evaluations provide one off picture only  utilisation and take up of recommendations very difficult to achieve in practice

24th ALNAP Biannual Meeting, December What have we learned from HPP?  Lots of data collected, but of different types, from different sources with different uses.  Many methodological and conceptual difficulties  Majority data gathered in needs assessment phase  Very little effort given to seeking the views of affected populations/ recipients of aid

24th ALNAP Biannual Meeting, December What have we learned from the Madrid biannual? a)Be realistic as to what can be achieved now, use existing evidence to assess performance, with a special emphasis on ‘impact’ b)Explore the use of beneficiary surveys in assessment of impact and performance c)Develop a ‘pilot’ to test these ideas d)Continue mapping and do not lose sight of developing a more precise way of assessing performance

24th ALNAP Biannual Meeting, December What are we going to do? The three-track approach:  Track one - fast track ‘State of the System’ pilot  Track two - medium track Learn more about use of beneficiary surveys and impact assessment and feed this into future ‘State of the System’ reports  Track three - slow track Continue mapping and work on developing key performance indicators

24th ALNAP Biannual Meeting, December Track One: State of the System Report What is it for?  Overall goal is to assess overall humanitarian performance against agreed criteria  Pilot will provide a ‘base-line’ to track future performance

24th ALNAP Biannual Meeting, December What problems will we face?  Analysing a system that is not strictly a system (i.e., not systematic)  Lack of data relating to outcomes and indicators

24th ALNAP Biannual Meeting, December How will we address the problems? a) Break down the system into different units of analysis. Disaggregate the data by looking at: state of response in individual crisis state of response in particular sectors (clusters) state of response in particular categories – natural disasters, wars, high profile crisis, neglected crises state of response in relation to types of actors UN, NGOs, donors, governments etc

24th ALNAP Biannual Meeting, December b) For each unit of analysis, performance will be analysed in relation to OECD-DAC criteria: Relevance/Appropriateness Connectedness Coherence Coverage Efficiency Effectiveness Impact

24th ALNAP Biannual Meeting, December c) Need to identify indicators to apply to OECD- DAC criteria. For example: was coverage adequate: are resources adequate global funding against needs (CAP, beneficiary surveys) funding across sectors and emergencies staffing coverage in key areas and so on..

24th ALNAP Biannual Meeting, December What do we want the report to tell us?  emerging themes  trends – how has sector y or response in emergency x changed over time  innovations and changes  performance indicators if/when they exist  perceptions of informed stakeholders about effectiveness, impact etc

24th ALNAP Biannual Meeting, December What methods shall we use? Building on the RHA  key informant interviews (NGO’s, donors, government) aiming for mix of HQ and field  financial data analysis – (OECD-DAC and FTS)  mapping of global footprint and across current emergencies  key informant survey – polling opinion about performance to provide base line  evaluation synthesis  literature review

24th ALNAP Biannual Meeting, December What now?  establish peer review advisory panel  undertake preliminary interviews/ consultations or input on scope and objectives of pilot  design detailed methodology and research plan in inception report to be peer reviewed