Innovative Approach to Evaluating Interventions in Fragile and Conflict-Affected States: The Case of Helmand 2011 IDEAS Global Assembly Session 6

Contents 1.Introduction 2.Challenges to evaluation in the Helmand context 3.HMEP: a cutting-edge M&E solution 4.Conclusions: Future prospects for M&E in fragile states

1 - Introduction Economic context Sharpest spending cuts since WW2 Cuts worth around £128 billion over 4 years 30-40% cuts for some Departments DFID… Ring-fenced ! What does that mean for the future ? Extra scrutiny to ensure DFID’s lucky escape is justified Focus on Value for Money for every taxpayer £

1 - Introduction Context for Helmand Extra pressure for results-based M&E Pledges to withdraw forces can only be honored if stabilization successful Everyone is watching closely: General public, Media, MPs, Cabinet Office Need to go beyond outputs to look at outcomes and impacts Need to communicate information to multiple audiences quickly and effectively

2 - Challenges to Evaluation in Helmand Context These fall into 3 main categories: 1.Political 2.Logistical 3.Cultural

2 - Challenges to Evaluation in Helmand Context Political Challenges: Weakness, non-existence or fractured nature of government systems Institutional complexity of the operating environment in Helmand The pre-eminence of political and military considerations Targeted interventions not for universal benefit Pressure to deliver ‘good news stories’

2 - Challenges to Evaluation in Helmand Context Logistical Challenges: Data is almost always patchy and of poor quality Freedom of movement is significantly curtailed Association with ISAF puts enumerators’ lives at risk Negativity about the feasibility of conducting research in Helmand Difficulties in establishing institutional memory

2 - Challenges to Evaluation in Helmand Context Cultural Challenges: Social desirability bias particularly high in Helmand The difficulties associated with reaching women

3 - HMEP: a cutting-edge M&E solution Project Background: The Helmand Monitoring & Evaluation Programme Commissioned by DFID and the Provincial Reconstruction Team (PRT) in late key outputs –Establish baselines –An operational, up-to-date, user-friendly database and GIS database –New knowledge and recommendations –Improved programming capacity

3 - HMEP: a cutting-edge M&E solution Data Collection: Became apparent that discrete primary data collection would be required to populate the indicators developed alongside the logical frameworks. Attention was given to developing quantitative, SMART indicators Data collection methodology was designed taking into account the lessons learned

3 - HMEP: a cutting-edge M&E solution Carefully Designed Sampling Strategy : 2004 census data and the Central Statistics Organization’s (CSO) raw data on settlements posed several issues –District boundaries changed over time –Location of CSO settlements contained spatial coordinate errors –Population per settlement from the CSO data did not always reflect what could be seen on the ground through spatial imagery

3 - HMEP: a cutting-edge M&E solution Carefully Designed Sampling Strategy : The decision was taken to target males Heads of Households (HOH) Number of 10 people per household was assumed Profiling results from different waves of the same HOH survey will be compared with one another to establish HOH profiles Multi-staged random probability sampling process was used with a random route for selecting households in each sampling point The HOH survey is implemented by an Afghan survey partner

3 - HMEP: a cutting-edge M&E solution Robust Sample Sizes at Provincial and District Level : District Estimated number. of residents % of total Province population Estimated number of households per District Target number of interviews Confidence interval at the 95% confidence level Lashkar Gah87,0627%8, /- 5 Nahri-e-Sarraj176,85115%17, /- 5 Musa Qala129,42711%12, /- 5 Naw Zad87,0127%8, /- 5 Nad Ali103,0829%10, /- 5 Nawa-I-Barak Zayi85,4407%8, /- 5 Garmser99,1728%9, /- 5 Sangin60,3245%6, /- 5 Reg (Khanshin)16,1751%1, /- 5 Nad Ali (Marjah)108,6629%10, /- 5 Kajaki113,2289%11, /- 5 TOTAL1,066,61088%106,6614,011+/- 1.5

3 - HMEP: a cutting-edge M&E solution Longitudinal Approach Providing Confidence in Time-Series Analysis: Essential to ensure: –Consistent sampling point selection method –Continuity in the questionnaire rolled out each quarter Building a panel proved impossible in Helmand Repeated cross-sections modelling to enable the development of pseudo-panels ?

3 - HMEP: a cutting-edge M&E solution A Combined Quantitative and Qualitative Research : Qualitative research to complement the quantitative findings Capture views of women Explore the most puzzling trends resulting from the quantitative research Provide a more in-depth understanding of the ‘story’ and causal factors behind some of the less straightforward trends in perceptions

3 - HMEP: a cutting-edge M&E solution Analytical thinking and theory of change

3 - HMEP: a cutting-edge M&E solution

Presentational Features

3 - HMEP: a cutting-edge M&E solution Database & Website:

3 - HMEP: a cutting-edge M&E solution GIS Mapping:

3 - HMEP: a cutting-edge M&E solution Intelligent use of information: HMEP supports more intelligent use of information in assessments, planning and daily operations: –look back on a quarterly and annual basis on progress towards strategic goals –look forward in terms of planning and future strategic direction –supports PRT’s daily operations through improved management in the collation and centralisation of relevant data and its storage in a centralised and accessible information management system.

4 - Conclusions: Future prospects for M&E in fragile states HMEP’s specificity lies in its holistic approach to implementing an M&E system in a part of the world where it is crucially needed but difficult to achieve. Before assessing the replicability of the HMEP model, it is essential to understand what made it possible in the first place. Most important factor of success : Presence of a PRT Important to recognise that each province, region and country contains its own, specific set of contextual issues and challenges No Magic formula, but it proves that meaningful and sustainable M&E systems is possible even in the most difficult environments

END