A new way of impact evaluation Who values Whose values Value for what

Contents Why do we do this? –Shortcomings of current evaluation methods How do we do this? Some first findings

Shortcoming 1 Mostly not on impact Focus on ongoing projects Focus on implementation (‘did we do well?’) Focus on direct results: –Outputs: ‘What did the project produce’? –Outcomes: ‘How did the target group make use of the outputs of the project?’ –Impact: ‘What are the sustainable results in the communities?’

Shortcoming 2 Lack of attention for context (even in impact assessments) What did other projects do? External factors?

backdonor INGO NGO Project a In community x Activities Results Outputs Outcomes Impact M&E

Project c Project b backdonor INGO NGO 1 Project a Community x Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Private initiatives Government Local State National Companies (e.g. telecom) Other INGO’s Other backdonors

Project c Project b backdonor INGO NGO 1 Project a Community x Very poor – poor – average – rich – very rich Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Government Local State National Companies (e.g. telecom) Private initiatives Other INGO’s Other backdonors Local influences National influences Global influences Natural Physical Economic Human Social Political Cultural

Community x Very poor – poor – average – rich – very rich Government Local State National Companies (e.g. telecom) Private initiatives Project c Project b backdonor INGO NGO 1 Project a Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Other INGO’s Other backdonors Local influences National influences Global influences Natural Physical Economic Human Social Political Cultural

Project c Project b backdonor INGO NGO 1 Project a Community x Very poor – poor – average – rich – very rich Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Government Local State National Companies (e.g. telecom) Private initiatives Other INGO’s Other backdonors Local influences National influences Global influences Natural Physical Economic Human Social Political Cultural

Shortcoming 3 Often mainly based on expert opinion –What the expert defines as ‘impact’ Often looking for predetermined results only –Little attention for unexpected effects

Shortcoming 4 If evaluations look at target group perceptions, there is almost always bias “yes sir, we like the project and we want some more”

Shortcoming 5 Focus on ‘what’ and little on ‘how’, ‘with which values’ or ‘why’ projects were done

How do we do this? Prisma with ICCO and Woord en Daad with University of Amsterdam and University of Development Studies and Expertise Développement Sahel In Burkina Faso and Ghana

backdonor INGO NGO Project a In community x Not this perspective backdonor INGO NGO Project a In community x backdonor INGO NGO Project Project a In community x

Community x Very poor – poor – average – rich – very rich But this perspective ProjectsActors Changes in context

3 day workshops 60 people from area of 20,000 Subgroups: men, women, old, young (officials and project staff separate group) Individual life history questionnaire –+ data about parents –+ data about brothers / sisters –+ data about children (total about 600 persons per workshop) 4 rounds of 3 workshops each

Part 1: context “What were major events?” “What are changes over past 30 years? Are these positive or negative” (natural, physical, human, economic, social/political, cultural) “Who are considered the –Very rich –Rich –Average –Poor –Very poor by local standards?”

Part 2: interventions “Which interventions happened?” (by government, private sector, secular ngo’s, christian ngo’s / churches, muslim ngo’s, individuals) “How do you assess their impact?” (negative impact, no impact, impact in past, positive impact) + explanations !!!

Part 3: Best and Worst “What were best 5 and worst 5 projects? and why?” “What did you tell your friends about these projects when they started? And what do you tell your friends about them now?” “What is the effect of these projects on the 5 wealth classes?” “What is the effect of these projects on the different domains?” “What is the link with the changes in context?” (cause / mitigation)

And then 60 small reports per workshop Analyse, compare and combine all subgroups

First findings Incredibly detailed historical profile Big picture: –Natural domain: negative –Human domain (education / health): positive –Socio/political: mixed –Cultural: mixed –Economic: ‘we have more but we feel poorer’ Wealth categories: –rich anthropological descriptions (meals, behaviour, jobs, funeral, family life, properties, etc.) –Very different per region

Numbers of Projects

Best projects by actor

Worst projects by actor (Ghana) But different in Burkina

Effects best pr. on Wealth Classes (Sandema, Ghana)

Effects best pr. on Wealth Classes (Tô, Burkina Faso)

Reasons exclusion very poor Miss preconditions for benefiting Knowledge about the existence of an intervention Access to the place where selection takes place Having time available for themselves (or their children in case of education) to take part in the activities of an intervention Having the feeling that they are allowed to participate (e.g. feeling that their dresses are sufficiently appropriate to be part of a meeting) Being involved in the activities at which the intervention is focused (e.g. veterinary services will not benefit the very poor in some societies, since they do not have livestock)

Reasons for best/worst ‘Hard results’ Relevance Manner in which projects are done –Respect –Mutual trust (‘we can trust them’ and ‘we feel that they trust us’) –Participation (decision making, monitoring) –Long term commitment (‘they have us at heart’) –Honesty (‘something went wrong and they did not explain to us’) –Local presence (‘they are one of us’) Results / expectations ratio –Expectations about length of stay (‘they left and never came back’) –Expectations about results (‘poverty will be ended here’) Negative side effects

Some final conclusion Projects have many side effects Best projects have effects in more than 1 domain (often including economic) There are no black and white conclusions –Always nuances and exceptions –Christian among best and worst –Differences between men / women, old / young Some of the biggest negative changes are hardly addressed –Climate change –Loss of cultural identity –Negative sides of individualism Men, Women and Project officers all have different opinions

Questions ?