1. Evaluation and impact assessment David Souter

2. BCO Impact Assessment session, Bern, September 2007 Why evaluate?  It is dangerous to assume that what has been decided will be achieved … … or that what happens is what was intended.  Accountability : to report to stakeholders on achievements against objectives  Learning : to improve practice in future, particularly programme design and implementation  Promotion : to demonstrate success to potential donors

3. BCO Impact Assessment session, Bern, September 2007 Why impact assessment?  “The purpose of most impact assessments is to demonstrate past impact and to improve future practice, and there may be tension between the two.”  “Proof of effectiveness is not the main goal of impact assessment: it is concerned more with understanding, and learning from, the processes and approaches that have led to achieving a particular impact so that such knowledge and experience could be used in comparable situations.”

4. BCO Impact Assessment session, Bern, September 2007 A hierarchy of outcomes : the MDG perception

5. BCO Impact Assessment session, Bern, September 2007 Output, outcome/effect and impact  Output measures the extent to which an activity’s inputs have been made and services deployed  Outcome (or effect) measures the use of inputs/services and the short-term results arising from these  Impact measures the long-term changes in people’s lives, social behaviour and economic and political structures that can be attributed – in whole or part – to the activity

6. BCO Impact Assessment session, Bern, September 2007 Activities A hierarchy of complexity Inputs Outputs Outcomes or Effects Impact Increasing influence of context (social, economic, political, etc.) Source: Roche

7. BCO Impact Assessment session, Bern, September 2007 An example:

8. BCO Impact Assessment session, Bern, September 2007 What is impact assessment?  “Impact assessment refers to an evaluation of how, and to what extent, development interventions cause sustainable changes in living conditions and behaviour of beneficiaries and the differential effects of these on women and men. “Impact assessment also refers to an evaluation of how, and the extent to which, development interventions influence the socio-economic and political situation in a country.”  Peter Oakley et al., Outcomes and Impact

9. BCO Impact Assessment session, Bern, September 2007 What is impact assessment?  “Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions.”  Novib/Oxfam research project, reported in C. Roche, Impact Assessment for Development Agencies

10. BCO Impact Assessment session, Bern, September 2007 What is impact assessment?  Impact assessment is the  systematic analysis of  the lasting or significant changes –  positive or negative,  intended or not –  in people’s lives  brought about by  a given action or series of actions.

11. BCO Impact Assessment session, Bern, September 2007 Why assess impact?  Impact assessment “is concerned … with understanding, and learning from, the processes and approaches that have led to achieving a particular impact so that such knowledge and experience could be used in comparable situations.”  Impact assessment should both:  look back at experience to date  look forward to improving programme design and implementation, and in helping to predict future impact

12. BCO Impact Assessment session, Bern, September 2007 How do we assess impact? : a view Stages of M&E developmentDegree of attention DesignExtensive ImplementationSubstantial Data generationMore Data interpretationLess UseSome EvaluationMinimal R. Davies, assessment of Bangladesh projects, 1995

13. BCO Impact Assessment session, Bern, September 2007 Causation is complex

14. BCO Impact Assessment session, Bern, September 2007 Impact is long-term

15. BCO Impact Assessment session, Bern, September 2007 The challenges of impact assessment  A two tier process  The challenge of context  The challenge of the baseline  The challenge of understanding change  The challenge of time  The challenge of attribution  The challenge of aggregation  The challenge of unexpected impact  The challenge of stakeholders  The challenge of perception  The challenge of distribution

16. BCO Impact Assessment session, Bern, September 2007 Two tiers in establishing impact  that it has had an effect on the behaviour of actors relating to the intervention itself  that this effect has led to change in the circumstances of those actors  that it has had an effect on policy  that the policy change concerned has resulted in change in people’s lives To establish that an intervention has had impact, it is necessary to establish two things: Where advocacy initiatives are concerned, this can be described as follows:

17. BCO Impact Assessment session, Bern, September 2007 The challenge of context  Impact measures change in context  Understanding context is therefore essentially to identifying impact  Context is complex:  social, gender, cultural, political, economic, environmental etc. – and technological  contextual factors inter-relate  Context is changing  Context can be disrupted

18. BCO Impact Assessment session, Bern, September 2007 The challenge of the baseline  ‘… without some sort of baseline, it is not really possible to assess impact.’  ‘… unless detailed and continuous M&E takes place at [programme and project] level, it will be impossible to build up the body of knowledge which is necessary to understand the wider outcomes.”  Measuring change requires understanding of the starting point from which change is measured  Most impact assessment suffers from the lack of baseline data  Impact assessment needs to be built into processes from the beginning stage if it is to be undertaken meaningfully  Measures can be taken to try to recover baseline, but these are difficult and relatively weak

19. BCO Impact Assessment session, Bern, September 2007 The challenge of change  Impact assessment is essentially the measurement of change  Change – particularly social change – results from a large number of diverse and differently inter-related factors  It is subject to unexpected externalities  It may be ‘sudden, discontinuous and unpredictable’  Chance is a significant factor in change processes  Understanding change requires an in-depth understanding of:  context  and the baseline from which change is to be measured

20. BCO Impact Assessment session, Bern, September 2007 The challenge of time  Lasting change cannot be established quickly.  Short-term impact assessments can identify near-project effects, rather than lasting change  Short-term assessments also lack perspective regarding the complexity of change  Impact varies over time:  short-term impacts can be few and differ from long- term impacts  and vice versa

21. BCO Impact Assessment session, Bern, September 2007 The challenge of attribution  Just because change occurs, it does not mean you caused it.  Many factors influence change: many people may wish to claim credit.  Attribution of change is complex.  There is a temptation to claim credit in order to emphasise the value of work done …  … but this does not help to understand what has been (or not been) achieved, and so does not contribute to the quality of future work

22. BCO Impact Assessment session, Bern, September 2007 The challenge of aggregation  Individual interventions usually target specific aspects of change  Other interventions target similar and related aspects of change  Change may result from the cumulative impact of numerous interventions  “the sum is more than the total of the parts”  implying that it is the “sum” rather than the “parts” that drives the impact

23. BCO Impact Assessment session, Bern, September 2007 The challenge of unexpected impact  All initiatives have unexpected impacts, e.g.  diversion of resources  changes in power relationships or the distribution of wealth  redistribution of income to other purposes  which may be positive or negative  It is as important to understand these unexpected impacts as to understand whether intended impacts are achieved  Negative impacts tend to be under-reported, with the result that important lessons are not learnt for future programme design and implementation

24. BCO Impact Assessment session, Bern, September 2007 The challenge of identifying stakeholders (illustrative) Donors Non-bene- ficiaries Local inter- mediaries Political authorities Other development initiatives Project managers Target beneficiaries Age group Education Caste or class Land- holding Wealth Gender Level of use

25. BCO Impact Assessment session, Bern, September 2007 The challenge of perceptions  Different stakeholders have different perceptions of impact.  In particular:  target beneficiaries may perceive impact differently from project managers  target beneficiaries are likely to focus on individual, family and community benefits  project managers are likely to have broader objectives for socio-economic or political change  non-beneficiaries are likely to perceive change differently from beneficiaries  Enabling different stakeholders to express divergent views of impact is crucial to successful impact assessment

26. BCO Impact Assessment session, Bern, September 2007 The challenge of distribution  Impacts affect non-target beneficiaries as well as target beneficiaries  Need to establish whether impact is favourable to target beneficiaries  and also if it is favourable to target beneficiaries and  neutral otherwise  favourable to the community in general  unfavourable to others or to the community in general  favourable to target beneficiaries at the expense of others in the community or neighbouring communities  Non-target beneficiaries may be reticent about negative impacts on themselves  e.g. women may be reticent about changes in the balance of power within households

27. BCO Impact Assessment session, Bern, September 2007 Impact assessment and advocacy  The impact of advocacy work requires both:  that advocacy leads to policy change and  that policy change leads to changes in people’s lives  Advocacy work requires a wide range of interactions, many of which will be unproductive  Much advocacy work is long-term and may have few short-term effects  Much advocacy work is undertaken through networks in which different actors play different roles  Impact assessment requires particular attention to aggregation issues

28. BCO Impact Assessment session, Bern, September 2007 Oxfam’s hierarchy of advocacy impact Heightened awareness Contribution to debate Change in opinions Change in policy Implementation of change in policy Positive change in people’s lives

29. BCO Impact Assessment session, Bern, September 2007 Impact assessment as process Diagnostics where are we now? Strategy design where do we want to go? Development and implementation how do we get there? Monitoring and evaluation how do we know we are getting there? Impact assessment needs to take place throughout the project cycle

30. BCO Impact Assessment session, Bern, September 2007 The impact assessment cycle  Understanding of context (diagnostics)  Development of baseline and indicators, which should be:  achievable and adequate  capable of providing data re inputs, outputs, effects and impact  Monitoring of indicators  including regular assessment (and storage)  Analysis  Interpretation  Input into subsequent project cycles

31. BCO Impact Assessment session, Bern, September 2007 Design of IA processes  Establishment of:  context within assessment is to be made (impact where  scope and scale of assessed intervention (impact of what)  level and target (impact on whom, at individual, household or community level, etc.)  timescale (impact when)  interaction with other influences/factors  selection of illustrative examples

32. BCO Impact Assessment session, Bern, September 2007 IA process requirements  Commitment to learn from experience  Adequate baseline data and indicators  A management framework which recognises real levels of resources  Continuous engagement and continued commitment  Evaluative rigour  Determination to complete the process  Honesty in interpreting results

33. BCO Impact Assessment session, Bern, September 2007 Impact assessment of ICD projects  Impact assessment of ICD projects does not obviously differ in principle from impact assessment in other development areas  It should therefore gain from sharing experience with the wider development community  However, there are some aspects of ICD projects which would seem to exacerbate the challenges of impact assessment:  the lack of baseline data  the very limited range of existing substantive impact assessment  the unpredictability of change in ICT technology and behaviour  uncertainty about the implications of different policy options

34. BCO Impact Assessment session, Bern, September 2007 Ways forward  There is a good deal of experience of impact assessment in other development sectors on which to draw  This experience emphasises:  development of baselines  continuous monitoring and assessment  stakeholder participation  exploration of unexpected and negative outcomes  openness to diverse conclusions

35. BCO Impact Assessment session, Bern, September 2007 Ways forward  “Lessons are more likely to be about the relative merits of various strategies in addressing specific audiences, as well as different sequences of tools and methods, than about developing a single model of change or assessment.”  C. Roche, Impact Assessment for Development Agencies

36. BCO Impact Assessment session, Bern, September 2007 The opportunity for BCO  The BCO Impact Assessment offers BCO partners an opportunity to:  learn from the experience in other development sectors  apply that experience to their own experience with ICD projects  develop approaches to impact assessment which have specific relevance to ICD, for their own future use and the wider ICD community