Download presentation
Presentation is loading. Please wait.
Published byEustace O’Neal’ Modified over 9 years ago
1
Evaluation and impact assessment David Souter
2
BCO Impact Assessment session, Bern, September 2007 Why evaluate? It is dangerous to assume that what has been decided will be achieved … … or that what happens is what was intended. Accountability : to report to stakeholders on achievements against objectives Learning : to improve practice in future, particularly programme design and implementation Promotion : to demonstrate success to potential donors
3
BCO Impact Assessment session, Bern, September 2007 Why impact assessment? “The purpose of most impact assessments is to demonstrate past impact and to improve future practice, and there may be tension between the two.” “Proof of effectiveness is not the main goal of impact assessment: it is concerned more with understanding, and learning from, the processes and approaches that have led to achieving a particular impact so that such knowledge and experience could be used in comparable situations.”
4
BCO Impact Assessment session, Bern, September 2007 A hierarchy of outcomes : the MDG perception
5
BCO Impact Assessment session, Bern, September 2007 Output, outcome/effect and impact Output measures the extent to which an activity’s inputs have been made and services deployed Outcome (or effect) measures the use of inputs/services and the short-term results arising from these Impact measures the long-term changes in people’s lives, social behaviour and economic and political structures that can be attributed – in whole or part – to the activity
6
BCO Impact Assessment session, Bern, September 2007 Activities A hierarchy of complexity Inputs Outputs Outcomes or Effects Impact Increasing influence of context (social, economic, political, etc.) Source: Roche
7
BCO Impact Assessment session, Bern, September 2007 An example:
8
BCO Impact Assessment session, Bern, September 2007 What is impact assessment? “Impact assessment refers to an evaluation of how, and to what extent, development interventions cause sustainable changes in living conditions and behaviour of beneficiaries and the differential effects of these on women and men. “Impact assessment also refers to an evaluation of how, and the extent to which, development interventions influence the socio-economic and political situation in a country.” Peter Oakley et al., Outcomes and Impact
9
BCO Impact Assessment session, Bern, September 2007 What is impact assessment? “Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions.” Novib/Oxfam research project, reported in C. Roche, Impact Assessment for Development Agencies
10
BCO Impact Assessment session, Bern, September 2007 What is impact assessment? Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions.
11
BCO Impact Assessment session, Bern, September 2007 Why assess impact? Impact assessment “is concerned … with understanding, and learning from, the processes and approaches that have led to achieving a particular impact so that such knowledge and experience could be used in comparable situations.” Impact assessment should both: look back at experience to date look forward to improving programme design and implementation, and in helping to predict future impact
12
BCO Impact Assessment session, Bern, September 2007 How do we assess impact? : a view Stages of M&E developmentDegree of attention DesignExtensive ImplementationSubstantial Data generationMore Data interpretationLess UseSome EvaluationMinimal R. Davies, assessment of Bangladesh projects, 1995
13
BCO Impact Assessment session, Bern, September 2007 Causation is complex
14
BCO Impact Assessment session, Bern, September 2007 Impact is long-term
15
BCO Impact Assessment session, Bern, September 2007 The challenges of impact assessment A two tier process The challenge of context The challenge of the baseline The challenge of understanding change The challenge of time The challenge of attribution The challenge of aggregation The challenge of unexpected impact The challenge of stakeholders The challenge of perception The challenge of distribution
16
BCO Impact Assessment session, Bern, September 2007 Two tiers in establishing impact that it has had an effect on the behaviour of actors relating to the intervention itself that this effect has led to change in the circumstances of those actors that it has had an effect on policy that the policy change concerned has resulted in change in people’s lives To establish that an intervention has had impact, it is necessary to establish two things: Where advocacy initiatives are concerned, this can be described as follows:
17
BCO Impact Assessment session, Bern, September 2007 The challenge of context Impact measures change in context Understanding context is therefore essentially to identifying impact Context is complex: social, gender, cultural, political, economic, environmental etc. – and technological contextual factors inter-relate Context is changing Context can be disrupted
18
BCO Impact Assessment session, Bern, September 2007 The challenge of the baseline ‘… without some sort of baseline, it is not really possible to assess impact.’ ‘… unless detailed and continuous M&E takes place at [programme and project] level, it will be impossible to build up the body of knowledge which is necessary to understand the wider outcomes.” Measuring change requires understanding of the starting point from which change is measured Most impact assessment suffers from the lack of baseline data Impact assessment needs to be built into processes from the beginning stage if it is to be undertaken meaningfully Measures can be taken to try to recover baseline, but these are difficult and relatively weak
19
BCO Impact Assessment session, Bern, September 2007 The challenge of change Impact assessment is essentially the measurement of change Change – particularly social change – results from a large number of diverse and differently inter-related factors It is subject to unexpected externalities It may be ‘sudden, discontinuous and unpredictable’ Chance is a significant factor in change processes Understanding change requires an in-depth understanding of: context and the baseline from which change is to be measured
20
BCO Impact Assessment session, Bern, September 2007 The challenge of time Lasting change cannot be established quickly. Short-term impact assessments can identify near-project effects, rather than lasting change Short-term assessments also lack perspective regarding the complexity of change Impact varies over time: short-term impacts can be few and differ from long- term impacts and vice versa
21
BCO Impact Assessment session, Bern, September 2007 The challenge of attribution Just because change occurs, it does not mean you caused it. Many factors influence change: many people may wish to claim credit. Attribution of change is complex. There is a temptation to claim credit in order to emphasise the value of work done … … but this does not help to understand what has been (or not been) achieved, and so does not contribute to the quality of future work
22
BCO Impact Assessment session, Bern, September 2007 The challenge of aggregation Individual interventions usually target specific aspects of change Other interventions target similar and related aspects of change Change may result from the cumulative impact of numerous interventions “the sum is more than the total of the parts” implying that it is the “sum” rather than the “parts” that drives the impact
23
BCO Impact Assessment session, Bern, September 2007 The challenge of unexpected impact All initiatives have unexpected impacts, e.g. diversion of resources changes in power relationships or the distribution of wealth redistribution of income to other purposes which may be positive or negative It is as important to understand these unexpected impacts as to understand whether intended impacts are achieved Negative impacts tend to be under-reported, with the result that important lessons are not learnt for future programme design and implementation
24
BCO Impact Assessment session, Bern, September 2007 The challenge of identifying stakeholders (illustrative) Donors Non-bene- ficiaries Local inter- mediaries Political authorities Other development initiatives Project managers Target beneficiaries Age group Education Caste or class Land- holding Wealth Gender Level of use
25
BCO Impact Assessment session, Bern, September 2007 The challenge of perceptions Different stakeholders have different perceptions of impact. In particular: target beneficiaries may perceive impact differently from project managers target beneficiaries are likely to focus on individual, family and community benefits project managers are likely to have broader objectives for socio-economic or political change non-beneficiaries are likely to perceive change differently from beneficiaries Enabling different stakeholders to express divergent views of impact is crucial to successful impact assessment
26
BCO Impact Assessment session, Bern, September 2007 The challenge of distribution Impacts affect non-target beneficiaries as well as target beneficiaries Need to establish whether impact is favourable to target beneficiaries and also if it is favourable to target beneficiaries and neutral otherwise favourable to the community in general unfavourable to others or to the community in general favourable to target beneficiaries at the expense of others in the community or neighbouring communities Non-target beneficiaries may be reticent about negative impacts on themselves e.g. women may be reticent about changes in the balance of power within households
27
BCO Impact Assessment session, Bern, September 2007 Impact assessment and advocacy The impact of advocacy work requires both: that advocacy leads to policy change and that policy change leads to changes in people’s lives Advocacy work requires a wide range of interactions, many of which will be unproductive Much advocacy work is long-term and may have few short-term effects Much advocacy work is undertaken through networks in which different actors play different roles Impact assessment requires particular attention to aggregation issues
28
BCO Impact Assessment session, Bern, September 2007 Oxfam’s hierarchy of advocacy impact Heightened awareness Contribution to debate Change in opinions Change in policy Implementation of change in policy Positive change in people’s lives
29
BCO Impact Assessment session, Bern, September 2007 Impact assessment as process Diagnostics where are we now? Strategy design where do we want to go? Development and implementation how do we get there? Monitoring and evaluation how do we know we are getting there? Impact assessment needs to take place throughout the project cycle
30
BCO Impact Assessment session, Bern, September 2007 The impact assessment cycle Understanding of context (diagnostics) Development of baseline and indicators, which should be: achievable and adequate capable of providing data re inputs, outputs, effects and impact Monitoring of indicators including regular assessment (and storage) Analysis Interpretation Input into subsequent project cycles
31
BCO Impact Assessment session, Bern, September 2007 Design of IA processes Establishment of: context within assessment is to be made (impact where scope and scale of assessed intervention (impact of what) level and target (impact on whom, at individual, household or community level, etc.) timescale (impact when) interaction with other influences/factors selection of illustrative examples
32
BCO Impact Assessment session, Bern, September 2007 IA process requirements Commitment to learn from experience Adequate baseline data and indicators A management framework which recognises real levels of resources Continuous engagement and continued commitment Evaluative rigour Determination to complete the process Honesty in interpreting results
33
BCO Impact Assessment session, Bern, September 2007 Impact assessment of ICD projects Impact assessment of ICD projects does not obviously differ in principle from impact assessment in other development areas It should therefore gain from sharing experience with the wider development community However, there are some aspects of ICD projects which would seem to exacerbate the challenges of impact assessment: the lack of baseline data the very limited range of existing substantive impact assessment the unpredictability of change in ICT technology and behaviour uncertainty about the implications of different policy options
34
BCO Impact Assessment session, Bern, September 2007 Ways forward There is a good deal of experience of impact assessment in other development sectors on which to draw This experience emphasises: development of baselines continuous monitoring and assessment stakeholder participation exploration of unexpected and negative outcomes openness to diverse conclusions
35
BCO Impact Assessment session, Bern, September 2007 Ways forward “Lessons are more likely to be about the relative merits of various strategies in addressing specific audiences, as well as different sequences of tools and methods, than about developing a single model of change or assessment.” C. Roche, Impact Assessment for Development Agencies
36
BCO Impact Assessment session, Bern, September 2007 The opportunity for BCO The BCO Impact Assessment offers BCO partners an opportunity to: learn from the experience in other development sectors apply that experience to their own experience with ICD projects develop approaches to impact assessment which have specific relevance to ICD, for their own future use and the wider ICD community
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.