1. KAMPALA EVALUATION TALK
How Can Trans-Boundary / Regional Evaluations Be Relevant To All Participating Countries?
Uganda Evaluation Association & Evaluation Capacity Development Project (ECD)
KAMPALA EVALUATION TALK
Kampala, Uganda
22nd Jan., 2015
John K. Ogwang:

2. Inside the presentation (asking questions)
Context and Background
(The Case of the Nile Basin Initiative)
Typical Operational set op of Regional Development Programs
Key Evaluation Design and Operation Issues?
Working within existing situations to achieve developmental goals

3. Context and Background
Regional Programs:
Offer greater benefits, efficiency & effectiveness in channeling and receiving development assistance.
… But:
Take time to set up & implementation pace is different in each country
Require flexible program designs for responsiveness and relevance.
Participating countries have varied landscapes and Risks (Political, Policy, Financial, Regulatory).
Inconsistencies between regional & national programs - through overlaps, contradictions, differential relevance and applicability.
reduced delivery costs, common approach, etc among Development Partners, and on account of increased significance of regional economic blocks for recipient countries to create larger markets, better trade, negotiating power, etc.

4. Regional Development Programs will typically have 3 tier operational levels
Coordinates, makes polices & plans,& commissions evaluations
Regional level with an Executive Organ
Receives policies, work plans, implementation guidelines and “adapts” to country context for implementation.
All participating countries treated as equals in one whole
National level sub-office for coordinating in-country implementation:
Ultimately impacted by the development program will be organized to facilitate receipt of program support as designed at regional level
Program perception and expectations varied and divergent from the objectives of the regional program
Community level (development design and evaluation subjects)

5. Nile Basin Initiative (NBI) is a Regional 10 countries’ River Basin Organization in Africa
ATP = Applied Training Project
NTEAP = Nile Trans-boundary Environment Action Project
CBSI = Confidence Building & Stakeholder Involvement
SVPC = Shared Vision Coordination Project
SDBS = Socio-Economic Development & Benefit Sharing
EWUAP = Efficient Water Use Project
RPT = Regional Power Trade
WRPM = Water Resources Planning and Management
ATP
Egypt
NTEAP
Eritrea
Sudan
Nile-SEC
CBSI
S. Sudan
WRPM
SVPC
Ethiopia
ENTRO
SDBS
Kenya
D.R. Congo
EWUAP
Uganda
NELCU
Rwanda
RPT
Burundi
Tanzania

6. NBI STRATEGIC PROGRAMMING 1999-2016
NBI Secretariat Services
Shared Vision Program
Basin Cooperation and IWRM building blocks
Nile-SEC
Cooperation & IWRM
Dialogue, knowledge & planning
Subsidiary Action Program
Expanding Nile Basin Investment Projects
Subsidiary Action Program
Action on the ground
NELSPAP & ENSAP
NBI Sub-basin & Countries
1999
2008
2012
2016 +
Establishment and Confidence Building
Consolidation and Delivering
Institutional Strengthening

7. Strategic Objectives (2012-2016)
NBI - Shared Vision
“To achieve sustainable socio-economic development through the equitable utilization of, and benefit from, the common Nile basin water resources.”
Strategic Objectives ( )
NBI Portfolio of Investment Projects
NBI Programs
Facilitating Basin Cooperation Program
Water Resources Development Program
Water Resources Management Program
Power
Equatorial Lakes
Eastern Nile
River basin management
Basin Monitoring
Power
Platform for Cooperation
Agriculture
Knowledge base
River basin management
Analysis
Climate change
Agriculture
Secretariat
Water Policies
Climate change
Environment Mgnt.
Institutional development
Lead: NILE - SEC for basin-wide
Lead: Member
States
Lead: NILE - SEC
For basin-wide
Lead: SAPs 7

8. Approaches, designs, practices and methods for evaluating regional programs should recognize complexity & country variations …
To achieve greater responsiveness of Trans-boundary/ Regional Evaluations to different contexts, policies, politics, local aspirations and communities’ ways of doing things”

9. Regional Block Countries have variations in:
Levels of rigor in application & use of M&E;
Systems & procedures (Policies, Guidelines, Institutional Arrangements) approaches & adequacy;
Local capacities, skill levels & expectations at all three key levels of operation;
Political appreciation and championed support for M&E versus being perceived as “threat” to the status quo;
Openness to criticism – (threats to conclusion validity, statistical power analysis, peer review etc);
Ability of local communities to demand accountability;
Times / paces at which costs are incurred and benefits accrued at different levels (complexity of results chain).

10. Achieving goals within existing situations requires multiple approaches to evaluation designs & implementation.
Regionally designed & nationally implemented programs to recognize and read at the same page with national systems and policies – to build national capacities and increase the efficacy of national systems, & build on synergies and support national governments achieve their development goals .
Mixed open ended evaluation approaches, designs and methods that measure essentially contribution of the development intervention need to be applied at different levels of the interventions at different times, and with staggered levels of rigor - in accordance with program stage of implementation in a given country.
Consider different stakeholder perspectives, knowledge, skills and capacities, governance issues,
Regionally designed & nationally implemented programs to recognize and read at the same page with national systems and policies – to build national capacities and increase the efficacy of national systems, & build on synergies and support national governments achieve their development goals .
Mixed open ended theory –based approaches, designs and methods that measure essentially contribution of the development intervention need to be applied at different levels of the interventions at different times, and with staggered levels of rigor - in accordance with program stage of implementation in a given country.
need for multiple solutions - outcome mapping and contribution analysis adapted to multi-country evaluations, with flexibility to take care of socio-economic, cultural, and political peculiarities and differences in countries' levels of "maturity", can greatly increase the relevance and objectivity of present day evaluation theories.
Consider partnership dimensions, different stakeholder perspectives, knowledge, skills and capacities, governance issues
Emphasizing process analysis, to include; analysis of complex characteristics of the evaluands, contexts at different geographic levels.
Methodological dimensions to seek alternative approaches to the counterfactual since external events and influence of other programs are critical in assessing effect of a regional program on national processes and outcomes.

11. Achieving goals within existing situations requires multiple approaches to evaluation designs & implementation.
Encourage local partnerships in addition to deliberate local capacity building,
Addressing data constraints - (reconstructing baseline data & control groups, working with non-equivalent control groups, collecting data on sensitive topics & difficult to reach groups, multi-method approaches)
Need for multiple solutions - outcome mapping and contribution analysis adapted to multi-country evaluations, with flexibility to take care of socio-economic, cultural, and political peculiarities and differences in countries' levels of "maturity", can greatly increase the relevance and objectivity of present day evaluation theories.
Outcome mapping focuses on one specific type of result: outcomes as behavioral change. As you probably recall, outcomes are defined as changes in the behavior, relationships, activities, or actions of other people, groups, and organizations with whom a program works directly.
e page with national systems and policies – to build national capacities and increase the efficacy of national systems, & build on synergies and support national governments achieve their development goals .
Mixed open ended theory –based approaches, designs and methods that measure essentially contribution of the development intervention need to be applied at different levels of the interventions at different times, and with staggered levels of rigor - in accordance with program stage of implementation in a given country.
need for multiple solutions - outcome mapping and contribution analysis adapted to multi-country evaluations, with flexibility to take care of socio-economic, cultural, and political peculiarities and differences in countries' levels of "maturity", can greatly increase the relevance and objectivity of present day evaluation theories.
Consider partnership dimensions, different stakeholder perspectives, knowledge, skills and capacities, governance issues
Emphasizing process analysis, to include; analysis of complex characteristics of the evaluands, contexts at different geographic levels.
Methodological dimensions to seek alternative approaches to the counterfactual since external events and influence of other programs are critical in assessing effect of a regional program on national processes and outcomes.

12. Achieving goals within existing situations requires multiple approaches to evaluation designs & implementation.
Emphasizing process analysis, to include; one-whole analysis of complex characteristics of the evaluands, contexts at different geographic levels.
Methodologies that seek alternative approaches to the counterfactual since external events and influence of other programs are critical in assessing effect of a regional program on national processes and outcomes.
Helping clients use the evaluation – (ensuring active participation of clients in the scoping phase, formative evaluation strategies, constant communication with all stakeholders, evaluation capacity building, appropriate strategies for communicating findings, developing and monitoring the follow-up action plans)

13. Useful Questions to ask
whose policies and strategies are applied when a regional evaluation is undertaken in a country with operational national M&E policies and strategies,?
Would it be possible to design multiple country-focused localized evaluations that seek to fulfill ToRs for evaluation of one regional development program?
Can an aggregate generalization on outcomes or impact of a regional program be made by summing findings from different targeted communities obtained through undifferentiated assessment?
In the sample case of NBI, are any water resources management changes and peoples' poverty reduction changes observed at communities and national level as a result of NBI or as a result of own efforts of the National Governments with some level of contribution by NBI's? What is this contribution?
What are the perceptions of the local communities in each of the countries' Nile Basin areas?
Can these communities be identified with the evaluation results at Regional level?

14. Concluding with Careful Attribution of Regional Interventions to National Efforts – The Missing Casual Link
Contribution analysis; assesses the contribution of a particular development agency (and the particular intervention supported) to the achievement of the overall changes resulting from the collaborative financial and technical interventions of a number of different development agencies.
Attribution analysis; is carried out when the treatment / project group is matched to a comparison / control group so that alternative explanations of the observed changes can be controlled for elimination of other “causes” of changes manifested. There is then a prima facie case for claiming that the difference between the treatment / project group and the comparison / control group indicates the contribution of the intervention / project to these changes.
Relationship between the intervention and observed results can take any one or combination of the following four aspects:
The intervention ‘caused’ the results;
The intervention made a difference;
The intervention contributed to the results;
The intervention can be attributed to a specific net result.
What kind of causal relation then exists between a development intervention (X) and an impact (Y)? X can cause Y; X may be necessary for Y to occur; X alone may be sufficient for Y to be realized. But we clearly want to make some causal link between the intervention X and the impact Y.
An intervention works as part of a broader causal package. And if it works, then this causal package is indeed sufficient to bring about the impact. Further, if the intervention is ‘working’, then it is an essential part of this causal package. Thus, an intervention “made a difference” when the intervention causal package was sufficient to bring about the impact, and the intervention as a trigger was a necessary component of the causal package. The intervention in this case is a contributory cause. On its own it is neither necessary nor sufficient.
THANKYOU FOR YOUR TIME