New frontiers Evaluation methods Theory of change Project cycle and risk management Jesper Johnsøn, CMI, U4 Bergen, February 4, 2014

Frontiers… Current plateau – (cross) country-level, bribery, fraud Evaluations of specific reforms Integration with risk management Proxy indicators

Imbalances and the “missing middle” Imbalances: – So far, resources have been invested in standardised diagnostic and advocacy tools, not learning and evaluation Diagnostics: follow trends – indicators, data, monitoring Evaluation: what works – question, design and methods, indicators, data, judgement Missing middle Output OutcomeImpact Individual Sector Country Level of results Indicator targets

When and how to evaluate corruption costs/anti-corruption effects? Diagnose corruption risk (does corruption matter, are patterns changing?) If yes, prepare for evaluation (how and how much does corruption matter, how effective is anti-corruption?) 1.Logic model: Think through how the initiative is intended to create results by contributing to behavioral, organisational, political, or societal change, expressed in a result chain or theory of change. 2.Monitor and evaluate: Work on increasing the evaluability of the reform by shaping design, establishing indicators, systematic data collection, baselines, comparison groups INDICATORS EVALUATION QUESTION, DESIGN, METHODS PROBLEM/ DIAGNOSIS DATA ANALYSIS, COMPARISON, JUDGEMENT, CHANGE

Overview of types of evaluations Impact evaluationsProgramme evaluationsProcess evaluations Focus: cause-and-effect, attribution, impact Methods: counterfactual, comparison groups, statistical analysis, randomisation Can answer question: What would have happened in the absence of the programme? Focus: effectiveness, contribution, outcomes Methods: comparative analysis, cost- effectiveness/benefit analysis, benchmarking Can answer question: Has the programme been effective in achieving its stated objectives? Focus: efficiency, implementation, internal mechanisms, outputs Methods: workflow analysis, compare performed vs planned activities and outputs. Can answer question: Is implementation done according to plan, and do results look attainable? Clear objectives Indicators Mixed methods Triangulation Controls Randomisation Comparisons Baselines Trends

Code of Conduct example, level of results

Code of Conduct example, building an impact story

Diagnostics, evaluation, risk management and the project cycle Framework for holistic use of different tools Shows interlinkages between diagnostics, risk management and evaluation tools

Diagnostics, evaluation, risk management and the project cycle re&utm_medium=copy&rc=ex0share re&utm_medium=copy&rc=ex0share

The Proxy Challenge

Challenges we confront Linked, but not inseparable: – Measuring changes in corruption levels – Evaluating whether anti-corruption efforts are successful Progress on measuring bribery and financial fraud, but two overall indicator problems remaining: – How to measure other types of corruption (patronage, conflict of interest, abuse of power, etc.) – How to present a measure of overall corruption levels in a country, region, sector, or organisation not biased towards measurable types of corruption, and can show trajectories of change

Why proxies? Good, existing “direct” indicators for bribery and financial fraud (need better application of methods) Can capture “missing middle” – provide needed innovation for some types of corruption, to capture trajectory of performance, and measure behavioural change “Proxy indicators are alternatives to direct indicators that more directly measure the phenomenon under study but that may be hard to operationalize or require overly costly data collection”

Criteria Standard criteria for indicator development (SMART) Reflect de facto changes and behavioral changes Prioritize sensitivity to context over standardization Measure specific types of corruption Ensure that changes in corruption trends are attributable to reforms

“To assemble a body of promising ideas…” Clearly define the proxy indicator (or indicators) you suggest as a good measure of anti-corruption results Identify the type of corruption measured by the proxy indicator Clarify how the proxy indicator reflects changes in corrupt behaviours Present ideas for how the validity of the proxy indicator can be tested Describe any relevant geographic, sectoral, or institutional context Explain how the indicator can be combined with other proxy and nonproxy (direct) indicators to obtain a better measurement of overall anti-corruption progress Comment on the strengths and weaknesses of the proxy indicator and the feasibility of its use by aid agencies, governments, civil society, and others for the purposes of monitoring and reporting