IPDET 2015 WHAT IS EVALUATION? Robert Picciotto King’s College, London "There are no facts, only interpretations“ Friedrich Nietzsche 1

Setting the stage Evaluation: a fledgling profession Defining evaluation in terms of evaluator capabilities: – Dispositions – Knowledge – Practice Evaluation is on the move IPDET has shaped its evolution 2

Conflicting definitions of impact evaluation DAC: the positive and negative, primary and secondary long-term effects produced by an intervention, directly or indirectly, intended or unintended” (sustainability) 3iE: measuring through experiments the difference that an intervention makes with respect to an indicator of interest (attribution) 3

The 3iE definition is popular… It claims scientific validity and promises definitive and quantitative judgments It champions micro-economics at a time when macro economics has lost its lustre It gives austerity oriented policy makers a convenient tool for closing down programmes that ‘do not work’ The academic establishment privileges it! 4

..but on its own it is not evaluation… Advantages: – Attributing results to social interventions is a legitimate question (Does it ‘work’?) – Randomization takes care of selection bias Disadvantages: – RCTs do not tell us whether the intervention was worth pursuing; how well it performed ; whether it is replicable (why/what/how?) – it blurs accountability(who?) 5

… since it is neither necessary, nor sufficient …nor easy to do! It may be redundant… or not feasible Validity may be threatened by unobserved factors The experiment may affect behaviour RCTs are often unethical or illegal Valid statistical treatment is demanding (e.g. sample size)… and finally … simpler/cheaper methods do exist! 6

What then is evaluation? There are many ways to describe what we do (or should do) but the following concise definition, crafted by Michael Scriven, has gained broad based acceptance in the evaluation community: “Evaluation is the process of determining the merit, worth and value of something or the product of that process” It is well worth pondering! 7

The merit dimension Merit is intrinsic to the intervention. It is about doing things right and it focuses on efficacy (theory of action) It verifies compliance with standards or norms or attest to the achievement of intended goals It makes evaluation akin to process auditing It is one facet of evaluation… but again, on its own, it is not evaluation 8

The worth dimension Worth is about doing the right things. It is extrinsic along two dimensions: – relevance from the perspective of stakeholders and the society – linkages between outcomes and impacts on society (theory of change) Aggregating preferences in the public interest impact is tricky: only ethics can solve the impossibility theorem 9

Value completes the trilogy The value criterion aims at an overall judgment of effectiveness (relevance, efficacy, efficiency, sustainability, impact) ‘Doing good’ as well as ‘doing right’ Value assessment ensures that intrinsic merit criteria are selected to measure the extrinsic worth of the intervention from a public interest perspective (the domain of moral philosophy) 10

How do current evaluation models measure up to the definition? The utilization- focused model stresses merit: it is consultancy not evaluation The empowerment model stresses worth: facilitation more than evaluation Knowledge oriented models are value free and akin to social research Social justice oriented models are new, untested and neglected 11

The future of evaluation? Given alarming inequality and unsustainable environmental trends will progressive evaluation be the next evaluation wave? Progressive evaluation would require transparency, broad based participation and fulsome democratic debate and resist the forces that threaten our independence It would be grounded in ethics and come to terms with the impossibility of reducing the social world to simplistic propositions 12

You are the future of evaluation: the world needs you! Living up to the merit/worth/value challenge is extraordinarily ambitious: – Professionalize – Speak truth to power – Accept the solitude of independence The future is risky and uncertain but the struggle itself is enough… “The best way to predict the future is to create it.” Peter Drucker 13

THANK YOU FOR YOUR ATTENTION! 14