Evaluating Socio-Economic Development - the ever-evolving EVALSED Guide Elliot Stern Presentation to Conference: Evaluating Public Interventions in Romania November 2nd 2006

EVALSED Guide Evolution of the Guide  The Means Collection 1999 developed by C3E Lyon  Guide to Socio Economic Development 2003 developed by Tavistock Institute, GHK Consulting and IRS (Instituto per la Ricerca Sociale) Milan  The Revised Guide focusing on evaluation capacity and an interactive ‘community of practice’ being developed by GHK & a network of experts

EVALSED Guide Philosophy of EVALSED:  Evaluation is an integral part of effective management and policy making – it is not a free standing activity nor is it just about accountability  Evaluation is not just about technique – it is also an institutionalised activity that needs to be integrated into the operations of all involved actors – which is why evaluation capacity is so important  The aim is effective socio-economic development not good evaluation on its own

EVALSED Guide Intended Users  Policy makers who have an interest in evaluation  Public sector managers and civil servants who commission evaluations  Programme managers who will wish to use evaluation in programmes management and planning  Programme partners involved as stakeholders  Evaluators wishing to broaden methods and strategies

EVALSED Guide The GUIDE is already being used widely:  The EVALSED site has over 5000 visits per month,  Each visitor clicks on about 10 pages and views about 5.5  Italy, UK and Poland largest users Usage does not include translations:  Translations are now complete not only in Romania but also in the Czech Republic, Italy and Poland  Others translations are planned

EVALSED Guide  Currently consists of a GUIDE targeted at non-technical readers  ‘Sourcebooks’ that are more specialist covering: Themes and Policy Areas; Methods and Techniques; Capacity Building  Glossary of evaluation terms

EVALSED Guide  Section 1: Contribution of Evaluation to Socio-Economic Development  Section 2: Designing and Implementing Evaluation for Socio- Economic Development  Section 3: Developing Capacity for Socio-Economic Evaluation  Section 4: Choosing Methods & Techniques - Using Evidence

EVALSED Guide  Section 1: The benefits of evaluation; rationales and purposes for evaluation; the nature of evaluation capacity; the theories that underpin interventions; the current policy context; a review of the main types and source ideas of evaluation; the role of evaluation in the policy cycle; and how evaluation strengthens socio- economic development

EVALSED Guide  Section 2: The main processes for designing and implementing evaluations; involving stakeholders; defining evaluation questions and criteria; preparing terms of reference; managing evaluations; quality assurance and evaluation standards; and ensuring evaluations are useful and used.

EVALSED Guide  Section 3: Evaluation capacity as part of institutional capacity; the role of civil society; how to develop capacity; the architecture of evaluation systems - centralised and decentralised models; strengthening evaluation demand and evaluation supply including the role of evaluation societies; institutionalising evaluations

EVALSED Guide  Section 4: Identifies the main factors that influence choices of methods and techniques - especially policy and stakeholder priorities and evaluation question being asked; discusses quantitative and qualitative approaches; methods for different types of programmes/ interventions; different programme stages; different evaluation stages - e.g. Scoping, obtaining information, drawing conclusions etc; how primary and secondary data is obtained; and how indicator systems are created

EVALSED Guide Sourcebook 1:Themes and Policies Themes: Sustainable development; increasing institutional and administrative capacity; equal opportunities; local and territorial development; social inclusion; information society Policy areas : Agriculture; transport; employment; enterprise and innovation; justice; energy etc.

EVALSED Guide Sourcebook 2: Methods and Techniques Organised under headings: Planning and structuring, e.g. Formative evaluation, logic models, stakeholder consultation Obtaining data, e.g. Surveys, observation, focus groups Analysing data, e.g. Econometric models, expert panels, delphi techniques Evaluative judgements, e.g. Cost benefit analysis, expert panels, gender impact analysis

EVALSED Guide Sourcebook 3: Capacity Development National cases: Italy, Ireland, Netherlands - all now updated  New cases (nearly complete): Spain, Czech Republic and Poland  Plus a new cross cutting regional and cross border synthesis

EVALSED Guide Latest review tasks  Finalise additional national cases  Review case examples – and we are accumulating practical examples of innovative evaluations, specific terms of reference, examples of evaluation use  Just finished revising capacity development guidance to reflect emerging experience in new member states  Introduce resources, interactivity, forum – an electronic ‘community of practice’

EVALSED Guide Interactivity  Overall purpose is to create a more dynamic interactive and useful site that will support a community of practice among those who evaluate socio-economic development

EVALSED Guide  Specific interactive elements  A resource library that can accept documents, examples and methods that will be tagged or catalogued so that others can retrieve  An online ‘forum’ or bulletin board for exchange on specific topics  FAQs (Frequently Asked Questions) to accumulate forum know- how  Occasional asynchronous ‘conferences’ on selected topics – some suggestions: indicators and data sources, participative methods of evaluation, causality and explanation

EVALSED Guide  The aim is to create a dynamic resource that will be an ‘evolving object’ rather than something always about to become obsolete and requiring revision!  To ensure that evaluation is both methodologically and even theoretically ‘fit for purpose’ whilst reinforcing management and policy-making systems so that they use evaluations well  But the overall aim is to support better socio-economic development that will contribute to the economic opportunities and quality of life of Europe’s citizens