1. Welfare Economics, Project and Programme Appraisal and Evaluation
Lecture 2: Programme Evaluation
David Hegarty
7 October, 2011

2. Structure
Part 1: Definition, purpose and key issues in programme evaluation
Part 2: Overview of methodological tools
Part 3: Evaluation capacity and practice in Ireland

3. Part I: Definition and Purpose
No single, preferred definition of evaluation
“Evaluation” is a common, everyday activity
People evaluate films, restaurants etc
Firms evaluate investments
“Evaluation is an elastic word that stretches to cover judgements of many kinds” (Weiss)

4. Some Definitions
“The process of collecting and analysing information and reaching conclusions on specific questions” (Dept. of Finance VFM guidance manual)
“Judgement of interventions according to their results, impacts and needs they aim to satisfy” (EU Commission)
“Evaluation is the process of determining the merit, worth, and value of things, and evaluations are the products of that process” (Scriven)
“The systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy” (Weiss)

5. Key Features Definitions point to some key features
Evaluation can be carried at various levels: policy, programme or project
A systematic exercise based on accepted social science research standards
Involves forming a judgement to be based on certain criteria
Focus can be either on the process (operation) of the programme or on its impacts (outcomes)
Purpose of exercise is to improve the intervention under evaluation

6. Evaluation Purposes: Why evaluate?
Planning
Is the programme justified?
Programme design
Resource allocation
Implementation
Is the programme working and/or how can the programme be improved?
Accountability
What was achieved?

7. Evaluation Purposes Knowledge Development
What interventions work and in what circumstances?
Does the logic of the programme and its assumptions need to be questioned?
Development
Institutional performance and strengthening
Service quality

8. Evaluation Purposes
But…
evaluation sometimes used for other, “covert” purposes
Justify decisions already made
Postpone decisions
Public relations
Compliance
“A rational exercise often undertaken for non-rational reasons” (Weiss)

9. Summary
Evaluation can be seen as serving an overarching learning purpose
“To learn through systematic enquiry how to better design, implement and deliver public programmes and policies” (EU Evalsed Guide)

10. Formative and Summative Evaluation
Accountability focus
What has been achieved?
Formative (“Process”) evaluation
Development or learning focus
How can we improve performance and delivery of programme?
“When the cook tastes the soup that’s formative evaluation. When the guest tastes it, that’s summative evaluation” (Scriven)
Most evaluations lie along this continuum
Combining elements of each
Both relevant and useful to public sector
Dept. of Finance VFM manual has strong summative emphasis

11. Key issues in programme evaluation
What's the basis for evaluation judgements?
Dept. of Finance VFM Manual refers to 5 main evaluation criteria
Relevance
Rationale
Effectiveness
Efficiency
Impact
Framework originally developed for evaluation of EU-funded programmes (CSF Evaluation Unit, 1996)

12. Evaluation Questions
Rossi et. al (2004) identify 5 main question types in evaluation of social programmes
Needs assessment
Assessment of programme theory
Assessment of programme process
Impact assessment
Efficiency assessment
DOF framework has stronger summative or economic emphasis

13. Relevance Two main dimensions Policy relevance External relevance
Domestic EU
External relevance
What societal needs or problems does programme address? (needs assessment)
Is the programme “fit-for purpose”?
Implications of external changes for programme (continued relevance)

14. Rationale Why is the State involved? Types of market failure
Is there a market failure?
“A necessary but not sufficient for government intervention to improve economic efficiency is that there is some form of market failure” (HM Treasury)
Could the problem be addressed through more direct means?
Danger of “second-best” solutions
Types of market failure
Public goods
Externalities
Redistribution

15. Effectiveness Is the programme meeting its objectives?
Generally addressed at level of
Inputs: Is the money being spent?
Outputs:
Results or immediate benefits
Are the above in line with expectations?
If not, why not?
Almost a monitoring question
Effectiveness and impact questions often overlap
A lot depends on how objectives are framed

16. Efficiency Some definitions:
“Efficiency in the public sector involves making the best use of resources available for the provision of public services” (Gershon UK efficiency review 2004)
“Optimising the ratios of inputs to outputs” (DOF VFM Manual)
Can be viewed in a number of ways
Reduced inputs for same level of service
Additional outputs for same level of inputs
Improved unit cost ratio
Changing mix of activities/outputs to better deliver a given objective for same input level
Using alternative delivery approaches, e.g., outsourcing to private sector
Efficiency a core, perhaps overarching, element of value for money agenda
Getting the best return from a given level of resources is the essence of value for money

17. Impact What difference has the programme made? Need to consider
To its beneficiaries
In terms of wider socio-economic objectives
Need to consider
Deadweight effects
Displacement
Unintended side-effects
So-called horizontal issues a sub-set of impact
Rural development
Poverty
Gender equality

18. Evaluation Cycle
Evaluation cycle is a function of the wider programme and policy cycles (see diagram)
Aim should be to conduct evaluations at the right time to influence programme design and policy formulation
Easier said than done!

19. Policy, Programme & Evaluation Cycles
Delivery
Formulation
Review
Programme
Implementation
Design
Conclusions
Evaluation
Source: The EVALSED

20. The Evaluation Cycle
EU Structural Fund regulations require evaluations at three stages
Ex ante (before)
Interim or ongoing
Ex post (after)
Value for Money reviews generally take form of ongoing evaluations

21. Ex Ante Evaluation
Focus: “to optimise the allocation of resources and improve the quality of programming” (EU Regulation)
A planning purpose
Key evaluation questions
What is the rationale for programme and is it robust?
Is the programme relevant or fit-for-purpose
Programme design issues

22. Interim Evaluation Focus Key questions
Largely an implementation purpose
But much depends on programme maturity
Key questions
Relevance or continued relevance
Effectiveness
Efficiency

23. Ex Post Mainly an accountability purpose
what has been achieved and at what cost
summative in character
Not widely practiced in Ireland except for EU programmes

24. Evaluation Cycle and Focus
Stage
Ex ante
Ongoing
Ex Post
Purpose
Planning
Implementation
Accountability
Questions
Rationale
*** *
Relevance
Effectiveness **
Efficiency
Impact

25. Overview of methodological tools
Sourcing information
and data
Data analysis
techniques
Tools to inform
evaluation judgements 25 25

26. Data Sourcing All evaluations require data Primary and secondary data
“the raw material that once collected is organised, described, grouped, counted and manipulated by various methods and techniques” (EVALSED Guide)
Primary and secondary data
Primary data are data generated as a consequence of programme (uptake of services, data relating to beneficiaries)
Secondary data are generated for other purposes and pre-exist the programme (e.g., socio-economic and administrative data) 26 26

27. Data Types
Key distinction between quantitative and qualitative approaches
Quantitative used to gather “hard” data
Who, what, how many
Expressed in terms of averages, ratios or ranges
In practice much hard or quantitative data may be categoric or ordinal in nature
Qualitative methods used to gather “soft” data
Focus on understanding or why questions
Quantitative/qualitative a continuum
Distinction stronger in terms of analytical intent
Quantitative for aggregation and generalisation
Qualitative to understand complexity 27 27

28. Data sourcing techniques
Main techniques/sources include
Monitoring indicators
Documentary analysis
Administrative data
Socio-economic data
Beneficiary surveys
Stakeholder interviews
Focus groups
Case studies 28 28

29. Data Analysis Techniques
Once the data is collected, how do we analyse it?
Main techniques include
Statistical analysis
SWOT analysis
Econometric models
Experimental designs
Quasi-experimental designs (control groups) 29 29

30. Tools to inform evaluation judgements
Having gathered and analysed the data, how do we arrive at evaluation judgements?
Main tools
Benchmarking
Multi-criteria analysis
Cost benefit analysis and cost effectiveness analysis (will be addressed in project evaluation stream)
Economic impact assessment
Macro
Micro
Intervention logic analysis
Specialist thematic tools
Gender impact assessment
Strategic environmental assessment 30 30

31. Factors Affecting Choice of Method
Programme type
Stage in programme/evaluation cycle
Evaluation purpose
Evaluation scope and questions
Data availability
Resources 31 31

32. Factors Affecting Choice of Method32 32

33. Part 3: Evaluation capacity and practice in Ireland
Concept of evaluation capacity
Influence and evolution of EU Structural Funds evaluation systems
Development of national programme evaluation processes
Expenditure Review Initiative
Value for Money and Policy Review Initiative
Where are we now?

34. The concept of evaluation capacity
Evaluation capacity concerns
the process of setting-up the necessary systems and infrastructures to undertake evaluation
Some definitions
“the development of national or sectoral evaluation systems” (MacKay, World Bank)
“the institutional, human, resource, skill and procedural base for conducting evaluations in public policy and public management systems” (Evalsed Guide)
Concerned with creating and sustaining factors that support evaluation in government sector

35. Key dimensions of evaluation capacity
Fair degree of consensus in literature as to key building-blocks
4 key dimensions generally highlighted in literature
Architecture: organisation of evaluation function
Demand: is there an effective demand for evaluation?
Supply: evaluation resources (methods, resources, skills)
Institutionalisation: building evaluation into policymaking systems
The wider, cultural factors or conditions that determine the influence of evaluation on policy

36. Critical Success Factors
Key lessons as to factors needed to strengthen government evaluation systems
Substantive government demand essential
Incentives important for demand
Limitations of reliance on rules and regulations
Need to work on demand and supply sides in parallel
Need for evaluation champions
Adequate evaluation resources
Including good data systems
Importance of structural arrangements/architecture including links with other functions
Danger of over-engineering the system
Utilisation is key
A long-haul effort requiring patience, persistence and leadership

37. Evolution and development of EU evaluation systems
Evaluation context (as of late 1980s)
Little prior tradition of programme evaluation in Ireland prior to Structural Funds
evaluation limited in scope and largely peripheral to decision making
A low evaluation capacity baseline
Evaluation impetus driven by
compliance considerations
EU Commission pressure and support
political priority attached to EU funds
Leading to ….
gradual creation of evaluation structures (CSF1, )
major expansion in evaluation capacity and output (CSF2, )

38. Key Developments in Capacity Development
1994 to 1999 CSF saw gradual establishment of programme evaluation structures
3 internal evaluation units
6 external evaluators to other programmes
By end-1996 each programme (9) had dedicated evaluation function
Central CSF Unit with a coordination and good practice promotion remit set up in 1996
Lessons learnt influenced design of evaluation arrangements for 2000 – 2006 period

39. 2000 to 2006 Evaluation System: Key Features
Evaluation system extended from just EU-funded elements to entire NDP (€51 bn)
Applied to up to 20% of total public expenditure
Centralised system with 1 Evaluation Unit (NDP/CSF Evaluation Unit)
Main responsibilities and activities
Development of performance indicators
Advice on project appraisal techniques
Drafting evaluation terms of reference
Commissioning evaluations
Doing evaluations
Extensive ongoing evaluation effort

40. Development of National Evaluation Systems
Background/origins
Increased emphasis on public service management
Public management reforms in early/mid 1990s
International developments in public management
Key milestones
C&AG amendment Act, 1993
Gave C&AG mandate to carry out VFM audits
And to examine adequacy of departments’ systems to evaluate effectiveness of their operations
76 VFM audits to date
Launch of Strategic Management Initiative (1994)
Delivering Better Government (1996)
Public Service Management Act, 1997
Departments required to produce Statements of Strategy and Annual Reports

41. Expenditure Review Initiative
Expenditure Review Initiative (ERI)
Non-EU evaluation system introduced in 1997
ERI influenced by Australian system
A “whole of government” evaluation strategy
Objectives
to provide a systematic analysis of what is actually being achieved by expenditure in each programme; and
to provide a basis on which more informed decisions can be made on priorities within and between expenditure programmes

42. ERI: Key Features
Aim was to review all expenditure areas every 3 years
Programme of reviews agreed by each department with Department of Finance
Central Steering Committee and secretariat in DOF
Reviews undertaken by line departments and by programme managers
Department of Finance represented on steering committees

43. Evolution of ERI ERI reviewed by C&AG in 2001 Key findings
3 year target not met, significant delays
Reviews focused on minor programmes
Quality highly variable
Limited influence on resource allocation
Number of reforms introduced
Establishment of network of reviewers and training supports
Independent quality review procedure
Efforts to track impact of reviews and review process generally

44. Evolution of ERI
Central steering committee (ERCSC) reviewed progress in October 2004
Key findings
Taken time to for earlier reforms to take effect
Slippage in timeframe for completion of reviews
Topics selected for review relatively small scale
Evaluative capacity of departments variable
Process had led to improvements in approach to evaluation and evaluation culture of departments
Extent to which reviews driving resource allocation decisions unclear

45. ERI Review Series of recommendations made by ERCSC
Changes to structures and reporting arrangements in departments
Independent steering committees
Reporting on review results
Intensify efforts to develop performance indicators
Use trainee analysts and graduates from IPA policy analysis masters course to support review process

46. Value for Money and Policy Review Initiative
ERI replaced in 2006 by “Value for Money and Policy Review Initiative”
Somewhat wider evaluation focus
Ninety reviews approved for period
2 per department per year
Guidance manual published 2007
Mix of internal and external reviews
Target for number of reviews does not appear to have been reached
Progress rather uneven

47. Current Situation Post 2007, EU funding very limited
EU funding of just €900 mn. for 2007 to 2013
Limited evaluation requirements under EU regulations
Earlier NDP/CSF Evaluation Unit replaced by Central Expenditure Evaluation Unit in MOF
Responsible for evaluation of all national programmes
Main focus of Unit now on Value for Money and Policy Review Initiative
Undertaking and overseeing VFM reviews
Issuing guidelines
Unit also has important role in project evaluation area

48. Reflections on Irish Experience
EU requirements and external influences a key driver
EU funds contributed to development of capacity and expertise
Increased awareness and understanding of evaluation amongst policymakers
Led to creation of internal evaluation structures
And improved supply-side capacity in response to evaluation demand
Important long-term benefits of Structural Funds

49. Reflections on Irish Experience
Ireland now in a “post Structural Funds” era
Evaluation system no longer organised around EU Funds
Slow progress under ERI and VFM processes
Some signs of a loss of momentum in evaluation practice over recent years
Current economic difficulties means heavy emphasis is on expenditure control and reduction and broader expenditure review exercises
McCarthy review (2009)
Comprehensive Spending Review

50. Conclusions: Some quotes
“In the age of evaluation Ireland has been encouraged or even compelled by the pressure of very influential external forces to adopt a culture of evaluation. Despite the fact that this culture dates back over some three decades it remains a somewhat uneasy and unconvincing addition to the tools of governance”
“One thing seems clear: policy developments in the field of evaluation will continue to be largely driven by external pressures since there is very little evidence of an appetite for evidence-driven policy among senior political or public-sector leaders” (McNamara et al, Developing a Culture of Evaluation In Ireland, 2009)