1. Lesson 3: Performance, effectiveness, efficiency
Macerata, 12 October
Andrea Gramillano, t33 srl

2. Agenda What do we need from the last lesson? What is performance?
What does effectiveness mean?
What does efficiency mean?

3. 1) What is needed from the last lesson?

4. Appraisal and approval
Phase
Criteria
Tool
Time
Identification (of needs)
Relevance
SWOT - PEST
Before the project starts
Formulation (feasibility study)
Coherence (at least)
Matrix analysis and document analysis
Implementation
Performance
Efficiency
Effectiveness
Monitoring / indicator
Multicriteria analysis
Cost effectiveness
Comparison achieved value and target
During the project implementation
Evaluation and audit
Impact
Sustainability
In an advanced phase and at the end
Appraisal and approval

5. Relevance and Coherence ( EX ANTE)
Needs
Resources (inputs)
Output (implementation)
Result
(Specific Objective)
Relevance
Programme / policy
Utility
Internal
Coherence
External coherence

6. WORK OUT – Solution of the exercice lesson 3 Priority axis: Improving accessibility of the country Specific objective: Increase the accessibility of the rural villages located in the North of the country
Type (input, output, outcome)
Physical, financial, procedural S M A R T
Clear (only for outcome)
Km of roads renovated in 2015 (source: MS)
Output
Physical Y ? /
% increase in the accessibility index
outcome N
20 % increase in the accessibility index in 2015 (knowing that similar target have never been achieved in the past)
Number of project activities finalized each year (source: MS)
Input
Procedural
Number of inhabitants improving their life conditions (source not identified)
Increase of wellbeing in 2015 in the country (source: NAT STAT)
Euro spent (source: MS)
Financial
Number of bridges built in 2014 in rural villages located in the South of the country (source: MS)
10% increase in satisfaction of health services in 2016 (source: NAT STAT, knowing that similar targets have been achieved in the past)
Legend:
Y = Yes;
N = No;
? = it depends or not enough information available; / =not pertinent question

7. Indicators Definition Unit of measurement Source of information
Baseline (i.e. the initial value against which an indicator is subsequently measured)
Refer to local context; or
Refer to activities in previous programmes
Target (combined with baseline information, provides information concerning the importance of the planned interventions)

8. Appraisal and approval
Phase
Criteria
Tool
Time
Identification (of needs)
Relevance
SWOT - PEST
Before the project starts
Formulation (feasibility study)
Coherence (at least)
Matrix analysis and document analysis
Implementation
Performance
Efficiency
Effectiveness
Monitoring / indicator
Multicriteria analysis
Cost effectiveness
Comparison achieved value and target
During the project implementation
Evaluation and audit
Impact
Sustainability
In an advanced phase and at the end
Appraisal and approval

9. 2) Performance

10. Performance: problematic defintion.
OECD definition:
EU Commission Defintion:
The degree to which a development intervention or a development partner operates according to specific criteria/ standards/ guidelines or achieves results in accordance with
stated goals or plans.
The meaning of the word performance is not yet stable; it is therefore preferable to define it whenever it is used. Performance might mean that intended results were obtained at a reasonable cost, and/or that the beneficiaries are satisfied with them. Efficiency and performance are two similar notions, but the latter extends, more broadly, to include qualitative dimensions.

11. Output (implementation)
Performance: When?
On going Evaluation Criteria
the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules.
Needs
Resources (inputs)
Output (implementation)
Outcome
(Specific Objective)

12. Performance questions:
The project:
is able to spend all the financial resources;
meets the procedural deadlines;
achieved the target in terms of physical realization.
And in the next period:
What is needed to increase the project performance?(human resources? political support? administrative enforcement?
Will the project be successfully completed?
Which might be future challenges
Performance evaluation is based on monitoring indicators (procedural, financial, physical, output indicators)

13. Procedural monitoring
Most public activities have to follow a more or less rigid schedule in which the different steps are mandated and the deadlines fixed (i.e.). Procedural monitoring usually provides information about how project pipelines are progressing (where and when calls for tenders have been published, contracts have been awarded, …).
Procedure
Status
Specifications ready
Call published
Contract awarded ……
Final payment
Expected
Actual P1
Open

14. Physical monitoring Example of physical monitoring: Micro N 142 91
Indicator (number of enterprises)
Unit of Measurement
Target
Achievement
Micro N
142 91
Small
133
111
Medium 39 21
Owner (women) 50 40
Owner (<30y) 26 3
Start-up 54

15. Financial monitoring Example of financial monitoring: Priority
Expected expenditures
Resources committed
Expenditures
Amount %
(a)
(b)
(b/a)
(c)
(c/a) P1
133.4
100.4
75.2
71.8
53.9

16. Multicriteria analysis

17. Multicriteria (assessment tool for performance)
Tool used to compare several interventions in relation to several criteria. Multicriteria analysis is used also in the ex ante evaluation for comparing proposals. It can also be used in the ex post evaluation of an intervention, to compare the relative success of the different components of the intervention. Finally, it can be used to compare separate but similar interventions, for classification purposes. Multicriteria analysis may involve weighting, reflecting the relative importance attributed to each of the criteria. It may result in the formulation of a single judgement or synthetic classification, or in different classifications reflecting the stakeholders' diverse points of view. In the latter case, it is called multicriteria-multijudgement analysis. (from EVALSED)

18. Step 2 scoring or ranking
Process
Step 1 criteria
Step 2 scoring or ranking
Step 3 weighting
Step 4
aggregating
Before Step 1 : Definition of the projects or actions to be judged (here results/impacts?)

19. STEP 1: setting criterion
Financial Performance
Criterion 2:
Procedural
Criterion 3:
Physical realisation
Project 1
Project 2
Project 3
Project 4
Project n…

20. STEP 2: Score (judgment)
It is needed to find a way to appraise the project according to different aspects since we used different measuring units for different aspects of different process
We can opt for:
A) Scoring: by assigning a numeric value to different “interval” of performance . For example 3 for “above average” –1 for “on line with average” – 3 for “below average”
B) Ranking: we simply order the different projects according to their performance from the first to the last

21. STEP 2: Scoring Criterion 1: Criterion 2: Criterion 3: Project 1 12
Project 3 3
Project 4
Project n…

22. STEP 3: Establishment of weight
If some Criteria is more important than others it shall be given more importance. To do it we simple apply a multiplication factor > 1 (e.g. 1,5). Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold (For example “Physical” if some project has 0 performance, it is excluded by the analysis).

23. STEP 3: Apply the weight Criterion 1: ( * 1,5) Criterion 2:
Project 1
1,5 1
Project 2 2
Project 3
4,5
Project 4 3
out
Project n…

24. STEP 4: Aggregate the score
Criterion 1:
( * 1,5)
Criterion 2:
Criterion 3:
Total (with weight)
Total (without weight)
Project 1
1(1,5) 1
3.5 3
Project 2 2
Project 3
3(4,5)
6,5 5
Project 4
out 6
Project n…

25. WORK OUT: SME INCUBATOR
Criterion 1:
Economic (average increase of turnover)
Criterion 2:
New Jobs
Criterion 3:
Satisfaction for quality service
Total
Project 1
Project 2
Project 3
Project 4

26. WORK OUT DATAS Quality levels Low Medium High Project 1 2 3 4 Economic
80%
70%
30%
Physical 50 10 30 5
Quality
High
Low
Medium
Quality levels
Low
Medium
High

27. Apply weight A Economic * 1,5 Physical Veto = N. of job < 10 B
* 2
Quality
Veto for “Low”

28. 3) Effectiveness

29. Effectiveness (definition 1)
The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgement about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional developmental impact.
(OECD)

30. Effectiveness (definition 2)
The term effectiveness has many possible meanings. The most common definition identifies effectiveness with “achievement of objectives”. This leaves open the definition to the different meanings of “objectives”. Objectives can be expressed quantitatively in terms of expected output or results. The effectiveness is evaluated simply by comparing what has been obtained with what had been planned: outputs and results indicators are all is needed. (European Commission – DG REGIO EVALSED GUIDE)

31. EFFECTIVENESS: WHAT TO EVALUATE ?
OECD
EUROPEAN COMMISSION
Quality: effectiveness is evaluated by comparing results with quality standards.
Ability of a given action to produce a desired change: comparing what is observed after the action has taken place with what would have happened without the action. One needs data that allow recovery of the counterfactual situation.
Effectiveness assesses whether the results outlined in the logframe are delivered and if they are likely to produce the expected objective. Evaluating effectiveness should include assessment of how people (women and men) benefit from the results brought by the project

32. Evaluation Questions
To what extent were the originally defined objectives of the development intervention realistic?
To what extent have the (direct) objectives of the development intervention been achieved in accordance with the (adjusted, if applicable) target populations?
What are the (concrete) contributions of interventions for achieving the objectives of the development intervention?
What factors were crucial for the achievement or failure to achieve the project objectives so far (indication of strengths and weaknesses, e.g. the monitoring and evaluation system)?
What is the quality of development-policy, technical planning and coordination ?

33. 4) Efficiency

34. Efficiency Definition Type of Questions
OECD: the project results have been achieved at reasonable cost with minimum waste of effort, time, money and skills.
European Commission: obtaining a given output at the minimum cost or, equivalently, with maximizing output for a given level of resources.
1. Was the budget adequate? Was the spending commensurate with the delivery of activities and achieving results? 2. Were the human, financial, material resources adequate in terms of quality and quantity to achieve the project results? 3. To what extent were costs of the project justified by the benefits in comparison with similar projects or known alternative approaches?

35. Information and Data on OUTCOME
Indicators
Methods
Quantitative
Financial
Physical
Procedural
Qualitative
Opinions on the level of achievement
Perceptions on satisfaction
Literature review
Interview
Community interview
Project visit
Focus group
Case study
Survey

36. Different way for collecting information

37. Cost-effectiveness assessment

38. Cost - effectiveness Project A Project B
Public contribution to each firm / project
100,000 Euro
155,000 Euro
Number of jobs created in each firm
2 FTE
3 FTE
Ratio
50000
51666

39. Cost – effectiveness Project A Project B
Public contribution to each firm / project
100,000 Euro
155,000 Euro
Number of jobs created in each firm
2 FTE
3 FTE
Ratio
50000
51666

40. Cost - effectiveness Project A Project B
Public contribution to each firm / project
100,000 Euro
155,000 Euro
Number of jobs created in each firm
2 FTE
3 FTE
Ratio
50000
51666

41. 4) Exercise on effectiveness

42. New jobs created – ACHIEVED value
Question n.5 “Project benchmarking” (score of the exercise 5 points)
Three projects have been just finalized. Their objective was to create new jobs. The following table provides figures on the performance of the projects.
Performance indicator:
New jobs created (in full time equivalent employees) – TARGET value
New jobs created (in full time equivalent employees) – ACHIEVED value
Total financial resources actually spent to reach the achieved value in dollars
Project
New jobs created –
TARGET value
New jobs created – ACHIEVED value
Total financial resources spent to reach the achieved value A 2
200 B 25 5
100 C 8 4 40

43. Effectiveness / Efficiency - exercise
PROJECT: ONE WAY TO THE CENTER
OBJECTIVE: The project aimed at increasing accessibility by building three bridges in the period investing 3 millions.
At the end 4 millions were spent and two bridges built.
EXPECTED OUTCOME: At the project start, the idea was to increase the accessibility by reducing the time to go from the city center to the periphery (from 50 minutes to 25 minutes).
In 2016, going from the city center to the periphery takes 20 minutes (thus less than in the expected outcome). The overall circulation in all the city improved a lot, with a general reduction by 5% of time to move in the city (go from any area in the city to another one).
TASKS
In which phase of the project cycle are we ?
Fill in the following table
Can you give any explanations of what happened in terms of increased accessibility (at project area and city level)?
Was the project effective? Was the project efficient?
Type of indicator
Indicator
Baseline value
Target value
Achieved value
Built bridges …

44. Definitions
Relevance : the project outcomes and Impact CAN produce a change
External Coherence: the project is strategically aligned with the concerning policy (vertical) – the program works in synergy and complementarily with other contextual intervention (horizontal)
Internal coherence: the project objectives, activities, output, outcomes, impacts are logically connected
Performance: the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules.
Effectiveness: the project achieves outcomes / Impact accordingly with the targets
Efficiency: the project achieves outcomes / Impact accordingly with the targets with the minor costs and in the shorter time.
Impact: the project is capable to contribute significantly to the change
Sustainability: the project changes can last after the conclusion
Utility: Impacts obtained by the intervention correspond to society needs