Lesson 7: Performance Macerata, 28th October Andrea Gramillano, t33 srl

Verify the consistency of the analysis of needs Obstacles (needs) Potential for solutions Ostacoli – colli di bottiglia

Agenda Definition and meanings Indicators Multicriteria

Performance: problematic defintion. OECD definition: Commission Defintion: The degree to which a development intervention or a development partner operates according to specific criteria/ standards/ guidelines or achieves results in accordance with stated goals or plans. The meaning of the word performance is not yet stable; it is therefore preferable to define it whenever it is used. Performance might mean that intended results were obtained at a reasonable cost, and/or that the beneficiaries are satisfied with them. Efficiency and performance are two similar notions, but the latter extends, more broadly, to include qualitative dimensions.

Performance: When it does take place On going Evalaution Criteria the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. Needs Resources (inputs) Output (implementation) (Specific Objective) Result

Performance questions: The project: is able to spend all the financial resources; meets the procedural deadlines; achieved the target in terms of physical realization. And in the next period: What is needed to increase the project performance?(human resources? political support? administrative enforcement? Will the project be successfully completed? Which might be future challenges Performance evaluation is based on monitoring indicators (procedural, financial, physical, output indicators)

Operational life of a Project Setting Tender or Purchaise procedures Start of Work / service Implementation Test, End and final payment

Procedural monitoring Most public activities have to follow a more or less rigid schedule in which the different steps are mandated and the deadlines fixed (i.e.). Procedural monitoring usually provides information about how project pipelines are progressing (where and when calls for tenders have been published, contracts have been awarded, …). Procedure Status Specifications ready Call published Contract awarded …… Final payment Expected Actual P1 Open 08-08-10 14-08-10 19-08-10

Physical monitoring Example of physical monitoring: Micro N 142 91 Indicator (number of enterprises) Unit of Measurement Target Achievement Micro N 142 91 Small 133 111 Medium 39 21 Owner (women) 50 40 Owner (<30y) 26 3 Start-up 54

Financial monitoring Example of financial monitoring: Priority Expected expenditures Resources committed Expenditures Amount % (a) (b) (b/a) (c) (c/a) P1 133.4 100.4 75.2 71.8 53.9

Multicriteria Tool used to compare several interventions in relation to several criteria. Multicriteria analysis is used also in the ex ante evaluation for comparing proposals. It can also be used in the ex post evaluation of an intervention, to compare the relative success of the different components of the intervention. Finally, it can be used to compare separate but similar interventions, for classification purposes. Multicriteria analysis may involve weighting, reflecting the relative importance attributed to each of the criteria. It may result in the formulation of a single judgement or synthetic classification, or in different classifications reflecting the stakeholders' diverse points of view. In the latter case, it is called multicriteria-multijudgement analysis. (from EVALSED)

Step 2 scoring or ranking Process Step 1 define criteria Step 2 scoring or ranking Step 3 weighting aggregating Step 4 Before Step 1 : Definition of the projects or actions to be judged (here results/impacts?)

STEP 1: setting criterion Financial Performance Criterion 2: Procedural Criterion 3: Physical realisation Project 1 Project 2 Project 3 Project 4 Project n…

STEP 2: Score or rank for judgment It is needed to find a way to appraise the project according to different aspects since we used different measuring units for different aspects of different process We can opt for: A) Scoring: by assigning a numeric value to different “interval” of performance . For example 3 for “above average” –1 for “on line with average” – 3 for “below average” B) Ranking: we simply order the different projects according to their performance from the first to the last

STEP 2: Scoring Criterion 1: Criterion 2: Criterion 3: Project 1 1 2 Project 3 3 Project 4 Project n…

STEP 3: Establishment of weight If some Criteria is more important than others it shall be given more importance. To do it we simple apply a multiplication factor > 1 (e.g. 1,5). Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold (For example “Physical” if some project has 0 performance, it is excluded by the analysis).

STEP 3: Apply the weight Criterion 1: ( * 1,5) Criterion 2: Project 1 1,5 1 Project 2 2 Project 3 4,5 Project 4 3 out Project n…

STEP 4: Aggregate the score Criterion 1: ( * 1,5) Criterion 2: Criterion 3: Total (with weight) Total (without weight) Project 1 1(1,5) 1 3.5 3 Project 2 2 Project 3 3(4,5) 6,5 5 Project 4 out 6 Project n…

WORK OUT: SME INCUBATOR Criterion 1: Economic (average increase of turnover) Criterion 2: New Jobs Criterion 3: Satisfaction for quality service Total Project 1 Project 2 Project 3 Project 4

WORK OUT DATAS Quality levels Low Medium High Project 1 2 3 4 Economic 80% 70% 30% Physical 50 10 30 5 Quality High Low Medium Quality levels Low Medium High

Apply weight A Economic * 1,5 Physical Veto = N. of job < 10 B * 2 Quality Veto for “Low”

See you www.t33.it a.gramillano@t33.it