Lesson 7: Performance Macerata, 13 December Alessandro Valenza, Director, t33 srl

Agenda  Definiton and meanings  Indicators  Multicriteria

Performance: problematic defintion. OCSE definition: The degree to which a development intervention or a development partner operates according to specific criteria/ standards/ guidelines or achieves results in accordance with stated goals or plans. The degree to which a development intervention or a development partner operates according to specific criteria/ standards/ guidelines or achieves results in accordance with stated goals or plans. Commission Defintion: The meaning of the word performance is not yet stable; it is therefore preferable to define it whenever it is used. Performance might mean that intended results were obtained at a reasonable cost, and/or that the beneficiaries are satisfied with them. Efficiency and performance are two similar notions, but the latter extends, more broadly, to include qualitative dimensions.

Performance: When it does take place? the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. On going Evalaution Criteria NeedsResources (inputs) Output (implementation) Result (Specific Objective)

Performance questions: The project: -is able to spend all the financial resources; -meets the procedural deadlines; -achieved the target in terms of physical realization. And in the next period: -What is needed to increase the Project performance: human resources? Political support? Administrative enforcement? -Will the project be successfully completed? -Which might be future challenges Performance evaluation is based on monitoring indicators

Operational life of a Project Setting Tender or Purchaise procedures Start of Work / service Implementation Test, End and final payment

Procedural monitoring Most public activities have to follow a more or less rigid schedule in which the different steps are mandated and the deadlines fixed (i.e.). Procedural monitoring usually provides information about how project pipelines are progressing (where and when calls for tenders have been published, contracts have been awarded, …). Procedure Status Specificationsready Call published Contractawarded …… Final payment Expected Actual Expected Actual Expected Actual …… Expected Actual P1Open ……

Physical monitoring Example of physical monitoring: Indicator (number of enterprises) Unit of Measurement TargetAchievement MicroN14291 SmallN MediumN3921 Owner (women)N5040 Owner (<30y)N263 Start-upN543

Financial monitoring Example of financial monitoring: PriorityExpected expenditures Resources committed Expenditures Amount % % (a)(b)(b/a)(c)(c/a) P

Multicriteria Tool used to compare several interventions in relation to several criteria. Multicriteria analysis is used also in the ex ante evaluation for comparing proposals. It can also be used in the ex post evaluation of an intervention, to compare the relative success of the different components of the intervention. Finally, it can be used to compare separate but similar interventions, for classification purposes. Multicriteria analysis may involve weighting, reflecting the relative importance attributed to each of the criteria. It may result in the formulation of a single judgment or synthetic classification, or in different classifications reflecting the stakeholders' diverse points of view. In the latter case, it is called multicriteria-multijudge analysis. (from EVALSED)

Process Step 1 define criteria Step 2 scoring or ranking Step 3 weighting Step 4 aggregating

STEP 1: setting criterion Criterion 1: Financial Performance Criterion 2: Procedural Criterion 3: Physical realisation Project 1 Project 2 Project 3 Project 4 Project n…

STEP 2: Score or rank for judgment -It is needed to find a way to appraise the project according to different aspects since we used different measuring units for different aspects of different process -We can opt for: - A) Scoring: by assigning a numeric value to different “interval” of performance. For example 3 for “above average” –1 for “on line with average” – 3 for “below average” - B) Ranking: we simply order the different projects according to their performance from the first to the last

STEP 2: Scoring Criterion 1:Criterion 2:Criterion 3: Project 1111 Project 2021 Project 3311 Project 4330 Project n…

STEP 3: Establishment of weight If some Criteria is more important than others it shall be given more importance. To do it we simple apply a multiplication factor > 1 (e.g. 1,5). Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold (For example “Physical” if some project has 0 performance, it is excluded by the analysis).

STEP 3: Apply the weight Criterion 1: ( * 1,5) Criterion 2:Criterion 3: Project 11,511 Project 2021 Project 34,511 Project 44,53out Project n…

STEP 4: Aggregate the score Criterion 1: ( * 1,5) Criterion 2:Criterion 3: Total (with weight) Total (without weight) Project 11, Project Project 34,5116,55 Project 44,53out 6 Project n…

WORK OUT: SME INCUBATOR Criterion 1: Economic (average increase of turnover) Criterion 2: New Jobs Criterion 3: Satisfaction for quality service Total Project 1 Project 2 Project 3 Project 4

WORK OUT DATAS Project1234 Economic80%70%30%70% Physical QualityHighLowHighMedium LowMediumHigh Quality levels

Apply weight A Economic* 1,5 PhysicalVeto = N. of job < 10 B Physical* 2 QualityVeto for “Low”

See you

See you