Lesson 7: Effectiveness and efficiency

Lesson 7: Effectiveness and efficiency
Macerata, 18th November Andrea Gramillano, t33 srl

Agenda What does effectiveness mean? What does efficiency mean?
Cost / effectiveness analysis CBA analysis

Efficiency, Impact and sustainability in the evalaution Process
Effectiveness: which is the change? Project Result 1 Result 2 Result 3 Efficiency At what cost?

Example Effectiveness: Does the new project (road) produce the positive result (objective) of a better accessibility? Yes, now people more can get the island saving 2 hours and 3 millions of euros Effectiveness: Are the two regions directly connected? Yes Efficiency: was it the cheapest solution? No a railway would have been cheaper

Effectiveness (definition 1)
The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgement about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional developmental impact. (OECD)

Effectiveness (definition 2)
The term effectiveness has many possible meanings. The most common definition identifies effectiveness with “achievement of objectives”. This leaves open the definition to the different meanings of “objectives”. Objectives can be expressed quantitatively in terms of expected output or results. The effectiveness is evaluated simply by comparing what has been obtained with what had been planned: outputs and results indicators are all is needed. (European Commission – DG REGIO EVALSED GUIDE)

EFFECTIVENESS: WHAT TO EVALUATE ?
OECD EUROPEAN COMMISSION Quality: effectiveness is evaluated by comparing results with quality standards. Ability of a given action to produce a desired change: comparing what is observed after the action has taken place with what would have happened without the action. One needs data that allow recovery of the counterfactual situation. Effectiveness assesses whether the results outlined in the logframe are delivered and if they are likely to produce the expected objective. Evaluating effectiveness should include assessment of how people (women and men) benefit from the results brought by the project

Evaluation Questions To what extent were the originally defined objectives of the development intervention realistic? To what extent have the (direct) objectives of the development intervention been achieved in accordance with the (adjusted, if applicable) target populations? What are the (concrete) contributions of interventions for achieving the objectives of the development intervention? What factors were crucial for the achievement or failure to achieve the project objectives so far (indication of strengths and weaknesses, e.g. the monitoring and evaluation system)? What is the quality of development-policy, technical planning and coordination ?

Efficiency Definition Type of Questions
OECD: the project results have been achieved at reasonable cost with minimum waste of effort, time, money and skills. European Commission: obtaining a given output at the minimum cost or, equivalently, with maximizing output for a given level of resources. 1. Was the budget adequate? Was the spending commensurate with the delivery of activities and achieving results? 2. Were the human, financial, material resources adequate in terms of quality and quantity to achieve the project results? 3. To what extent were costs of the project justified by the benefits in comparison with similar projects or known alternative approaches?

Information and Data on OUTCOME
Indicators Methods Quantitative Financial Physical Procedural Qualitative Opinions on the level of achievement Perceptions on satisfaction Literature review Interview Community interview Project visit Focus group Case study Survey

Different way for collecting information

Customer satisfaction
THE ANALYSIS Cost - effectiveness Comparing cost and outcome Customer satisfaction Comparing perception Cost and Benefit Comparing Cost and Benefit

Cost - effectiveness Project A Project B
Public contribution to each firm / project 100,000 Euro 155,000 Euro Number of jobs created in each firm 2 FTE 3 FTE Ratio 50000 51666

Effectiveness - exercise
The project aimed at increasing accessibility by building three bridges in the period investing 3 millions. At the end 4 millions were spent. In 2016, the achieved value is 2. At the project start the idea was to increase the accessibility by reducing the time to go from the city center to the periphery (from 50 minutes to 25 minutes). In 2016, going from the city center and to the periphery takes 20 minutes). The overall circulation in all the city improved a lot, with a general reduction of 5% of time to go from any area in the city to another one. TASKS In which phase of evaluation are we ? (ex-ante, on-going, ex-post) Fill in the following table Can you give any explanations of what happened to increased accessibility (at project area and city level)? Was the project effective? Was the project efficient? Type of indicator Indicator Baseline value Target value Achieved value Built bridges …

The Customer Satisfaction
Customer satisfaction is a technique coming from Marketing. It aims to understand how customer expectations are met by products and services. More in detail, CS is based on the comparison of customers’ expectations and their perceived performance. It is positive when a product performs over the expectations. CS is measured at the individual level, but it is almost always reported at an aggregate level and it is often measured along various dimensions. There are different methods adopted in business. One of the most famous is the SERVQUAL by Parasuraman, Zeithaml and Berry.

IDENTIFY GAPS GAP: between service quality specification and service delivery This gap may arise through service personnel being poorly trained, incapable or unwilling to meet the set service standard. GAP: between service delivery and external communication Consumer expectations are highly influenced by statements made by company representatives and advertisements. GAP: between expected service and experienced service This gap arises when the consumer misinterprets the service quality.

STEPS TO DEVELOP CUSTOMER SATISFACTION
Establish Dimensions Define Questions Fix Scales Set questionnaire Deliver Survey Analyse results

DIMENSION AND QUESTIONS
DIMENSIONS QUESTIONS Dimensions represent different aspects of the services and products SERVQUAL model, refers to five dimensions of quality: Reliability Responsiveness Assurance Empathy Tangibles They represent the concrete translation of the dimensions in items to be asked to the customers. ES. A project related to vocational training defines the content of training, the methods and the materials. It might ask to students “how are you satisfied with the on line available documents?” Reliability: the ability to perform the promised service dependably and accurately Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence Tangibles: the appearance of physical facilities, equipment, personnel and communication materials Empathy: the provision of caring, individualized attention to customers Responsiveness: the willingness to help customers and to provide prompt service

FACTORS of the RATER model
Reliability: the ability to perform the promised service dependably and accurately Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence Tangibles: the appearance of physical facilities, equipment, personnel and communication materials Empathy: the provision of caring, individualized attention to customers Responsiveness: the willingness to help customers and to provide prompt service

Demographic information
Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…)

FACTORS of the RATER model
Demographic information Responses (based on a Likert scale) Age / Year (Master degree – Bachelor degree) Employment situation (yes / no) City and County (living in Macerata or not) Frequency of the service (Attendance…) FACTORS of the RATER model Reliability: the ability to perform the promised service dependably and accurately  QUALITY OF THE SERVICE (LESSONS/CONTENTS) Assurance: the knowledge and courtesy of employees and their ability to convey trust and confidence  QUALITY OF THE SERVICE (LESSONS/PROFESSOR-QUALITY OF EXPLANATION) Tangibles: the appearance of physical facilities, equipment, personnel and communication materials  QUALITY OF THE ROOM / TIMING / ACCESSIBILITY Empathy: the provision of caring, individualized attention to customers  AVAILABILITY FOR INDIVIDUALISED SUPPORT Responsiveness: the willingness to help customers and to provide prompt service  AVAILABILITY FOR “HELP DESK” …………. OPEN QUESTION FOR SUGGESTIONS

SCALE (LIKERT) and QUESTIONAIRE

SURVEY and ANALYSIS ANALYSIS (descriptive statistics) Survey
DEFINE YOUR TARGET SAMPLE ? tools: PHONE WEB PERSONAL INTERVIEW Questions Resp. score Mode Exp. 1 … TOT.

WORKOUT: DO YOU LIKE THIS COURSE?
STEPS From dimensions to questions Establish Dimensions Define Questions Fix Scales Set questionnaire Deliver Survey Analyse results Dimensions Questions

QUESTIONAIRE Questions Very bad (1) More bad than good (2)
More good than bad (3) Very good (4) 1) Are the materials / contents relevant / useful for your professional preparation? 2) Do you think that materials and contents provided during the lessons are adequate to prepare for the exam? 3) Did the course respect the predefined programme? 4) Was the professor available for providing additional information, when needed? 5) Are contents exposed in a clear way during the lessons ? 6)Is the classroom (furniture, room, computers ….) adequate for the course? 7) Is the lesson schedule adequate compared to the entire students’ week timetable? 8) Is the professor answering the questions in an effective way? 9) Is the distribution of the hours dedicated to the subjects adequate? 10) Does the course fit your degree? 11) Was the professor able to attract the students’ attention? 12) Was the professor ready to provide supports and guidance after the class? 13) Did the professor respect the timetable (he was there for the lessons)? 14) Did the course meet your expectations? IF not please provide any suggestion… QUESTIONAIRE

Questions 1 2 3 4 Questions 1 2 3 4

ANALYSIS 1 2 3 4

Lesson 8: Efficiency, Impact and sustainibility
Macerata, 8th April Andrea Gramillano, t33 srl

Agenda What do Impact and sustainability mean?
What do you mean by counterfactual analysis? CBA analysis: Financial Aanalysis Economic Analysis Risk Analysis

Efficiency, Impact and sustainability in the evaluation Process
Effectiveness: which result? Project Result 1 Result 2 Result 3 Sustainibility: Will they last? Efficiency At what cost? Impact: Are they directly related?

Sustainability Definition Type of Questions
OECD: Will the political, institutional, managerial, and technical, change last beyond the initiative? 1. Are the results likely to be sustained in the future? 2. Did national/regional/local authorities, civil society take ownership of the results of the project? 3. Are changes in attitudes, behaviour patterns and institutional arrangements likely to last once the project is closed?

Impact Definition Type of Questions
OECD: Did the project bring real change ? European Commission:The change that can be credibly attributed to an intervention. 1. To what extent was the project objective achieved? How did the results influence the indirect beneficiaries? 2. Did the project produce unintended effects? Why did these unintended results (positive or negative) happen? 3. What was the influence of external factors (negative, positive, little/no influence) such as changes in policy legislation, general economic and financial conditions?

COUNTERFACTUAL APPROACH

Example Effectiveness: Does the new project (road) produce the positive result (objective) of a better accessibility? Yes, now people more can get the island saving 2 hours and 3 millions of euros Efficiency: was it the cheapest solution? No a ferry boat would have been cheaper Sustainability: will the result last? Yes Impact: Are the effect directly connected?

Outputs and results and impact: principle of the pie / apple (pie)
External Factors Needs Outputs CHANGE (results) External factors Contribution of projects

COST AND BENEFIT ANALYSIS

Sustainability Analysis
FRAMEWORK STEPS Key concepts Economic vs. Financial Discounting Externalities Shadow prices ( conversion factor) Conditionality Time Horizon Sustainability Analysis Financial Analysis Economic Analysis Risk Analysis

DISCOUNTING Discount rate: The rate at which future values are discounted to the present. Usually considering roughly equal to the opportunity cost of capital.

 Example: DISCOUNTING
1 Euro invested at 5% yearly rate, will become (1+5%)=1,05 after one year; (1,05)x(1,05)=1,1025 after two years; (1,05)x(1,05)x(1,05)=1, after three years. The economic present value of 1 Euro that will be spent or gained two years later is 1/1,1025=0,907029; three years later is 1/1,157625=0,

DISCOUNTING Most Member States have their own social discount rates for public sector projects. These discount rates comprise between a maximunm 10% rate and a minimum 3% real rate. In this context a real 5% discount rate may be an appropriate benchmark value: low enough to focus attention on project not passing the test, but not too far from the average official discounting rate.

5 FINANCIAL ANALYSIS The financial analysis is made up of a series of tables that collect the financial flows of the investment: total investment, operating costs and revenue, sources of financing, The financial analysis should finally result in two tables summarising the cash flows to verify: SUSTAINABILITY RETURN OF INVESTIMENT

FINANCIAL SUSTAINABILITY TABLE
Financial sustainability is verified if this row is more than or equal to zero for all the years considered.

FINANCIAL INTERNAL RATE OF RETURN OF INVESTMENT (FRR/C)
This value is calculated with a financial discount rate of 6% This rate of return measures the capacity of operating net revenues to sustain the investment costs regardless the way they are financed.

Net present value Rate of return

ECONOMIC ANALYSIS “The economic analysis appraises the project contribution to the economic welfare of the region or country. It is made on behalf of the whole society (region or country) instead of just the owner of the infrastructure like in the financial analysis.”

The three steps for economic corrections
STEP 1 - Fiscal corrections STEP 2 - Externalities corrections STEP 3 - From market to accounting price

(1). Phase 1. Fiscal correction.
It is necessary to deduct from the flows of financial analysis, payments that have no real resources counterpart, as for subsidies and indirect taxes on input and output. For direct public tranfers they are already not included in the starting table for financial analysis which considers investment costs and not financial resources.

(2). Phase 2. Externalities correction.
It is necessary to include among outflows and inflows also external costs and benefits for which there is no cash flow. Some examples could be costs for health services or losses in fisheries due to incresed pollution, time saved for investment in transports, specific infrastructures supplied by public sector for the project (a road built specifically for the project…) increased touristic flows, increased accessibility of the region…

(3). Phase 3. From market to accounting prices.
It is necessary to determine a vector of conversion factors including distortion of the market into prices. Such distortions affect input, outputs and wages.

STEP1: fiscal corrections
STEP 3: From market to accounting prices This value is calculated with a social discount rate of 5% STEP 2: Externalities corrections

8 SENSITIVITY AND RISK The project examiner should check if risks associated with the project have been assessed by the applicant. This check can be done by sensitivity analysis alone, and implies a certain amount of reasoning, if not of calculations, in terms of probabilities of the main variables affecting the expected outcome of the project.

8 SENSITIVITY AND RISK Sensitivity analysis: A study of the impact that changes in crucial estimates concerning costs and benefits would have on the profitability or present value of a project Probability distribution: One is often interested to know what the probability is of a project having a economic rate of return less than the minimum required rate.

8 SENSITIVITY AND RISK Sensitivity analysis is a deterministic procedure, aimed at understanding which variables influence the economic net present value or ERR of the project. Any independent variable for which 1% change results in 1% change or more of ENPV or ERR is certainly a critical one, and needs to be estimated as reliably as possible.

8 SENSITIVITY AND RISK Risk assessment implies assigning probabilities to these critical variables and calculating the probability distribution of ERR.

SUM UP

Other delivery policy tools: law, tax, agency, campaign.
PROGRAM AND PROJECT The Project / Program cycle The Chain of Public Interventions Identification Formulation Implementation Evaluation and review Policy Program Project (1) Project (2) Other delivery policy tools: law, tax, agency, campaign.

Why do we need a project? To make a change…
The project is ….. To… Project vs. routine Project is limited in time Resources are defined Beneficiary identified Solve a problem / social need / change a behaviour: Economic growth Accessibility Social inclusion …..

Monitoring To monitor means to observe.
Monitoring is the regular observation and recording of activities taking place in a project or program. It is a process of routinely gathering information on all aspects of the project. To monitor is to check on how project activities are progressing. It is observation; ─ systematic and purposeful observation. Monitoring also involves giving feedback about the progress of the project to the donors, implementers and beneficiaries of the project. Reporting enables gathered information to be used in making decisions for improving the project performance.

Evaluation Based on monitoring, evaluation is the systematic collection and analysis of data needed to make decisions, a process in which most well-run programs engage from the outset ( American Evaluation Society) Evaluation tries to answer two distinctive questions: Did the public intervention have an effect at all and if yes, how big – positive or negative – was this effect. The question is: Does it work? Is there a causal link? This is the counterfactual question Why an intervention produces intended (and unintended) effects. The goal is to answer the “why and how it works?” question. To answer this question is the aim of the theory-based impact (European Commission – DG REGIO) An evaluation is an assessment, as systematic and objective as possible,of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors ( OECD/DAC)

Monitoring, Auditing and Evalaution
Monitoring: observing and collecting data form the project. What time is it? 12,34 At what time is the train ? 11,50 Evaluation: interpeting data to provide a subjuctive judgement for accountability and learning: You did not catch the train because of….. An alternative can be……..

Project development SELECTION Identification and Formualtion
Needs Resources (inputs) Realisation (output) Result / outcome SELECTION Identification and Formualtion Implementation END OF PROJECT After Conlcusion

Project development and Evaluation
Needs Resources (inputs) Realisation (output) Result / outcome Ex ante Evaluation On going Evaluation Monitoring Ex Post Evaluation

Relevance and Coherence ( EX ANTE)
Needs Resources (inputs) Output (implementation) (Specific Objective) Result Relevancece Program / policy Internal Coherence External coherence

Output (implementation)
Performance, Efficiency, Sustainibility, Effectiveness (on going / ex post) Needs Resources (inputs) Output (implementation) (Specific Objective) Result Performancece Efficiencyce Effectivenessce Sustainibility

Definitions Relevance : the project results and Impact CAN produce a change External Coherence: the project is strategically aligned with the concerning policy (vertical) – the program works in synergy and complementarily with other contextual intervention (horizontal) Internal coherence: the project objectives, activities, output, results, impacts are logically connected Performance: the project activities are delivered on time, the outputs respect the targets, the resources are duly absorbed, the procedures are done according to the rules. Effectiveness: the project achieves results / Impact accordingly with the targets Efficiency: the project achieves results / Impact accordingly with the targets with the minor costs and in the shorter time. Impact: the project is capable to contribute significantly to the change Sustainability: the project changes can last after the conclusion Utility: Impacts obtained by the intervention correspond to society needs

TOOL : FOCUS GROUP

Focus group Definiton Facilitation tools
A focus group is a form of qualitative research in which a group of people are asked about their perceptions, opinions, beliefs, and attitudes towards a product, service, concept, advertisement, idea, or packaging. Questions are asked in an interactive group setting where participants are free to talk with other group members. Concept mapping Metaplan ® Dual Group Moderator

Performing a Focus group
Cabaret setting presentation Brain storming / break question Structured discussion Wrap up

RISK AND CHALLENGES BIZARRE TOOLS AND OBVIOUSE OUTCOME
ONE PREDOMINANT VOICE POLARIZATION AND AMBIVALENCE

SWOT ASSESSMENT MATRIX CUSTUMER SATISFACTION MUTICRITERIA ACB Relevance Internal coherence External Coherence Performance Efficiency Effectiveness Impact and Utility Sustainibility

QUESTIONS FOR FOCUS GROUP
RELEVANCE EFFICIENCY EFFECTIVENESS SUSTAINIBILITY BEST TOPIC

See you

Lesson 7: Effectiveness and efficiency

Similar presentations

Presentation on theme: "Lesson 7: Effectiveness and efficiency"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Lesson 7: Effectiveness and efficiency

Similar presentations

Presentation on theme: "Lesson 7: Effectiveness and efficiency"— Presentation transcript:

Similar presentations

About project

Feedback