Presentation is loading. Please wait.

Presentation is loading. Please wait.

IPA Funds Monitoring and Evaluation 12-16 December 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına.

Similar presentations


Presentation on theme: "IPA Funds Monitoring and Evaluation 12-16 December 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına."— Presentation transcript:

1 IPA Funds Monitoring and Evaluation 12-16 December 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey This project is co-financed by the European Union and the Republic of Turkey Evaluation: questions and methods Ankara, 16 December 2011 Laura Trofin (PhD): TAT Non-Key Expert, Monitoring and Evaluation lauratrofin@yahoo.com

2 IPA Funds Monitoring and Evaluation 12-16 December 2011 2 This project is co-financed by the European Union and the Republic of Turkey We will : 1.Discuss about evaluation questions: how to design good evaluation questions 2.Go through some standard evaluation methodologies 3.Determine the methodologies we would select to carry out the evaluations determined yesterday

3 IPA Funds Monitoring and Evaluation 12-16 December 2011 3 This project is co-financed by the European Union and the Republic of Turkey Evaluation questions Descriptive -(what happened?) Causal questions -(how and to what extent is that which occurred attributable to the programme?) Normative questions - are the results and impacts satisfactory in relation to targets, goals, etc? Predictive questions - will the measures create negative/positive effects?) Critical questions – e.g. how can equal opportunity policies be better accepted by SMEs?

4 IPA Funds Monitoring and Evaluation 12-16 December 2011 4 This project is co-financed by the European Union and the Republic of Turkey What makes a good evaluation question? question? Good evaluation questions include the following characteristics: They are specific They are measurable They are answerable They are realistic and reasonable They are appropriate to the local needs They can contribute to knowledge development

5 IPA Funds Monitoring and Evaluation 12-16 December 2011 5 This project is co-financed by the European Union and the Republic of Turkey Where do evaluation questions generally come from? The first major source of evaluation questions includes the evaluation and programme stakeholders Most stakeholders will provide some questions if you ask them. You (evaluation expert) will need to work with these stakeholders to develop their questions into reasonable, testable evaluation questions. The second major source of evaluation questions comes from the programmes evaluator’s independent analysis of the programme/ToR.

6 IPA Funds Monitoring and Evaluation 12-16 December 2011 6 This project is co-financed by the European Union and the Republic of Turkey Methodologies for evaluation… According to “The Evaluation of Socio-Economic Development – The GUIDE”… There are 32 methodologies..! Strategic environmental assessment, stakeholder consultation, social surveys, regression analysis, priority evaluation, participatory methods, observational techniques, multi-criteria analysis, input-output analysis, cost-benefit analysis…

7 IPA Funds Monitoring and Evaluation 12-16 December 2011 7 This project is co-financed by the European Union and the Republic of Turkey The main methodologies are… 1.Studying monitoring data 2.Benchmarking 3.Interviews 4.Beneficiary/stakeholder/social surveys 5.Case studies 6.Focus groups 7.Stakeholder consultation 8.Gender Impact Assessment 9.Cost-Benefit Analysis 10.Econometrics

8 IPA Funds Monitoring and Evaluation 12-16 December 2011 8 This project is co-financed by the European Union and the Republic of Turkey 1. Studying monitoring data VITAL to every evaluation You cannot properly evaluate without monitoring data! Financial data and physical data (performance indicators) Watch out for “variance” in analysing financial data Do not read too much into performance indicator data if programme at an early stage, particularly impact.

9 IPA Funds Monitoring and Evaluation 12-16 December 2011 9 This project is co-financed by the European Union and the Republic of Turkey Challenges with studying monitoring data monitoring data The data itself may be incorrect! The data may be missing! The performance indicators may be inappropriate! (often an evaluator’s conclusion) Monitoring templates are often too clumsy, too big, unnecessary information Extracting relevant data may be difficult

10 IPA Funds Monitoring and Evaluation 12-16 December 2011 10 This project is co-financed by the European Union and the Republic of Turkey 2. Benchmarking 1.Select the parts of the programme that will be subject to benchmarking 2.Identify a partner for the benchmarking 3.Observe the process in the partner’s organisation 4.Analyse the differences in performance. Can be external (e.g. schools) or internal (e.g. faculties within a university) In the context of the Structural Funds, examples might include comparing the operation of similar programmes in different member states or comparisons of EU funded programmes and domestic initiatives within the same country.

11 IPA Funds Monitoring and Evaluation 12-16 December 2011 11 This project is co-financed by the European Union and the Republic of Turkey Challenges with benchmarking For example, schools – recruitment practices, socio-economic background of pupils, better teachers go to better schools, etc. You have a to find a similar intervention in a very similar setting!

12 IPA Funds Monitoring and Evaluation 12-16 December 2011 12 This project is co-financed by the European Union and the Republic of Turkey Beneficiary/Stakeholder/Socialsurveys The evaluator can survey: The final beneficiaries (e.g. participants on a training course) Relevant stakeholders (e.g. managers of the training course) The general population (e.g. members of a target group that did not take part in a the training course)

13 IPA Funds Monitoring and Evaluation 12-16 December 2011 13 This project is co-financed by the European Union and the Republic of Turkey Evaluators use questionnaires when… they wish to generate new, quantitative data on the programme they wish to reach large number of stakeholders the population to be observed is large and homogeneous (e.g. employers in the food processing sector, etc.); they have a precise and clear idea of what they want to observe. Any other???

14 IPA Funds Monitoring and Evaluation 12-16 December 2011 14 This project is co-financed by the European Union and the Republic of Turkey Steps in carrying out a survey Step 1. Designing the questionnaire Step 2. Sampling Step 3. Pre-test or pilot Step 4. Administration of the questionnaire Step 5. Codifying the data Step 6. Analysing the data Step 7. Reporting the data

15 IPA Funds Monitoring and Evaluation 12-16 December 2011 15 This project is co-financed by the European Union and the Republic of Turkey Do’s and don’ts tips for good surveys… 1.DO ask questions that can be quantified!! 2.DO ask questions that are clear 3.DO direct the person easily through the questionnaire (e.g. “If yes, go to question 9, if no, go straight to question 10”) 4.DON’T ask open-ended questions (e.g. “Tell us what you think about…..”) 5.DON’T ask a) too many questions on one page, b) questions that are unnecessary, c) questions of an intrusive nature (e.g. “are you a Roma?”) 6.DON’T make the questionnaire too long 7.DON’T ask questions of a “leading” nature (e.g. “Would you agree that the programme was a success?”)

16 IPA Funds Monitoring and Evaluation 12-16 December 2011 16 This project is co-financed by the European Union and the Republic of Turkey Challenges/problems with surveys Surveys They are ONE way to gather data, but should never be the exclusive method Need to standardise answers leads to simplified effects? Telephone, face-to-face surveys very expensive Postal surveys poor response rates Need big enough samples! If you have 10 categories of beneficiaries, do you survey them all?!

17 IPA Funds Monitoring and Evaluation 12-16 December 2011 17 This project is co-financed by the European Union and the Republic of Turkey Case Studies How to carry out a case-study evaluation 1.Select the cases 2.Collect and process the information 3.Report on the case 4.Summarise (and extrapolate) the cases, where possible

18 IPA Funds Monitoring and Evaluation 12-16 December 2011 18 This project is co-financed by the European Union and the Republic of Turkey Case studies (contd.) Of these the most fundamental is selecting the cases. Use three criteria 1.Convenience/access 2.Information you are trying to get from the cases 3.The extent to which you can extrapolate to provide wider insights beyond the particular case in question. Issues to consider in selecting cases are: 1.What will count as a ‘case’? How many cases will be chosen? 2.What units of analysis will be included within the case? 3.How must the data be organised to allow meaningful comparisons to be made? 4.What kind of generalisation is possible?

19 IPA Funds Monitoring and Evaluation 12-16 December 2011 19 This project is co-financed by the European Union and the Republic of Turkey Challenges with case studies Can you extrapolate? Expensive to conduct a lot of case studies Credibility problems with inferring causality

20 IPA Funds Monitoring and Evaluation 12-16 December 2011 20 This project is co-financed by the European Union and the Republic of Turkey Focus groups A homogenous group of people who meet. The evaluator supplies the topics for and “chairs” the discussion. A variation is the workshop, a larger group with a more structured agenda When is it used? analysing issues with very different opinions or complex issues! testing an innovative measure (ex ante evaluation) identifying problems and needs and the improvements required during the implementation of a programme.

21 IPA Funds Monitoring and Evaluation 12-16 December 2011 21 This project is co-financed by the European Union and the Republic of Turkey Focus groups (continued) Also good for validating quantitative data findings NB - Focus groups most suited to evaluations of programmes with a large “people” context (i.e. not good for transport or environmental infrastructure, good for HRD, or services-based programmes!)

22 IPA Funds Monitoring and Evaluation 12-16 December 2011 22 This project is co-financed by the European Union and the Republic of Turkey Focus groups – how do you do it? Step 1 – Selection of participants homogenous but diverse! you may have to offer them something! Step 2 – Choice and training of facilitators Don’t assume that you can do this yourself! Skills are needed! Don’t forget someone to take notes! Step 3 – Defining the interview topics Don’t look to cover too much! Start with the easy questions to promote discussion!

23 IPA Funds Monitoring and Evaluation 12-16 December 2011 23 This project is co-financed by the European Union and the Republic of Turkey Focus groups – how do you do it? (continued) Step 4 – The discussion “a combination of assertiveness and tact may be required,” because… Watch out for bullies! Try to involve everyone! Step 5 – Analysis and reporting of results compare the information given! look for shared and divergent opinions! code the data so as to organise the results in relation to the objectives of the evaluation include some direct quotes in your report!

24 IPA Funds Monitoring and Evaluation 12-16 December 2011 24 This project is co-financed by the European Union and the Republic of Turkey Challenges with focus groups groups “Hawthorne” effect Contrasting experiences of a minority of the participants Bullies! Straying outside the terms of the discussion

25 IPA Funds Monitoring and Evaluation 12-16 December 2011 25 This project is co-financed by the European Union and the Republic of Turkey Stakeholder consultation Purpose – Ensuring that the preferences, interests and perspectives of different stakeholders are considered in the evaluation Why? to allow stakeholders feel they have “ownership” of the evaluation to increase the likelihood that evaluation findings will be used and acted upon EU requirement!

26 IPA Funds Monitoring and Evaluation 12-16 December 2011 26 This project is co-financed by the European Union and the Republic of Turkey Stakeholder consultation (continued) (continued) Used at 3 stages: 1.Identifying the evaluation priorities identifying common ground and differences in what stakeholders’ want from the evaluation Arriving at a set of priorities, preferably through consensus. 2.Gathering data on programme performance 3.Discussing possible conclusions

27 IPA Funds Monitoring and Evaluation 12-16 December 2011 27 This project is co-financed by the European Union and the Republic of Turkey Stakeholder consultation – how do you do it? Step 1 – Identify the stakeholders The Steering Committee (or commissioner) has to assist with this! Don’t assume that programme managers represent’ the interests of the beneficiaries! Step 2 – Structuring consultation with stakeholders How you achieve this will depend on: how important is stakeholder consultation to the main purpose of the evaluation? the time and budget for the evaluation, The number of stakeholders Key point - you can use other methodologies to consult with stakeholders! (e.g. focus groups, interviews, questionnaires, etc.)

28 IPA Funds Monitoring and Evaluation 12-16 December 2011 28 This project is co-financed by the European Union and the Republic of Turkey How do you do it? (continued) (continued) Step 3 – Analysis of the data This will not be a separate section of the report! Step 4 – Establishing mechanisms for ongoing consultation Give periodic feedback (through presentations to the Steering Committee, etc.)

29 IPA Funds Monitoring and Evaluation 12-16 December 2011 29 This project is co-financed by the European Union and the Republic of Turkey Challenges with stakeholder consultations consultations The more stakeholders consulted, the more difficult the consultation process will be to manage. There is a limit to how many stakeholders can be consulted – which stakeholders to involve? Do the views of a participant equal the views of the programme manager?

30 IPA Funds Monitoring and Evaluation 12-16 December 2011 30 This project is co-financed by the European Union and the Republic of Turkey Individual interviews – some points… – some points… Don’t assume people have all day! Tell people how long you think it will last! Explain their role in what you are trying to achieve! Focus on a number of key issues (not more than 5 in 1 hour) Let them read what you write! Permission to use material from the interview must be granted Give them the overall results of the process at a later date!

31 IPA Funds Monitoring and Evaluation 12-16 December 2011 31 This project is co-financed by the European Union and the Republic of Turkey Strengths and weaknesses of interviews? People can speak freely, no bullies! Real insight into the issues Allows for direct, anonymous quotes Sensitive issues can be covered BUT…if interviewing participants, can’t reach the numbers that a survey can Relies on the integrity of the interviewer Interviewer shouldn’t be the interviewee!

32 IPA Funds Monitoring and Evaluation 12-16 December 2011 32 This project is co-financed by the European Union and the Republic of Turkey Gender Impact Assessment estimates the different effects (positive, negative or neutral) of any policy or activity on gender allows changes to be proposed that will reduce any negative gender effects

33 IPA Funds Monitoring and Evaluation 12-16 December 2011 33 This project is co-financed by the European Union and the Republic of Turkey GIA – how do you do it? Step 1 – Analysis of the present situation Step 2 – Analysis of future trends Step 3 – Determining priority Step 4 – Assessing general impact of the Programme Step 5 – Assessing specific direct impact of the Programme Step 6 – Assessing specific indirect impact of the Programme Step 7 – Defining further development Step 8 – Defining a set of gender sensitive evaluation indicators

34 IPA Funds Monitoring and Evaluation 12-16 December 2011 34 This project is co-financed by the European Union and the Republic of Turkey Challenges with GIA If women benefit equally, but the overall number of people benefiting decreases, is that positive or negative?? How do programmes that are targeted on men only or women only score under GIA? Is it okay to fund programmes that accept inequality?

35 IPA Funds Monitoring and Evaluation 12-16 December 2011 35 This project is co-financed by the European Union and the Republic of Turkey Group exercise 4 For the types of evaluations developed under GE 3 define evaluation questions and choose the most appropriate methodological tools to carry out your evaluation THINK ABOUT: Purpose(s) and type of evaluation Principles of good evaluation questions Triangulation principle Type and depth of information you want to collect Strengths and challenges of each tool


Download ppt "IPA Funds Monitoring and Evaluation 12-16 December 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına."

Similar presentations


Ads by Google