Evaluation Methodologies May 18, 2006 MINISTERUL FINANŢELOR PUBLICE Autoritatea de Management pentru Cadrul de Sprijin Comunitar Unitatea Centrală de Evaluare Evaluation Working Group – Second Training Seminar for Evaluation Staff of the 2007-2013 Romanian NSRF and Operational Programmes Evaluation Methodologies May 18, 2006

Today we will cover… Go through some standard evaluation methodologies (with group exercises) Review some imaginary programmes Determine the methodologies we would select to evaluate the imaginary programmes Evaluation from the external evaluator’s perspective.

Methodologies for evaluation… According to “The Evaluation of Socio-Economic Development – The GUIDE”… There are 32 methodologies..! Strategic environmental assessment, stakeholder consultation, social surveys, regression analysis, priority evaluation, participatory methods, observational techniques, multi-criteria analysis, input-output analysis, cost-benefit analysis…

The main methodologies are… Studying monitoring data Benchmarking Beneficiary/stakeholder/social surveys Case studies Focus groups Stakeholder consultation Gender Impact Assessment Cost-Benefit Analysis Econometrics Regression analysis

Today we will focus on… Studying monitoring data Benchmarking Beneficiary/stakeholder/social surveys Case studies Focus groups Stakeholder consultation Gender Impact Assessment Cost-Benefit Analysis Econometrics Regression analysis

June 15 we will focus on… Studying monitoring data Benchmarking Beneficiary/stakeholder/social surveys Case studies Focus groups Stakeholder consultation Gender Impact Assessment Cost-Benefit Analysis Econometrics Regression analysis

3. Beneficiary/Stakeholder/Social surveys The evaluator can survey.. The final beneficiaries (e.g. participants on a training course) Relevant stakeholders (e.g. managers of the training course) The general population (e.g. members of a target group that did not take part in a the training course)

Evaluators use questionnaires when… they wish to generate new, quantitative data on the programme they wish to reach large number of stakeholders the population to be observed is large and homogeneous (e.g. employers in the food processing sector, etc.); they have a precise and clear idea of what they want to observe. Any other???

Steps in carrying out a survey Step 1. Designing the questionnaire Step 2. Sampling Step 3. Pre-test or pilot Step 4. Administration of the questionnaire Step 5. Codifying the data Step 6. Analysing the data Step 7. Reporting the data

Do’s and don’ts tips for good surveys… DO ask questions that can be quantified!! DO ask questions that are clear DO direct the person easily through the questionnaire (e.g. “If yes, go to question 9, if no, go straight to question 10”) DON’T ask open-ended questions (e.g. “Tell us what you think about…..”) DON’T ask a) too many questions on one page, b) questions that are unnecessary, c) questions of an intrusive nature (e.g. “are you a Roma?”) DON’T make the questionnaire too long DON’T ask questions of a “leading” nature (e.g. “Would you agree that the programme was a success?”)

Challenges/problems with surveys They are ONE way to gather data, but should never be the exclusive method Need to standardise answers leads to simplified effects? Telephone, face-to-face surveys very expensive Postal surveys poor response rates Need big enough samples! If you have 10 categories of beneficiaries, do you survey them all?!

EXERCISE - Surveys Two examples of programmes – a) a training programme for long-term unemployed persons, b) a programme of R&D grants to SMEs Please design a “follow-up” survey with 10-12 questions to be included on a survey of beneficiaries

4. Case Studies Select one or more examples of the programme in operation and study it in-depth How to carry out a case-study evaluation Select the cases Collect and process the information Report on the case Summarise (and extrapolate) the cases, where possible

Case studies (contd.) Of these the most fundamental is selecting the cases. Use three criteria Convenience/access Information you are trying to get from the cases (see pg. 4) The extent to which you can extrapolate to provide wider insights beyond the particular case in question. Issues to consider in selecting cases are: What will count as a ‘case’? How many cases will be chosen? What units of analysis will be included within the case? How must the data be organised to allow meaningful comparisons to be made? What kind of generalisation is possible?

Challenges with case studies Can you extrapolate? Expensive to conduct a lot of case studies Credibility problems with inferring causality

EXERCISE – Case Studies Which cases to select? A programme of training grant subsidies to introduce computerised skills to older workers in industrial firms has 36 firm beneficiaries. 2000 workers have benefited. There have been some outstanding successes, but other firms claim that the programme has had no benefit to them. Which firms do we select for case studies if the objective of the evaluation is to find out: What happens at the extremes and what explains the differences between these firms experiences? What explains the success of a project for some firms? Why does the programme not work for some firms? Do firms in different industries perform better than firms in other industries? For a typical site, what happens and why? In firms with specific circumstances (such as a majority of foreign workers), what happens and why?

5. Focus groups A homogenous group of people who meet. The evaluator supplies the topics for and “chairs” the discussion. A variation is the workshop, a larger group with a more structured agenda When is it used? analysing issues with very different opinions or complex issues! testing an innovative measure (ex ante evaluation) identifying problems and needs and the improvements required during the implementation of a programme.

Focus groups (continued) Also good for validating quantitative data findings NB - Focus groups most suited to evaluations of programmes with a large “people” context (i.e. not good for transport or environmental infrastructure, good for HRD, or services-based programmes!

Focus groups – how do you do it? Step 1 – Selection of participants homogenous but diverse! you may have to offer them something! Step 2 – Choice and training of facilitators Don’t assume that you can do this yourself! Skills are needed! Don’t forget someone to take notes! Step 3 – Defining the interview topics Don’t look to cover too much! Start with the easy questions to promote discussion!

Focus groups – how do you do it? (continued) Step 4 – The discussion “a combination of assertiveness and tact may be required,” because… Watch out for bullies! Try to involve everyone! Step 5 – Analysis and reporting of results compare the information given! look for shared and divergent opinions! code the data so as to organise the results in relation to the objectives of the evaluation include some direct quotes in your report!

Challenges with focus groups “Hawthorne” effect Contrasting experiences of a minority of the participants Bullies! Straying outside the terms of the discussion

6. Stakeholder consultation Purpose – Ensuring that the preferences, interests and perspectives of different stakeholders are consideration in the evaluation Why? to allow stakeholders feel they have “ownership” of the evaluation to increase the likelihood that evaluation findings will be used and acted upon EU requirement!

Stakeholder consultation (continued) Used at 3 stages: Identifying the evaluation priorities identifying common ground and differences in what stakeholders’ want from the evaluation Arriving at a set of priorities, preferably through consensus. Gathering data on programme performance Discussing possible conclusions

Stakeholder consultation – how do you do it? Step 1 – Identify the stakeholders The Steering Committee (or commissioner) has to assist with this! Don’t assume that programme managers represent’ the interests of the beneficiaries! Step 2 – Structuring consultation with stakeholders How you achieve this will depend on: how important is stakeholder consultation to the main purpose of the evaluation? the time and budget for the evaluation, The number of stakeholders Key point - you can use other methodologies to consult with stakeholders! (e.g. focus groups, interviews, questionnaires, etc.)

How do you do it? (continued) Step 3 – Analysis of the data This will not be a separate section of the report! Step 4 – Establishing mechanisms for ongoing consultation Give periodic feedback (through presentations to the Steering Committee, etc.)

Challenges with stakeholder consultations The more stakeholders consulted, the more difficult the consultation process will be to manage. There is a limit to how many stakeholders can be consulted – which stakeholders to involve? Do the views of a participant equal the views of the programme manager?

EXERCISE – stakeholder consultation Who are the stakeholders of our JobSearch and ResDev programmes? Who do we consult? How do we consult them?

Analysing programme objectives… Programmes can have a number of objectives… A road-building programme may have its objectives… To reduce journey times To reduce accident rates To accommodate higher traffic volumes To facilitate more efficient transport of cargo To contribute to faster GDP growth To relieve pressure on a major urban centre To improve access to other strategic assets (airports, tourist destinations, etc.)

EXERCISE – What could be the objectives for… Investing in the train network? Providing broadband internet access points for the public (from 2007-2013 Economic Competitiveness OP)? Developing R&D infrastructure in enterprises and creating new R&D jobs (from 2007-2013 Economic Competitiveness OP)?

Relationship between programme objectives and evaluation objectives? The objectives of the programme, and the stage of the programme’s implementation, determine the objectives of the evaluation! (The objectives of the evaluation determine the methodology!)

Examples… Road programme objective - to facilitate an increase in transport cargo levels So is one of the evaluation objectives… To assess the impact of the number of persons killed in traffic accidents on the local economy? To assess whether the local tourism sites have benefited? To assess whether the new roads have affected the growth of the freight transport sector firms?

Examples (continued).. The main objective of a programme to give grants to the private and not-for-profit sectors to build childcare centres is to increase the supply of childcare places. So is one of the evaluation objectives…. To determine whether the programme has contributed to higher female labour force participation? To determine whether the targets set under the number of childcare places to be provided are still relevant?

Group exercise…. The overall methodology for our evaluation of JobSearch and ResDev would be….

Group exercise… KAI – investment in trains network Objective – i) to meet increased demand for high-speed rail access to major urban centres, ii) to increase the proportion of people traveling by train Activity: purchase of new wagons for trains new engines with higher speeds upgrade signaling technology Provision of “park and ride” facilities at selected train stations upgrade train stations to include indoor heated waiting rooms, etc. Monitoring data at end 2008 shows 120% expenditure, 50% output on all output indicators to that point. Evaluation ToR states that The evaluator should review: continued relevance of the programme; The effectiveness of delivery mechanisms. How do we evaluate?