1. Monitoring & Evaluation, 12-16 Dec 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey This project is co-financed by the European Union and the Republic of Turkey MONITORING & EVALUATION for RCOP Ankara, 12 – 16 December 2011

2. Monitoring & Evaluation, 12-16 Dec 2011 2 This project is co-financed by the European Union and the Republic of Turkey SECTION I. MONITORING VS VALUATION Clarification of definitions

3. Monitoring & Evaluation, 12-16 Dec 2011 3 This project is co-financed by the European Union and the Republic of Turkey Distinction between evaluation, monitoring and audit Evaluation Assessment of the efficiency, effectiveness, impact, relevance and sustainability of aid policies and actions Monitoring Ongoing analysis of programme/project progress towards achieving planned results with the purpose of improving management decision making Audit Assessment of the legality and regularity of project expenditure and income i.e. compliance with laws and regulations and with applicable contractual rules and criteria; whether project funds have been used efficiently and economically i.e. in accordance with sound financial management; whether project funds have been used effectively i.e. for purposes intended. Primarily a financial and financial management focus, with the focus of effectiveness being on project results.

4. Monitoring & Evaluation, 12-16 Dec 2011 4 This project is co-financed by the European Union and the Republic of Turkey Evaluation-monitoring-projectmanagement Monitoring: to be aware of the present situation Evaluation: to rationally connect the present situation with the desired future situation Project management: operations and transactions (contracting, approval of reports, payments…) but activities leading to these rely on monitoring and to (interim) evaluation outcomes  Potential overlaps of monitoring and project management activities  Segregation of duties, clear definition of responsibilities  Cooperation

5. Monitoring & Evaluation, 12-16 Dec 2011 5 This project is co-financed by the European Union and the Republic of Turkey Distinction between and evaluation monitoring Monitoring What? Monitoring is an integral part of a day-to-day management. How? Monitoring embodies the regular tracking of inputs, activities, outputs, reach, outcomes, and impacts of development activities at the project program, sector and national levels Why?Monitoring provides information by which management can identify and solve implementation problems and assess progress towards project's objectives Evaluation What? Evaluation is an assessment that refers to design, implementation and results of completed or on-going project / program / policy. How? Evaluation should be systematic and objective. Key criteria to be used are: relevance, developmental efficiency, effectiveness, impact and sustainability. Why?Evaluation should provide credible and useful information to enable the incorporation of lessons

6. Monitoring & Evaluation, 12-16 Dec 2011 6 This project is co-financed by the European Union and the Republic of Turkey Distinction between and evaluation monitoring Monitoring focused on daily management issues typical questions: “How many?” "When?” “How?” “For how much?” assess whether activities are implemented effectively and efficiently Evaluation addresses strategic questions: “So what?”(impact and sustainability) and “Why?” (relevancy) analysis is deeper and seeks for actual cause-results relationships perception of “big picture"

7. Monitoring & Evaluation, 12-16 Dec 2011 7 This project is co-financed by the European Union and the Republic of Turkey Distinction between and evaluation monitoring Monitoring usually means a system data collected and analyzed more or less frequently according to a predefine timetable (Performance Measurement Plan) regularity and continuity of data collection methodology used to analyze it Evaluation flexibility in specifying, which aspects of the program should be assessed, when and how.

8. Monitoring & Evaluation, 12-16 Dec 2011 8 This project is co-financed by the European Union and the Republic of Turkey Distinction between and evaluation monitoring Monitoring Part of modern project management Generates useful information for project manager where are bottlenecks? how are we doing towards our objectives? are expenses under control? Utility is the primary feature Evaluation Target groups: donors, planners, assistance recipients and wider public Have we achieved our goals? Are our results sustainable? Have we learned anything for the future? Focus on transparency of evaluator’s approach revealing cause-effect relationships between subsequent layers of analysis.

9. Monitoring & Evaluation, 12-16 Dec 2011 9 This project is co-financed by the European Union and the Republic of Turkey Distinction between and evaluation monitoring Monitoring Project management needs  rapid assessment methods fast feedback not very expensive Evaluation Scrupulous research methodologies representative surveys comprehensive quantitative analyzes

10. Monitoring & Evaluation, 12-16 Dec 2011 10 This project is co-financed by the European Union and the Republic of Turkey M&E – Comparative characteristics CharacteristicsEvaluationMonitoring Subject: usually focused on strategic aspects addresses operational management issues Character: incidental, flexible subject & methods continuous, regular, systematic Primary client:stakeholders and external audienceprogramme management Approach:objectivity, transparencyutility Methodology: rigorous research methodologies, sophisticated tools rapid appraisal methods Primary focus: focus on relevancy, outcomes, impact and sustainability focus on operational efficiency and effectiveness Objectives: to check outcomes / impact, verify developmental hypothesis to identify and resolve implementation problems to document successes and lessons learned to assess progress towards objectives

11. Monitoring & Evaluation, 12-16 Dec 2011 11 This project is co-financed by the European Union and the Republic of Turkey M&E system diagramme INTERVENTION LOGICSOURCES OF DATAM&E TOOLSFOLLOW-UP OVERALL OBJECTIVESEXTERNAL REPORTS AND DATA FINAL EVALUATION, SURVEY, STATISTICAL ANALYSIS FINAL EVALUATION REPORT PROGRAMMING BENEFICIARIES OPINION MID-TERM EVALUATION BENEFICIARIES SURVEY MID-TERM EVAL. REPORT REVISING CONTRACTS PROJECT PURPOSE EXTERNAL EXPERTS' OPINION STRUCTURED INTERVIEW, FOCUS GROUPSQUARTERLY MONITORING REPORT RESULTS QUANTITAIVE DATA ON PROJECT PROGRESSPIPELINE ANALYSISMANAGEMENT MEETINGS IMPLEMENTERS' PERIODIC REPORTSREVIEW OF PERIODIC REPORTSREPORT APPROVAL PROCESS ACTIVITIES PRODUCTS DELIVERED BY CONTRACTORS REVIEW OF PRODUCTS AND SERVICESPRODUCT REVIEW REPORT MEANS & RESOURCES DIRECT OBSERVATIONSSITE VISITSSITE VISIT REPORT Monitoring Evaluation

12. Monitoring & Evaluation, 12-16 Dec 2011 12 This project is co-financed by the European Union and the Republic of Turkey SECTION II. EVALUATION EU Regulations on evaluation Introductory guide on basic notions

13. Monitoring & Evaluation, 12-16 Dec 2011 13 This project is co-financed by the European Union and the Republic of Turkey COUNCIL REGULATION 1085/2006 establishing IPA Article 22 Evaluation Commission shall regularly evaluate the results and efficiency of policies and programmes effectiveness of programming whether the objectives have been met Objectives of Evaluation to enable the COM to formulate recommendations results shall feed back into programme design and resource allocation

14. Monitoring & Evaluation, 12-16 Dec 2011 14 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Title II Common rules for implementation/Chapter V Evaluation and monitoring Article 57 Evaluation Objectives: to improve the quality of effectiveness and consistency of the assistance strategy and implementation of the programmes. Subject: Multi-annual indicative planning documents (ex ante evaluation) Forms: Strategic Ex ante Interim Ex post

15. Monitoring & Evaluation, 12-16 Dec 2011 15 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Article 57 Evaluation (cont’d) Interim: At least one during implementation Specifically when the monitoring reveals significant departure from the goals initially set. Ex post the responsibility of the Commission Shall include identifiable IPA component-specific results The results of ex ante and interim evaluation shall be taken into account in the programming

16. Monitoring & Evaluation, 12-16 Dec 2011 16 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 CHAPTER III/Implementation/Section 1 Framework for implementation and principles Article 78 Implementation principles in the event of participation in Community programmes and agencies (as amended by COM REGULATION 80/2010) All evaluations shall be carried out by the COM prior to the conferral of management powers on the beneficiary country, After the conferral BC is responsible for interim evaluation; COM’s rights to perform any ad-hoc evaluations remains; ex-ante and ex-post remain with COM BC’s right to carry out ex-ante and ex-post as it deems necessary.

17. Monitoring & Evaluation, 12-16 Dec 2011 17 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Title III Regional development and human resources development/Chapter III Implementation/Section 3 Evaluation and monitoring Article 166 Evaluation ex ante Beneficiary countries - under the responsibility of the operating structure. For each operational programme separately. If duly justified and in agreement with the COM BC may carry out a single ex ante evaluation covering more than one operational programme. Shall aim to optimise the allocation of budgetary resources under operational programmes improve programming quality identify and appraise the disparities, gaps and potential for development, the goals to be achieved, the results expected, the quantified targets, The coherence of the strategy proposed and the quality of the procedures for implementation, monitoring, evaluation Annexed to OP

18. Monitoring & Evaluation, 12-16 Dec 2011 18 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Article 166 Evaluation (cont’d) Interim During the programming period Beneficiary countries Linked to the monitoring of OPs, in particular where this reveals a significant departure from the goals initially set or proposals are made for the revision of OPs Results shall be sent to the sectoral monitoring committee for the OP and to COM Evaluations shall be carried out by experts or bodies, internal or external, functionally independent of the authorities The results shall be published

19. Monitoring & Evaluation, 12-16 Dec 2011 19 This project is co-financed by the European Union and the Republic of Turkey Purpose and principles Logical elements of purpose of evaluation: Assessment; Systematic and objective; Ongoing or completed project, programme or policy; Design, implementation and results; Determine the relevance and fulfillment of objectives, developmental efficiency, effectiveness, impact and sustainability; Provide information that is credible and useful; Enabling the incorporation of lessons learned into the decision- making process of both recipients and donors;

20. Monitoring & Evaluation, 12-16 Dec 2011 20 This project is co-financed by the European Union and the Republic of Turkey Purpose and principles Principles of the approach to evaluation: Impartiality and independence of the evaluation process from the programming and implementation functions; Credibility of the evaluation, through use of appropriately skilled and independent experts and the transparency of the evaluation process, including wide dissemination of results; Participation of stakeholders in the evaluation process, to ensure different perspectives and views are taken into account; Usefulness of the evaluation findings and recommendations, through timely presentation of relevant, clear and concise information to decision makers.

21. Monitoring & Evaluation, 12-16 Dec 2011 21 This project is co-financed by the European Union and the Republic of Turkey Evaluation types Ex ante Article 2 of the Financial Regulation Use of budget appropriations principles of sound financial management quantified objectives must be identified „mobilisation of Community resources must be preceded by an evaluation to ensure that the resultant benefits are in proportion to the resources applied.”

22. Monitoring & Evaluation, 12-16 Dec 2011 22 This project is co-financed by the European Union and the Republic of Turkey Evaluation types Ex ante Implementation Rules for FR – ex ante evaluation shall identify: need to be met in the short or long term; objectives to be realised; results expected and the indicators needed to measure them; added value of Community involvement; risks, including fraud; lessons learned from similar experiences in the past; volume of appropriations, human resources and administrative expenditures; monitoring system to be set up.

23. Monitoring & Evaluation, 12-16 Dec 2011 23 This project is co-financed by the European Union and the Republic of Turkey Evaluation types Ex ante Process that supports the preparation of interventions - related to programming ~ quality assurance Purpose: to gather and analyse information define objectives, ensure that these objectives can be met, that the instruments used are cost-effective, reliable later evaluation will be possible Independent from planners

24. Monitoring & Evaluation, 12-16 Dec 2011 24 This project is co-financed by the European Union and the Republic of Turkey Evaluation types Interim Assessment of the quality of programme implementation based on the first outputs and results relevance of the adopted strategy; newly occured factors having an impact on the implementation; objectives have been defined accurately; indicators are relevant; so-far effectiveness and extrapolation; management quality of the project implementation; reliable collected data, including the monitoring system Independent from implementors

25. Monitoring & Evaluation, 12-16 Dec 2011 25 This project is co-financed by the European Union and the Republic of Turkey Evaluation types Ex post After programme completion Examining long-lasting effects impact (more) visible verification to what extent objectives have been achieved Sustainability – estimated character Anticipated and unexpected effects Source of information for future programming excercises

26. Monitoring & Evaluation, 12-16 Dec 2011 26 This project is co-financed by the European Union and the Republic of Turkey Major tasks for an Evaluation Unit Identifying the need for an evaluation and selecting the topics/themes to be evaluated; Designing the evaluation, including preparing the Terms of Reference; Drafting tender documents for the evaluation study and selecting the contractor according to the established rules; Briefing the contractor and the parties involved, and supporting the evaluation mission; Ensuring the production of a high quality evaluation report and of the dissemination of evaluation findings and recommendations.

27. Monitoring & Evaluation, 12-16 Dec 2011 27 This project is co-financed by the European Union and the Republic of Turkey Tools and key documents Primary tools available to support a project evaluation: Terms of reference for the evaluation mission; The project’s Logframe matrix - to help assess what has been achieved against plan; Programme planning documents; Monitoring reports (internal and external); IPA Interim Evaluation Reports; IPA Monitoring reports Minutes of SMCs and other organisations involved in monitoring

28. Monitoring & Evaluation, 12-16 Dec 2011 28 This project is co-financed by the European Union and the Republic of Turkey Key documents produced Terms of Reference for the evaluation mission, Final Evaluation Mission Report should mirror the structure of the main evaluation criteria taking into account the nature of the project, the stage at which the evaluation is carried out, the users for whom the report is prepared.

29. Monitoring & Evaluation, 12-16 Dec 2011 29 This project is co-financed by the European Union and the Republic of Turkey Evaluation Criteria Relevance @ significance; pertinence To what extent do we do the right things ? Does it make sense? ~ to the identified problems or real needs to be addressed The appropriateness of project objectives to problems that it was supposed to address physical and policy environment within which it operated The extent to which the objectives of a programme/project are consistent with beneficiaries’ needs, country needs, global priorities.

30. Monitoring & Evaluation, 12-16 Dec 2011 30 This project is co-financed by the European Union and the Republic of Turkey Evaluation Criteria Relevance (cont’d) Relates primarily to programme design extent to which correctly address the identified problems or real needs. (Needs to be kept under review throughout the life of the project in case changes occur) logic and completeness of the project planning process internal logic and coherence of the project design at two points in time: when the project was designed, and at the time of the evaluation. Focus identification of real problems or needs correct beneficiaries

31. Monitoring & Evaluation, 12-16 Dec 2011 31 This project is co-financed by the European Union and the Republic of Turkey Evaluation Criteria Efficiency How well the various activities transformed the available resources into the intended results (outputs) quantity, quality and timeliness. value-for-money: similar at lower cost in the same time Focus: the quality of day-to-day management, i.e: budget personnel, information, property, etc; risk and change relations/co-ordination with actors time (deadlines)

32. Monitoring & Evaluation, 12-16 Dec 2011 32 This project is co-financed by the European Union and the Republic of Turkey Evaluation Criteria Effectiveness Contribution made by results to achievement of the Project Purpose Benefits accruing to target groups Did we achieve our objectives? To what extent did our outputs produce the desired outcomes? By how far the intended beneficiaries really benefited from the products or services it made available? Focus: whether the planned benefits have been delivered and received; appropriateness of the indicators of benefit used; how unplanned results may have affected the benefits received;

33. Monitoring & Evaluation, 12-16 Dec 2011 33 This project is co-financed by the European Union and the Republic of Turkey Evaluation Criteria Impact Effect of the project on its wider environment, and its contribution to the wider policy or sector objectives (as summarised in the project’s Overall Objective). Outcome: relationship between the project’s purpose and overall objectives Benefits received by the target beneficiaries had a wider overall effect on larger group Focus: to what extent the planned overall objectives have been achieved, how far that was directly due to the project; unplanned impacts’ effects on overall impact; Ex: higher standard of living, increased food security, democratic rule of law

34. Monitoring & Evaluation, 12-16 Dec 2011 34 This project is co-financed by the European Union and the Republic of Turkey Evaluation Criteria Sustainability Continuation of benefits, effects generated by a programme/project after its termination. Likelihood of benefits produced to continue to flow after external funding has ended, Focus: Ownership Policy support and the responsibility of the beneficiary institutions Institutional capacity, commitment Economic and financial factors, socio-cultural aspects, gender equality Technology, environmental aspects Financial sustainability Ex: a micro-credit scheme that is generating enough money for the scheme to operate, cover risks and develop its staff

35. Monitoring & Evaluation, 12-16 Dec 2011 35 This project is co-financed by the European Union and the Republic of Turkey A Quick reflection Appropriateness is not about money. It is about what was achieved against what was needed. Effectiveness is not about money. It is about how much it was achieved. Efficiency is about how much was achieved and how much money was spent. Sustainability is not about money. It is about the embedment of what was achieved in the given context (natural, societal, economic…)

36. Monitoring & Evaluation, 12-16 Dec 2011 36 This project is co-financed by the European Union and the Republic of Turkey Decision options Look out: one may be very efficient but not effective! or very effective but not relevant! to do things right is fine, but to do the right things is for sure far more important!

37. Monitoring & Evaluation, 12-16 Dec 2011 37 This project is co-financed by the European Union and the Republic of Turkey Decision options Continue project implementation as planned Re-orient/restructure the project To stop the project (mid-term evaluation); Modify the design of future projects or programmes in light of lessons learned (ex-post evaluation) Modify policies, co-operation strategies, and subsequent programming or identification exercises (sector, thematic or cross-sector evaluations)

38. Monitoring & Evaluation, 12-16 Dec 2011 38 This project is co-financed by the European Union and the Republic of Turkey Performance Ratings Performance rating in two steps: 1.Numerical rating on each criteria for each component of the sector from –2 to +2: 2.Conversion of numerical overall rating into a qualitative rating. Qualitative ratings « Highly satisfactory » The programmes reviewed are expected to achieve or exceed all the objectives set during their lifetime; most numerical ratings in the range of 1 to +2 « Satisfactory » The programmes reviewed are expected to largely achieve the objectives; most numerical ratings in the range of 0 to 1 « Unsatisfactory » The programmes reviewed are not expected to achieve most of the objectives; most numerical ratings in the range of –1 to 0 « Highly unsatisfactory » The programmes reviewed are not expected to achieve any of the objectives; most numerical ratings in the range of -1 to –2

39. Monitoring & Evaluation, 12-16 Dec 2011 39 This project is co-financed by the European Union and the Republic of Turkey Quality Assessment of IE Reports Criteria – Rate – Remarks A. General: Does the report design appropriately fit the evaluation? B. Sound sectoral overview: to what extent are the sector composition and priorities appropriately described? C1. Sound analysis: to what extent are the facts and data adequately analysed? C2. Sound analysis: to what extent have the indicators of achievement been adequately considered and have they been used properly where was possible? D. Robust Findings in the implementation evaluation: do the Conclusions follow logically from, and are they justified by, the data described in the Sectoral Overview?

40. Monitoring & Evaluation, 12-16 Dec 2011 40 This project is co-financed by the European Union and the Republic of Turkey Quality Assessment of IE Reports E. Impartial conclusions: does the report provide value judgements based upon the five evaluation criteria of relevance, efficiency, effectiveness, sustainability and impact? F. Useful recommendations: to what extent do the Recommendations follow logically from the Conclusions? Are they operational? Do they clearly address the monitoring sector and are they targeted to the different stakeholders? G. The executive summary: to what extent is the executive summary a synthesis and does it meet the requirements set out in the template guidelines? H. Annexes: to what extent do the Annexes support the analysis in the main text? I. Overall style, structure and text design: within the template’s framework, to what extent is the text easily readable and accessible to the various categories of readers so that the main messages are easily detectable?

41. Monitoring & Evaluation, 12-16 Dec 2011 41 This project is co-financed by the European Union and the Republic of Turkey Quality Assessment of IE Reports VerbalUnacceptablePoorSufficient / adequate GoodExcellent Single category -2012 Entire report <=-15-14 to -6 -5 to 56 to 14 >=15 Taking into account the contextual constraints on the evaluation, the overall quality rating of the report is considered to be:

42. Monitoring & Evaluation, 12-16 Dec 2011 42 This project is co-financed by the European Union and the Republic of Turkey Why Results Are Ignored? Implications may not be presented in a way that nonresearchers can understand. Results sometimes contradict deeply held beliefs. Vested interest in a program.

43. Monitoring & Evaluation, 12-16 Dec 2011 43 This project is co-financed by the European Union and the Republic of Turkey SECTION III. MONITORING EU Regulations on monitoring Monitoring system and its evaluation

44. Monitoring & Evaluation, 12-16 Dec 2011 44 This project is co-financed by the European Union and the Republic of Turkey MONITORING @ follow up, controlling A continuing observation uses systematic collection of relevant and selected data to provide management and the main stakeholders of a programme/project with indications of the extent of progress and achievement of objectives of process and impact.

45. Monitoring & Evaluation, 12-16 Dec 2011 45 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Title II Common rules for implementation/Chapter V Evaluation and monitoring Article 58 Monitoring in the case of decentralised management (IPA Monitoring Committee) BC in agreement with NIPAC and COM to ensure coherence and coordination in the implementation of the IPA components meeting the objectives set out in the financing agreements and MIIFP based on the elements given by the SMCs

46. Monitoring & Evaluation, 12-16 Dec 2011 46 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 IPA Monitoring Committee – proposals to COM, NIPAC and NAO for: any actions to ensure the coherence and crosscomponent corrective measures needed to ensure the achievement to SMCs for: decisions on any corrective measures to ensure the achievements of programme

47. Monitoring & Evaluation, 12-16 Dec 2011 47 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 IPA Monitoring Committee – organisation Internal rules of procedure in compliance with a monitoring committee mandate established by the COM within the institutional, legal and financial framework of the BC Members: Commission, NIPAC, NAO, OS, strategic coordinator COM and NIPAC co-chair Meetings at least once a year Intermediate meetings on a thematic basis.

48. Monitoring & Evaluation, 12-16 Dec 2011 48 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Article 59 Sectoral monitoring committees in the case of decentralised management Assist IPA Monitoring Committee Attached to programmes or components. May include representatives of civil society, Objective: effectiveness and quality of the implementation in accordance with the related sectoral and/or financing agreements. Proposals to the COM and NIPAC, cc. NAO for decisions on any corrective measures to ensure the achievements of programme objectives

49. Monitoring & Evaluation, 12-16 Dec 2011 49 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Sectoral monitoring committee - reports to IPA monitoring committee on progress by priority axis/ measures or operations; results achieved, financial implementation indicators any aspects of the functioning of the management and control systems raised by the audit authority, NAO or the competent accrediting officer

50. Monitoring & Evaluation, 12-16 Dec 2011 50 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Article 61 Annual and final reports on implementation Sectoral annual report and sectoral final report by OS Sectoral annual report: financial year Sectoral final report: whole period of implementation SMC examination  to be sent to NIPAC, NAO and COM

51. Monitoring & Evaluation, 12-16 Dec 2011 51 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Annual and final reports on implementation NIPAC sends annual and final reports based on synthesis of sectoral reports to COM and NAO by 31 August each year progress made in relation to the priorities of MIIFP and OPs financial implementation of Community assistance.

52. Monitoring & Evaluation, 12-16 Dec 2011 52 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Title III Regional development and human resources development/Chapter III Implementation/Section 3 Evaluation and monitoring Article 167 Sectoral monitoring committee - Organisation To be established by OS for each programme single SMC may be set up for several programmes within the same component meeting at least twice a year, at the initiative of BC or COM

53. Monitoring & Evaluation, 12-16 Dec 2011 53 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Sectoral monitoring committee – Organisation (cont’d) Internal rules of procedure in compliance with a monitoring committee mandate established by the COM within the institutional, legal and financial framework of the BC In agreement with the OS and the IPA monitoring committee COM and Head of OS co-chair composition to be decided by OS, in agreement with COM Members: COM, NIPAC, strategic coordinator for RD and HR components, OS Civil society and socio-economic partners where appropriate European Investment Bank in advisory role where EIB makes a contribution

54. Monitoring & Evaluation, 12-16 Dec 2011 54 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Sectoral monitoring committee – Tasks Consider/approve/revise criteria for selecting within six months of the entry into force of the financing agreement review progress in OP targets based on OS reporting examine (by reference to the indicators) results of implementation, achievement of the targets priority axis and measures interim evaluations sectoral annual and final reports relevant parts and recommendations of annual audit activity report any proposal to amend the financing agreement propose to OS any revision or examination of the programme (incl. its financial management)

55. Monitoring & Evaluation, 12-16 Dec 2011 55 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Article 169 Sectoral annual and final reports on implementation OS to COM and NIPAC after examination by SMC sectoral annual report by 30 June each year sectoral final report six months after the final date of eligibility of expenditure COM informs NIPAC and OS on admissibility of SAR within 10 working days from the date of its receipt – formal COM informs NIPAC and OS on its opinion on admissible SAR report within two months from the date of receipt SFR within a maximum of five months from the date of receipt

56. Monitoring & Evaluation, 12-16 Dec 2011 56 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Sectoral annual and final reports - content quantitative and qualitative progress in OP, priority axes, measures, operations against quantified targets financial implementation of OP, priority axis, measure: total expenditure paid out by the final beneficiaries included in NF’s payment applications total expenditure committed and paid out by NF accompanied by computerised forms listing the operations, total payments received from the COM breakdown of allocation by categories as listed in FA

57. Monitoring & Evaluation, 12-16 Dec 2011 57 This project is co-financed by the European Union and the Republic of Turkey COM REGULATION 718/2007 implementing 1085/2006 Sectoral annual and final reports – content (cont’d) steps taken by OS or SMC to ensure quality and effectiveness monitoring and evaluation measures, incl. data collection arrangements; summary of problems and subsequent measures; use of technical assistance. information and publicity activities; progress and financing of major projects.

58. Monitoring & Evaluation, 12-16 Dec 2011 58 This project is co-financed by the European Union and the Republic of Turkey Monitoring concepts Activity and/or result oriented monitoring? Activity based monitoring identify activities/deliverables and deadlines for each activities to be completed in a timely and appropriate manner Result based monitoring Results to be achieved Analogy to project management and monitoring Activities must be completed – legal and financial consequences Results must be achieved – effectiveness aspect

59. Monitoring & Evaluation, 12-16 Dec 2011 59 This project is co-financed by the European Union and the Republic of Turkey ROM and Internal monitoring in EU ROMInternal Monitoring ObjectivesTo provide independent assessment of project performance, with focus on ‘results’ To provide advice and recommendations to project stakeholders To generate aggregate data for reporting To support effective and timely decision making by project managers To promote accountability for resource use and achievement of results Responsibl e COM Project implementing partners/contractors

60. Monitoring & Evaluation, 12-16 Dec 2011 60 This project is co-financed by the European Union and the Republic of Turkey ROM and Internal monitoring in EU ROMInternal Monitoring MethodShort-visits to project sites by independent experts, on a periodic basis Analysis of project records and interviews with stakeholders Standardised assessment Project plans, ongoing data collection, analysis of data, progress reports Consultation with stakeholders Participation in review meetings

61. Monitoring & Evaluation, 12-16 Dec 2011 61 This project is co-financed by the European Union and the Republic of Turkey Key principles for information flow Results information needs at the project, program, and policy levels  results information must move both horizontally and vertically Horizontal: what information is being collected by their own organization and by other Vertical: bureaucratical and hierarchical burdens  need to coordinate and collaborate in sharing performance information Responsibility at each level needs to be clear for What data are collected (source) When data are collected (frequency) How data are collected (methodology) Who collects data Who reports data For whom data are collected

62. Monitoring & Evaluation, 12-16 Dec 2011 62 This project is co-financed by the European Union and the Republic of Turkey Needs of Monitoring System Ownership Demand part : who needs the information? Needs to be identified Critical issue: when levels of need and data collecting /maintenance do not fit Ownership vs enforcement Legal Financial Technical

63. Monitoring & Evaluation, 12-16 Dec 2011 63 This project is co-financed by the European Union and the Republic of Turkey Needs of Monitoring System Management Who, how, and where the system will be managed Data collection overlaps, duplication of data Time lags in receiving data Data maintenance Responsible levels Continuous development and trainings

64. Monitoring & Evaluation, 12-16 Dec 2011 64 This project is co-financed by the European Union and the Republic of Turkey Needs of Monitoring System Credibility Report all data—both good and bad Deliberate non-reporting of information demonstrating failure Political pressure Reliability: the extent to which the data collection approach is stable and consistent across time and space - measurement of the indicators is conducted the same way every time Validity: indicators should measure actual and intended performance levels Timeliness Frequency (how often are data collected?) Currency (how recently have data been collected?) Relevance (are data available frequently enough to support management decisions?)

65. Monitoring & Evaluation, 12-16 Dec 2011 65 This project is co-financed by the European Union and the Republic of Turkey Design of a monitoring system 1.Analyse project objectives to clarify project design 2.Review implementation procedures to determine information needs at the different levels of the project management structure 3.Review indicators 4.Design reporting schemes 5.Prepare an implementation plan for the monitoring system

66. Monitoring & Evaluation, 12-16 Dec 2011 66 This project is co-financed by the European Union and the Republic of Turkey Analyse project objectives Time factor since original design: chaning project environment and actors Logframe approach analysis of existing situation in light of future desired situation planning operational details Logframe matrix A possible tool: project start-up workshop all stakeholders review project documents and key assumptions revise project objectives

67. Monitoring & Evaluation, 12-16 Dec 2011 67 This project is co-financed by the European Union and the Republic of Turkey Review implementation procedures Level of detail of information need and frequency of reporting vary according to the level of management Define information needs to the different levels of the management structure From project administrators to IPAMC Workflow to be defined – procedures manual Problem: users don’t know in advance what they need  request more information than needed  users are not aware of what information is available Continual review of users requirements through: planning and review meetings: what is lacking or redundant systematic dialogue with users on content and format

68. Monitoring & Evaluation, 12-16 Dec 2011 68 This project is co-financed by the European Union and the Republic of Turkey Review indicators To avoid: selection of too many indicators selection of overly-complex indicators over-concentration on progress indicators

69. Monitoring & Evaluation, 12-16 Dec 2011 69 This project is co-financed by the European Union and the Republic of Turkey Design reporting schemes Reports Based on project administrators’ needs on daily/weekly/monthly basis Based on the needs of the hierarchy Incidental, Interim, Final as per contract Meetings ‘monthly’, ‘Project Review/Implementation Meetings’ EUD/NAC office vs CA, OBs Informal, but methodology shall be determined in advance (all the projects, only problematic ones) Who represents the organisation? Steering Committees

70. Monitoring & Evaluation, 12-16 Dec 2011 70 This project is co-financed by the European Union and the Republic of Turkey SECTION IV. MANAGEMENT INFORMATION SYSTEM Main concepts and hints An example: the Hungarian MIS

71. Monitoring & Evaluation, 12-16 Dec 2011 71 This project is co-financed by the European Union and the Republic of Turkey MIS All projects (service, supply, works, grant schemes): Similar characteristics Follow the same rules: PRAG rules and local Efficient use of limited resources: No need to rewrite again and again Harmonization and Standardization of Monitoring: Common to all

72. Monitoring & Evaluation, 12-16 Dec 2011 72 This project is co-financed by the European Union and the Republic of Turkey MIS Objectives Projects achieved their objectives Project implementations follow the rules Project progressing as planned and performing well Problems detected earlier Functions Follow-up of implementation Management control tool Support to beneficiaries Immediate data availability for the hierarchy and public

73. Monitoring & Evaluation, 12-16 Dec 2011 73 This project is co-financed by the European Union and the Republic of Turkey MIS Common communication platform and channel Enforces obeying the rules Provides monitoring system integrity Creates standards for reporting and data entry Verifies the declarations; Reports and Visits Improve the quality of monitoring activities

74. Monitoring & Evaluation, 12-16 Dec 2011 74 This project is co-financed by the European Union and the Republic of Turkey MIS Progress Ensuring project to produce expected outputs: with planned activities within the contract budget and duration Monitoring the Progress on: Financial Progress Technical Progress Risk Assessment Performance indicators

75. Monitoring & Evaluation, 12-16 Dec 2011 75 This project is co-financed by the European Union and the Republic of Turkey MIS Characteristics of the system Built with Participatory Approach Comprehensive and decentralized Open to all actors Easily Accessible (Web-based) Transparent System (Any data cannot be deleted or changed without permission) Up-to-date data collection from the source Different user levels according to responsibilities including project owners

76. Monitoring & Evaluation, 12-16 Dec 2011 76 This project is co-financed by the European Union and the Republic of Turkey MIS Some points of consideration Capacity issues Participatory approach during development and testing Database maintenance Need to assign staff specific to MIS (not IT) Source code issue For further upgrades Public procurement consequences

77. Monitoring & Evaluation, 12-16 Dec 2011 77 This project is co-financed by the European Union and the Republic of Turkey EMIR – the Hungarian model www.emir.gov.hu