Country-led Joint Evaluation Dutch ORET/MILIEV Programme in China NCSTE Country-led Joint Evaluation Dutch ORET/MILIEV Programme in China Chen Zhaoying NCSTE, China Hans Slot IOB, The Netherlands Sixth Meeting of the DAC Network on Development Evaluation, June 2007, Paris

2 Presentation Outline A brief note on the evaluation ■ A brief note on the evaluation The response from stakeholders and the dissemination of the evaluation ■ The response from stakeholders and the dissemination of the evaluation ssues for discussion ■ Issues for discussion

3 Joint Evaluation of Aid Programs Definition DAC Glossary defines joint evaluation as: An evaluation to which different donors and/or partners participate. Four categories of joint evaluation ■ Donor + Donor Donor + Partner Country ■ Multi-Donor + Multi-Partner ■ Partner + Partner

4 Joint Evaluation of Aid ProgramsOverview ■ Partnership in development leads to partnership in evaluation. Joint Evaluation has been on the international development agenda since the early 1990s. ■ Country-led Joint Evaluation of ORET/MILIEV Programme in China is regarded by DAC Network study report as one of the important joint evaluation cases since 1990.

5 Country-led Evaluation ■ Ownership in development leads to ownership of evaluation. ■ The World Bank, UNDP, the DAC, as well as some donor governments, have been developing approaches to promote country-led evaluations (CLEs) and to increase partner country ownership. ■ The majority of evaluations of development aid programmes are still donor evaluations: Led by donors and are carried out to meet donors’ requirements ■ What is needed are evaluations from the perspective of the partner country. Country-led evaluations will provide that.

6 The Country-led Joint Evaluation in China The Country-led Joint Evaluation in China ■ About the title of the evaluation ■ The recipient country leads the process, supported by the donor and both sides are jointly responsible for the evaluation The evaluation covers the period A total project portfolio of 84 projects were approved. 40 projects were finished and 44 were ongoing. Of these 44, 17 completed the transaction but still had to produce the final report. The grant amount of the transaction is about € 200 million, the total transaction amount of the evaluated portfolio € 470 million.

7 How is the Evaluation Initiated? ■ After the experiences of joint evaluation of aid development, NCSTE would like to have a more pronounced role in the development evaluation and IOB would like to be involved in a recipient- led joint evaluation. ■ The joint evaluation was co-initiated by IOB and NCSTE.

8 The Evaluation Aims to Assess to what extent the programme (through its activities / projects) has fulfilled the policy objectives, needs and priorities of the Netherlands and China Verify whether the funds have been appropriately and efficiently used Provide information for both the Chinese and the Dutch that could be used to improve the programme, as well as for policy formulation

9 Evaluation Criteria Evaluation Criteria ■ The relevance of the Programme ■ The efficiency of the Programme ■ The effectiveness of the Programme ■ The impact of the projects implemented under the Programme.

10 How is the evaluation organized? Governance Structure An equal governance structure was established for this particular type of evaluation. ■ The Steering Committee (SC) Make decisions on the evaluation ■ Team Leaders (TL) Organise the fieldwork and other tasks ■ The Reference Group (RG) Provide advice and support

11 How is the evaluation conducted? Four steps of the joint evaluation ■ Design/TOR ■ Setting up & organization ■ Implementation & Final Draft Report ■ Report dissemination/Following up

12 How Is the Evaluation Conducted? Evaluation Methodology ■ Desk Study: Cover all the 84 projects ■ Field Visits: 35 projects were chosen, 19 finished and 16 ongoing projects ■ Questionnaires: 68 were returned from end users and 60 from suppliers ■ Stakeholder Workshops: 17 provincial-level workshops were held during the field visits. More than 300 stakeholders were invited to participate in the workshops ■ Cross-verified & In-depth Analysis: The information in the fact sheets and the evidence collected from the field visits were cross-verified to support in-depth analysis; the database of initial evidence for the 35 visited projects has been built

13 The Responses from Stakeholders ■ The responses from Chinese stakeholders ■ The responses from Dutch stakeholders ■ The responses from other development agencies ■ The policy reaction in China and the Netherlands

14 The Dissemination of the Evaluation The dissemination has been given attention from the beginning. Some dissemination activities are planned and some are not predicted. ■ The final evaluation reports published in both Chinese and English ■ The dissemination workshop in Beijing China, March 29, 2007 ■ Communicate this evaluation case at the evaluation training courses (IPDET, ShIPDET and other national evaluation training workshops in China)

15 Comments for the Evaluation ■ The key benefits of the evaluation ■ The limitation of the evaluation ■ Potential challenges of the country-led joint evaluation ■ Making the decision to undertake a joint evaluation

16 The key benefits of the Country-led Joint Evaluation ■ Offer great opportunities for mutual capacity development ■ Increase ownership, the evaluation reflects better the voice of the partner countries ■ More relevant to the needs of both sides ■ Sharing the burden of work of evaluation

17 The limitation of the Country-led Joint Evaluation Institutional limitation in China ■ A national evaluation system has not been set up ■ No one in-charge of the official policy reaction ■ Lack of special budget for evaluation

18 The Potential challenges of the Country-led Joint Evaluation ■ The biggest challenge comes from different evaluation cultures and evaluation systems ■ Ensure that the evaluation meets the needs of both countries ■ Build consensus to design and conduct evaluation (such as: Identifying the key Issues in writing TOR, choice of field study cases) ■ Engage the key stakeholders in the design, conduct, and interpretation of evaluation

19 The Potential challenges of the Country-led Joint Evaluation ■ Develop capacity of partner countries to do AND use evaluation ■ Ensure that evaluation is USED appropriately ■ Enable partner countries to evaluate themselves rather than having evaluation done to them.

20 Listen to your Opinions and Advice ■ What do you think of this joint evaluation? What are the advantages and weaknesses? ■ How to promote the country-led joint evaluation as a successful evaluation model? ■ What is the driving force for this kind of evaluation? What are the major barriers to ? Lack of demand or weak evaluation capacity ? ■ What are the major barriers to the country-led joint evaluation ? Lack of demand or weak evaluation capacity ? ■ Donor’s role in CLEs, particularly in the “first generation” of CLEs, by nurturing the country’s demand and facilitating evaluation activities.

21 Thank you Thank you Any Questions, Opinions & Advices … … Sixth Meeting of the DAC Network on Development Evaluation June 2007, Paris