The Roles of Users in Enhancing Utility of Evaluation With Special Reference to the International Evaluation on the Funding and Management Performance of the National Natural Science Foundation of China Prof. Yonghe Zheng, Deputy Director, Bureau of Planning, NSFC Prof. Zhaoying Chen, Deputy Director, NCSTE

Outline  Brief overview of the national evaluation effort in China  NSFC case: International Evaluation on NSFC’s Funding and Management Performance  The Roles of Users in Enhancing Utility of Evaluation

1. Brief overview of the national evaluation effort in China  Currently, to establish an evaluation system within the public administration system in China has become an urgent need in Chinese governments.  The whole – of - government performance evaluation system has not established in China.

MoF is the leading body  Ministry of Finance (MOF) is the leading body of the performance evaluation of public sectors in China.  There are already some pilot activities of performance evaluation, which carried out by line ministries or local governments in China.

2. NSFC case: International Evaluation on NSFC’s Funding and Management Performance  The evaluation is commissioned by MOF and NSFC together. The State Council attached importance to this evaluation.  The NSFC case is the most influential evaluation in public sector in China up to now and represents good practice from China.

6 Dual Objectives of the Evaluation 6 ■ Accountability Provide an independent assessment of the overall performance of NSFC’s funding and management during the past 25 years ■ Lesson learning Improve the NSFC’s funding and management performance; redefine NSFC’s strategic role within the NIS of China

7 What the evaluation focuses on and what the evaluation does not attempt to do 7 The evaluation does ■ Focus on NSFC’s funding and management performance over 25 years ■ Focus on NSFC’s overall performance as a whole The evaluation does NOT Assess NSFC from the agency’s internal perspective Assess individual projects supported by NSFC

Evaluation Approach and Principals  Modality: domestic preparation followed by international evaluation  Approach: Evidence-based  Principals: Independent, Objective, Fairness

The scope of the Evaluation will cover four dimensions: (1) NSFC’s strategic positioning, (2) funding performance, (3) management performance, (4) impact on China’s S&T system.

Evaluation Issues Issue 1. NSFC’s strategic positioning in the national innovation system of China Issue 2. Funding strategy Issue 3. Contributions to original innovation Issue 4. Promoting the balanced development of scientific disciplines Issue 5. Fostering innovative talents Issue 6. Supporting the national demands and challenges Issue 7. Modality of funding management Issue 8. Funding instruments Issue 9. Peer Review System Issue 10. Impact of NSFC’s funding

11 Implementation of the Evaluation 11 PhaseMajor Evaluation Activities Phase － Inception and evaluation design Phase － Prepare evaluation evidence report The IEC first meeting in Beijing Phase － Complete the Evaluation Report Phase 年 7 － 10 月 Use and dissemination of the evaluation results

12 Use of the Evaluation 12 ■ NSFC leadership understands why the organization needs evaluation ■ The use of evaluation has been considered to be key issues to be addressed in the evaluation design from the very beginning and also made systematic arrangements ■ During the implementation of the evaluation, some findings have already been put into application. ■ There will be a series of follow-up activities to ensure that evaluation is used appropriately

3. User’s Role in Enhancing Utility of Evaluation NSFC: Inception and evaluation design  Ensure that the evaluation is targeted to the demands of users, respond to current concerns and debates;  Discuss with NCSTE team about the conception framework of the evaluation; Identify the 10 key issues and evaluation questions;  Promote IEC and NCSTE team to understand NSFC’s funding and management system;

NSFC: Evaluation Implementation  Provide coordination and support requested by IEC and NCSTE team;  Prepare a volume of case-based evidence for the evaluation;  Assure that NCSTE can access to all relevant information and select the target group for interview or focus group meeting;  Judge the credibility of the message in the evaluation report 3. User’s Role in Enhancing Utility of Evaluation

NSFC: Response and Dissemination  Take management response to the evaluation report at the draft stage  Distribute the evaluation report to line ministries, the National People's Congress, the Chinese People's Political Consultative Conference  Arranging dissemination events of the evaluation. 3. User’s Role in Enhancing Utility of Evaluation

NSFC: Follow-up  NSFC leading level and director level: One week seminar to discuss the main results of the evaluation; identify possible action in NSFC strategy and funding management;  NSFC staff: improving conditions for evaluation-based learning in NSFC;  Release the report to the public 3. User’s Role in Enhancing Utility of Evaluation

 Primary users should really understand why they need evaluation;  The use and dissemination of evaluation should be addressed from the beginning and also built into the work plan of the evaluation. Lesson Learned

Thanks ！