1. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Recent Experiences and Challenges of Research Program Evaluation in China: An Introduction to the International Evaluation of NSFC Funding and Management Performance XING Huaibin SHI Xiaoyong China National Center for S&T Evaluation (NCSTE) 25 October 2012, Minneapolis

2. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Provides a successful case of R&D program evaluation in China which has been done from 2010 to 2011 CASE: International Evaluation of NSFC’s Funding and Management Performance To show some recent experiences and challenges THIS PRESENTATION

3. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Founded in 1997, more than 50 staff now. www.ncste.org ABOUT NCSTE

4. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation OUTLINE Background Methodologies Implementation Future and Challenges

5. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Background Why to conduct this evaluation? ■ Performance of the government, including of the public funding on R&D have recieved more corcern in recent years in China ■ 2011 marks the 25th anniversary of NSFC ■ The Ministry of Finance and NSFC jointly commissioned this evaluation in Feb, 2010

6. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Background Purpose of This Evaluation ■ The Evaluation will, from historical angles, comprehensively examine NSFC’s funding and management performance, its strategic positioning over the past 25 years, and from an international perspective, assess its strengths, weaknesses as well as the challenges it faces.

7. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation OUTLINE Background Methodologies Implementation Future and Challenges

8. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Firstly,domestic preparation, international evaluation. Secondly, Issue and Key Question- Oriented Thirdly, Evidence-Based Methodologies

9. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Firstly ， “domestic preparation, international evaluation”. □ Domestic Preparation NCSTE, an independent and professional evaluation organization, will design the overall evaluation plan and prepare evidence needed for the Evaluation, with the support of NSFC. □ International Evaluation International Evaluation Committee (IEC) will evaluate the overall performance of NSFC’s funding and management based on the evidence prepared by NCSTE and its own investigations and international comparisons Methodologies

10. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Why need domestic preparation ■ The Evaluation covers a long time span and a thorough understanding of Chinese context are needed; ■ There are no systematic evaluation reports on NSFC; ■ Source materials are mostly in Chinese. Useful information needs to be selected from large amount of materials. ■ Most workshops and interviews should be implemented by NCSTE. Methodologies

11. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation domestic preparation conducted independently Independence of domestic preparation is reflected in ■ Institutional arrangement in NSFC ■ Ensure NCSTE team had full independence to conduct their work without interference ■ NSFC provided coordination and support request by NCSTE team ■ NCSTE team have the independence to: ■ Access to all relevant information ■ Select the target group for interview or focus group meeting ■ Conduct Questionnaire surveys ■ Present evidences and evidence based findings ■ Communicate with the IEC directly Methodologies

12. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation 12 Methodologies Secondly, Issue and Key Question-Oriented □ Strategic Positioning Issue 1 NSFC ’ s strategic positioning Q1-Q3 Issue 2 Funding strategy Q4-Q6 □ Funding Performance Issue 3 Contributions to originality and novelty in basic research Q7-Q8 Issue 4 Promoting the balanced development of scientific disciplines Q9-Q11 Issue 5 Fostering innovative talents Q12-Q14 Issue 6 Supporting the national demands and challenges Q15-Q17 □ Management performance Issue 7 Modality of funding management Q18-Q20 Issue 8 Funding instruments Q21-Q23 Issue 9 Peer Review System Q24-Q26 □ Impact Issue 10 Impact of NSFC ’ s funding Q27-Q30

13. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Methodologies Thirdly, Evidence-Based Ensure that the IEC has an adequate evidence base available for future evaluation Focus on answers to the evaluation questions defined in the TOR Both qualitative and quantitative evidences; multiple approach for evidence collection

14. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation 14 □ Strategic Positioning Issue 1 NSFC ’ s strategic positioning Q1-Q35 Evidences Issue 2 Funding strategy Q4-Q610 Evidences □ Funding Performance Issue 3 Contributions to originality and novelty in basic research Q7-Q84 Evidences Issue 4 Promoting the balanced development of scientific disciplines Q9-Q1112 Evidences Issue 5 Fostering innovative talents Q12-Q1414 Evidences Issue 6 Supporting the national demands and challenges Q15-Q176 Evidences □ Management performance Issue 7 Modality of funding management Q18-Q2011 Evidences Issue 8 Funding instruments Q21-Q2313 Evidences Issue 9 Peer Review System Q24-Q2613 Evidences □ Impact Issue 10 Impact of NSFC ’ s funding Q27-Q30 8 Evidences Evidence-Based Methodologies

15. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Evidence Collection ■ Cases prepared by the NSFC ■ Other evidence prepared by the NCSTE ■ Review of relevant policy documents ■ Analysis of NSFC documents and statistics ■ Bibliometric study on the SCI paper with NSFC ■ Interviews and focus groups with government officials, beneficiary institutions, Principal Investigators, peer reviewers and NSFC personnel ■ Questionnaire surveys of supporting institutions, Principal Investigators and peer reviewers of NSFC projects ■ Case analysis base on the Case Collection of the NSFC Methodologies

16. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation OUTLINE Background Methodologies Implementation Future and Challenges

17. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation organizational structure of the Evaluation Leading Group Evaluation Office IEC NCSTE Working Group Implementation

18. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Files and Reports Implementation International Evaluation Report Synthesis Evidence Report 10 issues study reprots cases report SCI papers report Questionnaire Survey report Interviews and Focus Group Meeting report Terms of Reference

19. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Type of IntervieweesNumber of Interviewees S&T management officials at the central government level 3 Academicians of CAS and CAE6 Former and current presidents and vice- presidents of NSFC 4 Staff of NSFC10 Total23 Interviews-4 types Implementation

20. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Focus Group Meeting-7 types Implementation Type of Focus Group MeetingNumberNumber of Participants With research managers of supporting institutions 540 With PIs of NSFC projects324 With academic leaders of related discipline14 With researchers of local colleges and universities 1194 With reviewers of proposals122 With rejected applicants642 With staff of NSFC323 Total30249

21. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Questionnaire Survey- 6 types Implementation Questionnaire SurveyNo. of Questionnaire Delivered No. of Questionnaire Returned Respo nd Rate supporting institutions2,257 84137.3% PIs of the General Program projects43,41210,22823.6% PIs of the Young Scientists Fund projects 16,092 6,10437.9% Grantees of the National Science Fund for Distinguished Young Scholar 2,164 44520.6% reviewers6,036 1,23220.4% rejected applicants7,838 1,37117.5% Total77,79920,22126.0%

22. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Contents and structure of the Synthesis Evidence Report by NCSTE ■ Full coverage of the requirements defined in the ToR ■ The logical framework: 4 dimensions, 10 key issues,30 questions and 96 evidences ■ The structure: Part A (Introduction), Part B (Background), Part C (Key issues and Evidences) and Annex Implementation

23. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation The Evaluation Report and Synthesis Evidence Report can be download at http://www.nsfc.gov.cn/nsfc/cen/pgbg/index.html

24. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation OUTLINE Background Methodologies Implementation Future and Challenges

25. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation  Evaluation increasingly received attention from central government but is still a lower priority.  Evaluation of R&D programs and projects has not been institutionalized in related ministries and agencies.  Evaluation results are seldom used in the research priority setting.  Lack of indicators to reflect the performance of different research program.  …… Future and Challenges

26. 国 家 科 技 部 评 估 中 心国 家 科 技 部 评 估 中 心 National Center for S&T Evaluation Many thanks for your attention!