1. Peer Review in CAS Institute Evaluation Zhou Jianzhong, Zhou Changhai, Li Xiaoxuan Institute of Policy and Management, Chinese Academy of Sciences Assessment Office, Bureau of Planning Strategy, Chinese Academy of Sciences 2012.10.24-27 Minneapolis

2. Outlines 1.A brief introduction of CAS 2.The practices of peer review in the evaluation of institute of CAS 3.A New model of Peer review in CAS institutes: “ONE-THREE-FIVE” Expert Diagnosis Assessment 4.Discussion and conclusion

3. 19492010 History & Position CAS was Founded on Nov. 1, 1949 ◆ Highest academic institution in natural sciences in China ◆ A national comprehensive research and development center in natural sciences and high technology areas in China ◆ A major advisory body to the government on science and technology

4. 2011 Staff ： 60700 Graduate Students ： 49000 CAS Academic Divisions CAS Headquarters Main Infrastructure － 17 Large-scale Scientific Research Facilities － 7 National Labs － 5 Field Stations Networks － 36 National Engineering Centers － 273 Knowledge Transformation Centers － 317 Journals － 46 National Associations and Societies – Division of Mathematics and Physics – Division of Chemistry – Division of Life Sciences and Medicine – Division of Earth Sciences – Division of Information Technical Sciences – Division of Technological Sciences Members 709 Foreign Members53 Committee for Consultation and Review Committee on Scientific Ethics Committee for Science Popularization and Publication Institutions Directly under CAS － 100 Research Institutes － 2 Universities and Schools － 2 Supporting Units － 3 Botanical Gardens － 12 Branches － 2 Press and Publication Companies － 1 Assets Management Company － 22 Holding Enterprises Overview of CAS

5. Jilin （ 3 ） Liaoning （ 4 ） Shandong （ 3 ） Shanghai （ 11 ） Jiangsu （ 6 个） Anhui （ 1 ） Hubei （ 5 ） Shanxi （ 1 ） Shanxi （ 3 ） Sichuan Yunnan （ 2 ） Guizhou （ 1 ） Hunan Fujian （ 2 ） Zhejiang （ 1 ） Guangdo ng （ 5 ） Qinghai （ 2 ） Gansu （ 3 ） Xinjiang （ 2 ） Beijing （ 40 ） 12 branches Beijing Branch Shenyang Branch Changchun Branch Shanghai Branch Nanjing Branch Wuhan Branch Guangzhou Branch Chengdu Branch Kunming Branch Xi‘an Branch Lanzhou Branch Xinjiang Branch Landscape of CAS affiliates （ 2010 ） Tianjin （ 1 ） inCAS affiliates could be found in 27 cities in 22 provinces, autonomous regions and municipalities directly under the central government

6. Layout of the main subject areas and research fields Four Aspects Coverage of the natural science and high technology of the main research fields  Basic Research Basic Research Basic Research  Life Sciences and Biotechnology Life Sciences and Biotechnology  Resources and the Environment Resources and the Environment Resources and the Environment  High Technology High Technology High Technology  Material Science and Technology  Life Science and Technology  Space Science and Technology  Earth Science and Technology  Information Science and Technology  Math and Big Science Facilities

7. Changes of Concepts in Evaluation 1993 ～ 1998 1999 ～ 2004 2005 ～ 2010 Output Evaluation  Focus on research outputs Output Evaluation  Focus on research outputs Performance Evaluation  Focus on achievements  Focus on level of completion Performance Evaluation  Focus on achievements  Focus on level of completion Comprehensive Quality Evaluation  Focus on S&T creativity  Care performance in integrity Comprehensive Quality Evaluation  Focus on S&T creativity  Care performance in integrity 15Years/3Period Analyzing tool Policy tool Incarnates guidance & inspiring Compare & construe states of development Managing tool Control & management of process Encouraging competition, enhance innovation capability and efficiency by performance evaluation and process control. The development of institute evaluation in CAS

8. Give priority to quantitative evaluation Qualitative evaluation Ranking of institutes Group by sort AnnuallyAnnually Evaluate every five year Isolated quantitative evaluation and peer-review Integration of quantitative quality assessment and peer-review Combination of self-evaluation & focus group discussion Simple external evaluation Comprehensive evaluation of Performance, management process, future development Performance evaluation only Changes of Evaluation Concepts and Methods of Institutes in CAS

9. The practice of peer review in the evaluation of institute of CAS StageObject and contentarrangementsExperts sources and types 1998-2001: Extent of completion of objectives The completion of scientific and technological objectives entered into between the institute and CAS Communication assessment Peers: domestic experts, including academic leaders and renowned experts inside and outside CAS; the proportion of experts inside CAS to those outside CAS was 1:2 2002-2004: Major results of scientific and technological innovation Major results of scientific and technological innovation, including the innovativeness, influence and the level of academic leaders and their teams (1-5 items) Communication assessment Peers: mainly experts outside CAS, accounting for 73% of the total of experts, and academicians accounting for 23% 2005-2010: Comprehensive Quality Evaluation the overall quality and level of the institute, and the international status and level of research units and fields On-site assessment 33 international experts Communication assessment 67 international experts Meeting assessment 221 Domestic peers and user experts Innovation outcome talents Communication assessment

10. criteria : Have considerable popularity in certain discipline Have generally experience in research management Currently in the first-line of research mechanism : The institute recommends candidates of about two times the number of review experts wanted The profession office of CAS authorities reviewed the list of experts Inviting experts in the name of the president of CAS to participate in the review Experts selection criteria and mechanism

11. Organization and procedural arrangements Communication assessment :  The Evaluation Research Center(ERC) contacts the assessment experts that have been determined, sends evaluation materials and questionnaires  Assessment experts give assessment feedback within the specified period  The assessment center organizes the review results to form the assessment report On-Site assessment:  CAS leaders introduce the basic situation of CAS and the background and requirements of assessment to international assessment experts  The experts group hear reports of the institute director and of a number of academic leaders  Visit labs and discuss with scientists and graduate students  The experts group carries out a closed meeting to communicate with the leading body

12. Issues The nature of scientific research activities of different research need different experts  The peer review system is suit to evaluate the basic research and the experts is easy to select from international.  However, the peer review isn’t suit to evaluate the applied research and high-tech research, which meets national strategic needs and national economic development.  Perhaps some research work is low levels by experts’ opinion, but for China's economic and social development is of great significance, such research work should not invite international experts.

13. How to select the experts and provide the criteria of evaluation  The selection of experts, as well as the evaluation criteria of scientific and technical work have been always plagued us. The International criteria accepted peer-reviewed in the scientific community is mostly innovative in the science and technology. This is the base for compare the different research work.  However, The general objectives of CAS are to develop into a base for scientific research, for training high caliber scientific talent and for incubating high-tech industries in China; to become a national scientific think tank and to evolve into a national research institution that boasts “first-class achievements, first-class efficiency, first-class management and first-class talent.” a lot of work of CAS are to meet national needs. Therefore, evaluation criteria can’t only be just the scientific research criteria, the evaluation experts can not only be scientists.

14. H ow to make use of the results of evaluation  The results of experts evaluation are not comparable because different experts use the different evaluation criteria. Some experts will give the lower evaluation that refer to the highest international research level criteria. Some expert will give the higher evaluation with reference to the relevant domestic research level criteria.  The discrepancy of expert evaluation results was less and was difficult to distinguish grade. Because the experts was recommended by institute and have the better relations with the institute. So these experts are easy to give a good evaluation result. So the management is difficultly to makes use of the result of the peer review.

15.  In 2010, the innovation capacity of CAS institutes had been greatly improved, the level of the staff was advanced and the institute entered a new stage and class in the CAS. The 105th executive meeting of the State Council fully affirmed the achievements made by KIP and adopted the “Innovation 2020” of CAS.  Innovation 2020 aims at solving major scientific and technological problems in basic, strategic and prospective research fields that concerning the overall and long-term development of the country.  In order to fulfill the goal of “Innovation 2020”, CAS put forward new strategies in the new developing era. That is “one-Three-Five” strategy ： One Positioning, Three Major Breakthroughs, Five Cultivating Directions. A New model of Peer review in CAS institutes: “ONE-THREE-FIVE” Expert Diagnosis Assessment

16. The Twelfth Five-Year Development Plan During the Twelfth Five-Year (2011~2015), required as the strategic planning called “One-Three-Five”, each institute of CAS will promotes its develop plan ， identify its position, and define its research priorities with potential breakthroughs and its prospective layout. One  One Positioning indicates each CAS institute should specify its major research areas, unique features, core competitiveness, and anticipated position in international circles and should avoid homogenization with other CAS research institutes. Three  Three Major Breakthroughs indicate major basic, strategic and prospective S&T innovative achievements to be made in the next five to ten years; generally each institute shall raise no more than 3 breakthroughs. Five  Five Potential Directions indicate research priorities with unique features, a future competitive advantage and potential breakthroughs; generally each institute shall set no more than 5 priorities.

17. During the 12 th five-year plan ， CAS has developed a major R&D outcome-oriented evaluation system Clarifying the value guidance ： 6 types of major innovation contribution and innovation in talents and mechanism Designing reasonable assessment process  Judging performance  Diagnosing and monitoring key element which is the benefit of performance  Key Scientific Issues  Key Core Technology  Open new directions  Significant Social and Economic Benefits  Systematic Solution  Great influence on decision making

18. 2 sections 1 base structure SectionsAim Supporting platform Theory tank 、 tank of methods and tools 、 expert tank 、 database ， network research organization…… Based on the assignment bookBased on the assignment book To judge the target completionTo judge the target completion Emphasizing key breakthroughEmphasizing key breakthrough Checking target completion of key breakthrough During 2015 Diagnosing key areas and factors Diagnosing key areas and factors Guiding the development of the institutes Guiding the development of the institutes Driving the implementation of “1,3,5” Driving the implementation of “1,3,5” “1-3-5” Expert diagnosis assessment During 2013-2015 Holding the changing and development of core competitivenessHolding the changing and development of core competitiveness Serving expert diagnosis assessmentServing expert diagnosis assessment Monitoring key indicators of institutes yearly yearly The major research outcome-oriented evaluation system of CAS

19. Objective and role of “One-Three-Five” expert diagnosis assessment Inviting high level experts domestic and overseas  Diagnosing the positioning, advantages and disadvantages of the institutes  Diagnosing the research quality and technical value of the key fields, research significance and impact, talents, resources and technology platform etc.  Diagnosing the positioning, core competitiveness of the institutes and their position among relative institutes around the world  Diagnosing the positioning, advantages and disadvantages of the institutes  Diagnosing the research quality and technical value of the key fields, research significance and impact, talents, resources and technology platform etc.  Diagnosing the positioning, core competitiveness of the institutes and their position among relative institutes around the world Through the diagnosis assessment  Help the institutions to improve management ， clarify core advantage 、 avoid homogenization  Lay a foundation for the future major innovation contribution  Help the institutions to improve management ， clarify core advantage 、 avoid homogenization  Lay a foundation for the future major innovation contribution

20.  As a national research institution, each institute in CAS has its own mission and position. What kinds of expert can evaluate not only research quality but also the research work that meets the national strategic needs? So it needs the peer has different type. The peer of the institute evaluation is not only to judge the research quality based on international level, but also from the perspective of users invited experts to evaluate their capacity of meeting the national need.  Some institutes in CAS are the comprehensive research institutes which include basic research and applied research. That leads to institutions in CAS covering many disciplines and fields. So, how to ensure the peers can evaluate every discipline and field in the institute while controlling the expert group proper scale?  There are so many institutes in CAS and in different development levels. Some institutes in high level can be evaluated and measured by international benchmark. However, when some institutes are still in a relatively low level, it is difficult for making a judgment. Discussion and conclusion

21.  The object and content of review are more focused In the institute peer review of the major research outcome-oriented evaluation system of CAS, the diagnostic assessment is carried out for each institute every five years by domestic and foreign high-level peer experts and user experts mainly from the international perspective and national needs. The experts do not assess the entire institute or diagnose the entire research staff, but focus on the advantages and core competencies of the institute and judge the advancement of the work of major fields.  The assessment results focus on diagnosis but are not linked with resource allocation In CAS’s new peer review, the assessment results are not linked with the allocation of resources but focus on diagnosis and help the institute to identify problems, grasp the direction, make a rational layout, improve management and enhance the efficiency of achieving significant achievements output. At the same time, the results are used to promote the exchanges between research staff and high-level experts in the same field and form the academic atmosphere of pursuing excellence and daring to innovate. Therefore, the institute has small pressure in the process of evaluation without too much burden and can organize the evaluation easily. This is a large difference from the previous peer review. Summary of the “1-3-5”Expert diagnosis assessment

22. Challenges  How to determine the criteria of Major Outcome?  How to select proper experts?  Can expert judge the Major Breakthroughs ?  ……

23. Thanks ！ Contact: jzzhou@casipm.ac.cn