1. A Study of the Model and Methodology for Institute Evaluation in CAS Xiaoxuan Li ， Bing Shi ， Jianzhong Zhou Institute of Policy and Management ， CAS Management Innovation and Evaluation Research Center, CAS

2. Contents 1.Brief introduction of CAS and Evaluation Practice in CAS Institutes 2.Three-hierarchy Evaluation Model 3.Conclusions

3. Brief Introduction of CAS and Evaluation Practice in CAS Institutes

4. Brief introduction of CAS  The Chinese Academy of Sciences (CAS) was founded in Beijing on November 1, 1949. Consisting of the Academic Divisions and various subordinate institutions, it is the lead national academic institution in natural sciences, a major advisory body to the government on science and technology related issues, and a national comprehensive research and development center in natural sciences and high technology areas in China.  The general objectives of CAS are to develop into a base for scientific research, for training high caliber scientific talent and for incubating high-tech industries in China; to become a national scientific think tank and to evolve into a national research institution that boasts “first-class achievements, first-class efficiency, first-class management and first-class talent.”

5. CAS Academic Divisions CAS Headquarters Main Infrastructure － 17 Large-scale Scientific Research Facilities － 7 National Labs － 5 Field Stations Networks － 36 National Engineering Centers － 273 Technology Transfer Centers － 317 Journals － 46 National Associations and Societies –Division of Mathematics and Physics –Division of Chemistry –Division of Life Sciences and Medicine –Division of Earth Sciences –Division of Information Technical Sciences –Division of Technological Sciences Members 709 Foreign Members53 Committee for Consultation and Review Committee on Scientific Ethics Committee for Science Popularization and Publication Institutions Directly under CAS － 94 Research Institutes － 2 Universities and Schools － 2 Supporting Units － 3 Botanical Gardens － 12 Branches － 2 Press and Publication Companies － 1 Assets Management Company － 22 Holding Enterprises Distributed in 22 provinces and cities over China. Overview of CAS

6.  In 1998, with the approval of the Chinese government, the CAS launched the Knowledge Innovation Program (KIP), and has since turned over a brand new page in its history of development.  In the process of conducting the KIP, in recognition of the national strategic requirements and the world trend in science and technology, the CAS has made the most profound and extensive adjustments in its disciplinary deployment and organizational structure since its founding in 1949. Scientific restructuring and management innovation has led to the preliminary formation of a structure and mechanism that are both geared to a socialist market economy and aligned with international standards.

7. Changes of Concepts in Institutes Evaluation in CAS 15Years/3Period  From 1993 to 1998, the institutes evaluation in CAS was focused on research outputs,it is called output evaluation.  From 1998 to 2004, the CAS launched the Knowledge Innovation Program (KIP), the institutes evaluation in CAS was focused on achievements and level of accomplishment, it is called performance evaluation.  From 2005 to now, the institutes evaluation in CAS was focused on S&T creativity and concerned performance in integrity, it is called quality-orientated comprehensive evaluation.

8. Three-hierarchy Evaluation Model

9. Background and Tasks Comprehensive Complementary Reform Pilot Project –In 2007, seven pilot institutes have been selected to investigate modern R&D institutions system –Four thematic issues haven been launched in the following aspects respectively: Resource Allocation, Human Resources, Organizational model of R&D, Evaluation. –Basic Tasks of Evaluation Issue: To study evaluation models for different types of institutes.

10. Categories of 7 Pilot Institutes Institute of High Energy Physics Cold and Arid Regions Environmental & Engineering Research Institute Shanghai Institute of Technical Physics Institute of Computing Technology Dalian Institute of Chemical Physics Institute of Microbiolo gy Institute of Physics Large scale scientific facilitiesLarge scale scientific facilities Resources, Environment and institute in western ChinaResources, Environment and institute in western China Engineering and big teamsEngineering and big teams Basic researchBasic research High-tech ResearchHigh-tech Research Comprehensive instituteComprehensive institute Biological researchBiological research

11. Yearly quantitative and qualitative monitoring Peer review At the bottom hierarchy are indicators that can reflect common characters of the institutes, including yearly quantitative monitoring and yearly communication review At the second hierarchy are the key indicators and benchmarks, which reflects the characters of certain category of institute. It can be used in both diagnosing evaluation for individual institute and comparison between institutes of the same category. At the top hierarchy is qualitative evaluation by experts review, and it is also a hierarchy of individual evaluation. Experts review include two stages: diagnosing evaluation for individual institutes and comparing evaluation among the institutes of the same category. QuantitativeQualitative Three-hierarchy Evaluation Model Key indicators

12. Key indicators and benchmark  The selection of key indicators: Based on the orientation and characteristics of institutes, we selected no more than 6 key indicators for each pilot institute. Totally, there are 3 common key indicators and 8 specific key indicators.  There are four methods to set the benchmark in our research : 1) the more the better, for example the more awards, the better; 2)different standards will be given based on international comparison; 3)bechmark will be set according to the development level of institutes; 4)judged by experts experience.

13. key indicators of 7 pilot institutes 共性指标： 不同研究所的 共性因素 Common key indicators S&T Talents per capita funding Awards Specific key indicators Invited reports of important conference or workshop Accomplishment of Major Tasks Construction and operation of large research facilities Platform of resource and data Intellectual Property High quality papers Engineering Application and Demonstration Advisory Reports

14. The methods to set common key indicators of benchmark CountryInstitution per capita funding (ten thousand dollars) per capita funding for r esearcher (ten thousand dollars) USANIH15.446.2 USANISTNA31.8 GermanMPG18.851.9 GermanHFG12.439.6 GermanFhG13.8NA FranceCNRS12.534.6 JapanRIKEN32.341.4 JapanAIST28.736.9 KoreaKIST17.727.5 AustraliaCSIRO13.915.6 indicatorsThe method to set benchmark S&T Talents Make use of the definition of quantitative monitoring on innovation capacity about the S&T Talents per capita funding With reference to international research institutions and the real per capita budget of this year. AwardsMore the better

15. The methods to set the specific indicators of benchmarks  Specific key indicators are determined by separately by the 7 pilot institutes.  The study is expected to be accomplished by the end of this year.

16. Conclusions ——Through the study of 7 pilot institutes, key indicators and benchmarks regarding different types of institutes were proposed, including key quantitative indicators and benchmarks as well as key qualitative indicators and anchoring method. Thereby, three hierarchy evaluation model has been set up. ——the new model puts more emphases on the character of evaluation by category and individual features of institutes. —— In summary, this model has developed Quality-orientated Comprehensive Evaluation system in methodology.

17. Thanks Contact information: www.casipm.ac.cn jzzhou@casipm.ac.cn