Logic Model and Methodology for Knowledge Innovation Program(KIP) Evaluation in Chinese Academy of Sciences Tao Dai Nov. 4, 2011 Evaluation 2011: Values and Valuing in Evaluation Anaheim, California Institute of Policy and Management, CAS CAS center for management innovation and evaluation research

2 Evaluation 2011, Anaheim, California Outlines Background Logic model for KIP evaluation Methods for KIP evaluation Main evaluation results Conclusion

3 Evaluation 2011, Anaheim, California Background What is KIP?  launched and implemented by CAS  three phases: 1998~2000,2001~2005,2006~2010  aims: improve the scientific performance of CAS ;make CAS pre-eminent Science and technology (S&T) center Why do KIP evaluation ?  Chinese government’s requirement  CAS’ requirement

4 Evaluation 2011, Anaheim, California Background Challenges of KIP evaluation  Systematic evaluation on all the work CAS carried out  Cover 13 years  CAS-centered self-evaluation How to ensure the objectiveness and fairness ?

5 Evaluation 2011, Anaheim, California Logic model of KIP evaluation

6 Evaluation 2011, Anaheim, California Objectives of KIP “By 2010, CAS will have built itself into a knowledge creation center with powerful and continuous innovation capability, aiming at national strategic objectives and international scientific frontier. It will have become a scientific research base with its overall level reaching advanced international standards, and an important base for bringing up high-caliber talents, and a base for promoting the development of high-tech industries in China. It will have developed into a knowledge base, and become the state’s think tank and talent bank with international influences.”

7 Evaluation 2011, Anaheim, California Evaluation methods

8 Evaluation 2011, Anaheim, California progress report to the central government during each stage During the implementation of KIP, hundreds of experts, managerial professionals and users participated in the evaluation and presented precious suggestions and comments monitoring mechanism  The bottom layer: basic materials and data collected

9 Evaluation 2011, Anaheim, California 1) Self-evaluation and communicating review of the research institutes 100 institutes conducted self-evaluation firstly related materials of self-evaluation were circulated on internet reviewed by 201 experts from CAS headquarters, bureaus, and institutes  The second layer: thematic evaluation

10 Evaluation 2011, Anaheim, California 2) Consultant review on S&T innovation base each innovation base conducted self-evaluation 132 experts were invited to discuss and review based on the self-evaluation, through meeting review, corresponding review consultant review committee made the comprehensive review  The second layer: thematic evaluation

11 Evaluation 2011, Anaheim, California 3) Self-evaluation and summary of key aspects of management work strategic planning and disciplinary deployment construction of academic divisions contingent construction system reform and construction resource allocation and asset management Local and international cooperation infrastructure construction …  The second layer: thematic evaluation

12 Evaluation 2011, Anaheim, California 4) Seminar on deepening the reform of CAS 3-day seminar 150 directors from both academy and institute level focused on the key issues in CAS developments  The second layer: thematic evaluation

13 Evaluation 2011, Anaheim, California carried out simultaneously and interactively with thematic evaluation (the second layer) made full use of the existing evaluating results and data (the bottom layer) the report titled ‘The Evaluation Report on KIP ( )’ was formed and submitted to the state council of China  The top layer: Overall evaluation of KIP

14 Evaluation 2011, Anaheim, California Evaluation Results  Innovation capability improved significantly Number of high quality papers （ Top 1% citation ） in 2009 is 11.5-fold increase from 1998 Undertaking 33%“973 program” and 34% “NSFC Key Project” in 1998~2009 Operating 80% large-scale scientific facilities of China …

15 Evaluation 2011, Anaheim, California Evaluation Results  Cultivating and Fostering high-level S&T talents Bring in 1292 outstanding talents from overseasBring in 1292 outstanding talents from overseas About 900 scientists held important positions in international organization or international academic journalsAbout 900 scientists held important positions in international organization or international academic journals Number of Ph.D. graduate from CAS is about 8% of the total in China, contributing to 18.5% of the “outstanding doctoral thesis”Number of Ph.D. graduate from CAS is about 8% of the total in China, contributing to 18.5% of the “outstanding doctoral thesis” …

16 Evaluation 2011, Anaheim, California Evaluation Results  Effective in S&T achievements transfer Number of transfer programs of S&T achievements in 2009 isNumber of transfer programs of S&T achievements in 2009 is 5-fold increase from 1998 The total Business income (1998~2009) of CAS invested enterprises is aboutThe total Business income (1998~2009) of CAS invested enterprises is about 1,086 billion yuan The total profit and tax(1998~2009) of CAS invested enterprises is about 52 billion yuanThe total profit and tax(1998~2009) of CAS invested enterprises is about 52 billion yuan …

17 Evaluation 2011, Anaheim, California Evaluation Results  Very active in domestic and international cooperation Built 335 technology centers with companies Built 10 research institutes with local government Built national labs andnational key labs with universityBuilt 2 national labs and 6 national key labs with university Number of international academic conferences organized by CAS increased from 57 in 1998 to 354 in 2008 Number of International cooperation and exchange visitors increased from in 1998 to in 2009 …

18 Evaluation 2011, Anaheim, California Conclusions feasible to evaluate the performance of CAS in KIP according to the evaluation practice based on objective data, facts, and experts judgment qualitative evaluation was combined with quantitative evaluation; self-evaluation was combined with experts review

19 Evaluation 2011, Anaheim, California Thanks for your attention!