A Novel Evaluation System of Public Research Institute in Japan A Novel Evaluation System of Public Research Institute in Japan International Workshop on the Evaluation of Publicly Funded Research Berlin, September 2005 Naoto Kobayashi National Institute of Advanced Industrial Science and Technology (AIST)

CONTENTS 1.Missions and Principles of AIST 2.Evaluation Policy in AIST 3.Novel Evaluation System from the Viewpoint of Outcome 4.Evaluation in the Future for the Publicly Funded Research Institutes

3 Interdisciplinary Researches to enable to plan long-range governmental policies attributing to the current and future needs of society Basic Researches to develop and maintain high standards of scientific and engineering research Advanced Researches to explore broad spectra of research fields and to integrate multidisciplinary subjects Standard & Measurements Type-II Basic Research & Full Research The mission of AIST is to generate innovative technologies for the new industries and to contribute to the economic development of our country and to the lives of people and, thus, finally to the sustainable global world!! Missions of AIST: for all mankind and global consciousness to our future Missions of AIST in association with the Ministry of Economy, Trade and Industry (METI) Trade and Industry (METI) Environment Energy Resources Geology Life Science Information Material Machine Nanotechnology Manufacturing Communication Electronics Oceanography

4 Auditor Fellow Advisory Board Technology Information Dept. Collaboration Dept. Intellectual Property Dept. International Affairs Dept. General Administration Dept. Human Resource Dept. Financial Affairs Dept. Research Facilities Dept. 〔 Sites 〕 AIST Hokkaido AIST Tohoku AIST Tsukuba AIST Tokyo AIST Chubu AIST Kansai AIST Chugoku AIST Shikoku AIST Kyushu AIST Innovations (TLO) AIST Innovations (TLO) President Senior Vice-President Vice-President President Senior Vice-President Vice-President Evaluation Dept. Safety & Environmental Protection Dept. Tsukuba Advanced Computing Center International Patent Organism Depository Planning Headquarter Planning Headquarter General Administration Headquarter General Administration Headquarter Research Center (28) Research Institute (21) Auditors’ Office Research Coordinator Research Coordinator Emeritus Advisor Emeritus Advisor Emeritus Councilor Emeritus Councilor AIST Innovation Center for Start-ups Public Relations Dept. Research administration departments Research support departments Research Initiative (5) Geo-information Center Metrology Management Center June, 2005 Researchers: ~2,500 Administrative staff: ~720 Budget: ~900M$

5 AIST Chugoku AIST Shikoku AIST Kansai AIST Chubu AIST Tohoku AIST Hokkaido AIST Kyushu AIST Tsukuba 産総研 Tokyo Headquarters AIST Tokyo Waterfront old AIST 15 Research Institutes 8 Institutes in Tsukuba 7 Institutes in other regions new AIST Merged into 1 Institute 9 Research Basis Research Bases of AIST In 2001

6 Concept of Full Research in AIST Concept of Full Research in AIST AIST defines Type-II Basic Research, which integrates multiple disciplines and creates methods for the use of integrated knowledge; traditional basic research is defined as Type-I Basic Research, which is the pursuit and discovery of novel laws, principles and theories that govern natural phenomena. AIST places its highest priority on the pursuit of complete research, "Full Research", ranging from Type-I Basic Research to the development of “products” by conducting intensive Type-II Basic Research. Type-I Basic ResearchType-II Basic Research Contribution to the Society Coherent and concurrent research based on scenario Full Research Unknown Phenomena Unknown Phenomena Economic & Social Needs Economic & Social Needs Law Integration of Knowledge Finding of New Knowledge Application Development Principle Theory Theorem President Dr. Yoshikawa

7 Sustainable Development by Full Research AIST Society Public resources Outcomes Full Research People’s Interest (applause) dreamnightmarereality Type-1 basic research Type-2 basic research Industrial development AIST UniversitiesIndustries Innovation!!

8 ： R&D Evaluation Guideline (1997, 2001, and 2005 ） ： ・ appropriate and fair evaluation→ flexible, competitive, and open research environment ・ appropriate and fair evaluation→ flexible, competitive, and open research environment ・ encouraging evaluation → original and promising R&D, researchers with great ambitions and ・ encouraging evaluation → original and promising R&D, researchers with great ambitions and abilities, and significant policy and program abilities, and significant policy and program ・ announcement of results of evaluation→ accountability to the nation ・ announcement of results of evaluation→ accountability to the nation ・ strict evaluation→ concentrated and effective allocation of resources ・ strict evaluation→ concentrated and effective allocation of resources ⇒ ⇒ internationally high level of R&D which contributes to society and economy, and to development of new discipline ： R&D Evaluation Guideline (1997, 2001, and 2005 ） ： ・ appropriate and fair evaluation→ flexible, competitive, and open research environment ・ appropriate and fair evaluation→ flexible, competitive, and open research environment ・ encouraging evaluation → original and promising R&D, researchers with great ambitions and ・ encouraging evaluation → original and promising R&D, researchers with great ambitions and abilities, and significant policy and program abilities, and significant policy and program ・ announcement of results of evaluation→ accountability to the nation ・ announcement of results of evaluation→ accountability to the nation ・ strict evaluation→ concentrated and effective allocation of resources ・ strict evaluation→ concentrated and effective allocation of resources ⇒ ⇒ internationally high level of R&D which contributes to society and economy, and to development of new discipline Council for Science and Technology Policy Independent Administrative Institution Act 2001 ・ setting Evaluation Department ・ annual evaluation 2001 ～ 2004 ・ setting Evaluation Department ・ annual evaluation 2001 ～ 2004 Introduction of evaluation system in AIST How to evaluate R&D in AIST as an Independent Administrative Institution promoting R&D ・ auto-evaluation ・ auto-evaluation ・ ・ evaluation interval ・ outcome How to evaluate R&D in AIST as an Independent Administrative Institution promoting R&D ・ auto-evaluation ・ auto-evaluation ・ ・ evaluation interval ・ outcome 2003 ～ 5 AIST Evaluation System Advisory Committee 2003 ～ 5 Evaluation from the viewpoint of Outcome

9 Medium-Term Plan Medium-Term Target Annual Report Evaluation Department Evaluation of Each Unit’s Research Units (Center, Institute and Initiative) Advisory Board Comment Comment for Management Evaluation of Activity Results of Evaluation Evaluation Committee for IAI in METI Report President Reflection on Future Management Evaluation Committees (Peers, Specialists or Intellectuals, Internal Reviewers) Research Resources Evaluation System of AIST

10  To evaluate research activities is to encourage researchers with great ambitions and abilities and to assist research organizations which contribute to the development of economy and the quality of our lives.  The top management of AIST requires the recognition of check and balance function in the evaluation of research activities, to consider the allocation of resources for research and to be equal to the occasion such as the reorganization.  AIST is also responsible for the accountability for public resources by showing performances and achievements to the society.  To evaluate research activities is to encourage researchers with great ambitions and abilities and to assist research organizations which contribute to the development of economy and the quality of our lives.  The top management of AIST requires the recognition of check and balance function in the evaluation of research activities, to consider the allocation of resources for research and to be equal to the occasion such as the reorganization.  AIST is also responsible for the accountability for public resources by showing performances and achievements to the society. of Evaluation Significance of Evaluation

11 Toward Outcome Every a few years Based on Output Every year The 1st Term (FY )The 2nd Term (FY )  Activation and Increasing in Efficiency  Making sure the research strategy  Reference to the Managements Purpose: Novel Evaluation System of Research Units ・ Reconsideration of roadmap ・ Output Indexes to outcomes ・ Management procedure Roadmap toward outcomes Research Units StartupInterimSummarized RI and RC Following RC Evaluation Intervals Change

12 Transfer Startups New Industry New Market Market ShareSample Novel Idea Output Research in AIST Pattern of Outcome Creation (Product Type) Acceptance in Market & Society Market Outcome Solar Panel Liquid Crystal Display Collaboration/ Technology Transfer (Prototype ・ Improvement ）

13 DB Construction New Industry New Market Research in AIST System Development System Improvement ・ Data Accumulation Acceptance in Domestic and International Society α Version ， β Version Supply DB Open to the Public Extension of External Access System Adjust and Implementation in Startups and/or Companies System Adjust and Implementation in AIST Collaborative Work Pattern of Outcome Creation (Database Type) TransferMarket Share Market Outcome Sample Novel Idea Output Acceptance in Market & Society Startups Organic Compounds Spectrum DB

14 ①： Evaluation of Realized Outcomes ②： Evaluation of Scenario/Roadmap ③： Evaluation of Outputs to contribute to Outcomes ④： Evaluation of Management to contribute to Outcomes ①： Evaluation of Realized Outcomes ②： Evaluation of Scenario/Roadmap ③： Evaluation of Outputs to contribute to Outcomes ④： Evaluation of Management to contribute to Outcomes Outputs in the Past Expected Outcomes in the Future Present Research PastFuture ① ： Realized Outcomes ② ： Roadmaps ③ ： Outputs ④ ： Management Evaluation is performed through Items ②～ ④ Evaluation from the Viewpoint of the Outcome

15 Evaluation from the viewpoint of outcome to reflect the future Element of Evaluation to evaluate the process of R&D Roadmap Output Management

16 external reviewers: Specialists or Intellectuals external reviewers: Specialists or Intellectuals ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation Evaluation: Rating with comments Evaluation: Rating with comments Evaluation Department Strategy of AIST President of AIST Research Units internal reviewers: Principal Reviewers internal reviewers: Principal Reviewers Procedure of Evaluation Nation Evaluation Committee Resource Allocation Reorganization Comments ＊ Roadmap for outcomes ＊ Roadmap for outcomes ＊ Output indexes ＊ Output indexes ＊ Management ＊ Management Roadmap Evaluation

17 Evaluation of Roadmap to make sure the research strategy Evaluation IndexTo know Outcome Social and economical effects MilestoneProgress plan Technical elementCore technique BenchmarkComparison with others

18 Example of Roadmap for Outcomes Example of Roadmap for Outcomes: Ultra-Low-Loss power devices Subjects (Targets) （ Power Electronics Research Center ） Substrate / Wafer Epitaxy Processing Packaging FET Devices Modules Applications InstrumentsInstruments  Few 10μm/h 3 inch Wafer Lamp Ballasts Factory Motor drive Factory Motor drive Electric Power Distribution System Dispersed Power supply Dispersed Power supply Power Transmission System Power Transmission System General Inverter （ＦＥＴ） （ＩＧＢ Ｔ） Technology transfer 600V10A 600V100A 2.5kV100A Low-EMI Bipolar Devices The 1st Term The 2nd Term R&D in AIST low-Ron 4 inch Wafer 6 inch Wafer Air-Con. Multi-Wafer-Epi. Self-Alignment Bipolar-Tech. High-reliable SiO2 Low-Temp Process T j >250 ℃ Wire-less Heatproof Capacitor-less

19 external reviewers: Specialists or Intellectuals external reviewers: Specialists or Intellectuals ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation Evaluation: Rating with comments Evaluation: Rating with comments Evaluation Department Strategy of AIST President of AIST Research Units internal reviewers: Principal Reviewers internal reviewers: Principal Reviewers Procedure of Evaluation Nation Evaluation Committee Resource Allocation Reorganization Comments ＊ Roadmap for outcomes ＊ Roadmap for outcomes ＊ Output indexes ＊ Output indexes ＊ Management ＊ Management Output Evaluation

20 Evaluation of Output Indexes to know the level of research unit PapersPress releases PatentsPrizes StandardsPrototypal products Activities at academic partyetc.

21 external reviewers: Specialists or Intellectuals external reviewers: Specialists or Intellectuals ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation Evaluation: Rating with comments Evaluation: Rating with comments Evaluation Department Strategy of AIST President of AIST Research Units internal reviewers: Principal Reviewers internal reviewers: Principal Reviewers Procedure of Evaluation Nation Evaluation Committee Resource Allocation Reorganization Comments ＊ Roadmap for outcomes ＊ Roadmap for outcomes ＊ Output indexes ＊ Output indexes ＊ Management ＊ Management Management Evaluation

22 Evaluation of Management LeadershipProject formation BudgetIntellectual property Human resourceTechnology transfer CollaborationStart-ups etc. Concept of Full Research : Policy statement

23 external reviewers: Specialists or Intellectuals external reviewers: Specialists or Intellectuals ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation ＊ Roadmap Evaluation ＊ Roadmap Evaluation ＊ Output Evaluation ＊ Output Evaluation ＊ Management Evaluation ＊ Management Evaluation Evaluation: Rating with comments Evaluation: Rating with comments Evaluation Department Strategy of AIST President of AIST Research Units internal reviewers: Principal Reviewers internal reviewers: Principal Reviewers Procedure of Evaluation Nation Evaluation Committee Resource Allocation Reorganization Comments ＊ Roadmap for outcomes ＊ Roadmap for outcomes ＊ Output indexes ＊ Output indexes ＊ Management ＊ Management Nation Evaluation Committee

24 Toward Outcome Every a few Years Based on Output Every year The 1st Term (FY )The 2nd Term (FY )  Activation and Increasing in Efficiency  Making sure the research strategy  Reference to the Managements Purpose: Novel Evaluation System of Research Units ・ Reconsideration of roadmap ・ Output Indexes to outcomes ・ Management procedure Roadmap toward outcomes Research Units StartupInterimSummarized RI and RC Following RC Evaluation Intervals Change

25 Mid-term Objectives by the minister ( e.g. METI) Mid-term Objectives by the minister ( e.g. METI) Mid-term Plans by each Institute Mid-term Plans by each Institute Check of Performances & Achievements Evaluation Committee in each ministry Evaluation Committee in Ministry of Internal Affairs and Communications Evaluation Committee in Ministry of Internal Affairs and Communications 3-5 years

26 Auditor Fellow Advisory Board Technology Information Dept. Collaboration Dept. Intellectual Property Dept. International Affairs Dept. General Administration Dept. Human Resource Dept. Financial Affairs Dept. Research Facilities Dept. 〔 Sites 〕 AIST Hokkaido AIST Tohoku AIST Tsukuba AIST Tokyo AIST Chubu AIST Kansai AIST Chugoku AIST Shikoku AIST Kyushu AIST Innovations (TLO) AIST Innovations (TLO) President Senior Vice-President Vice-President President Senior Vice-President Vice-President Evaluation Dept. Safety & Environmental Protection Dept. Tsukuba Advanced Computing Center International Patent Organism Depository Planning Headquarter Planning Headquarter General Administration Headquarter General Administration Headquarter Research Center (28) Research Institute (21) Auditors’ Office Research Coordinator Research Coordinator Emeritus Advisor Emeritus Advisor Emeritus Councilor Emeritus Councilor AIST Innovation Center for Start-ups Public Relations Dept. Research administration departments Research support departments Research Initiative (5) Geo-information Center Metrology Management Center June, 2005 Researchers: ~2,500 Administrative staff: ~720 Budget: ~900M$

Improvement in services Increase in efficiency Activation of operation ・ Setting of “Proposal Box” for accepting opinions to improve administrative work ・ List-up and sorting of the importance of work ・ Re-arrangement of the staffs so as to maximize the total efficiency of the work ・ Quantitative target-setup ・ Preparation of “Work Flow Manual” Aim: Support for research activities of AIST More effective execution of administrative and supportive work Aim: Support for research activities of AIST More effective execution of administrative and supportive work Evaluation of Research Administration Departments and Research Support Departments Evaluation of Research Administration Departments and Research Support Departments

28 Strategy ImplementationEvaluation Plan Do See Evaluation Planning Management Research Input Output Report Advice Plan Action Do Check Strategy of Evaluation Evaluation of R&D Policy Proposal Evaluation of R&D in the Research Institute

What is Innovation? ◆ Adequate Evolution of the Economy, Society and Environment based on Novel Science & Technology The trend to intensify the interface of Science and Industrial Innovation is prevailing at universities and public research industries in the world. Especially it is necessary to make breakthroughs by technological fusion in the interdisciplinary fields (ICT, Nanotechnology, Biotechnology etc.).

R&D Industrial Society Innovation!!  Industrial Transformation is necessary toward the Sustainable Global Society.  Innovation on the Basis of Novel R&D is necessary for the Industrial Transformation.  Adequate Evaluation System of R&D which contributing to the Innovation is important, especially for the Publicly Funded Research Institutes.  Evaluation from the Viewpoint of the Outcome is one of the attractive tools for the novel evaluation. Conclusion: Evaluation for Innovation

31 Evaluation is for Encouraging with Love. Evaluation is for Communication. Evaluation is for Strategy. Evaluation is for Creation. Evaluation is to be fed back. Evaluation reflects the Future. Notes: Evaluation is like a Compass for Voyage.

32 Thank you !

33 Appendix

Definition of independent administrative institution 【 Performance 】 - to be highly public - need not to be performed by government directly - hard to be achieved by private organization independently ① effective and with high quality ② independent ③ open 【 Performance 】 - to be highly public - need not to be performed by government directly - hard to be achieved by private organization independently ① effective and with high quality ② independent ③ open

35 ① To perform effectively and qualitatively - to set mid-term objectives and to receive periodical evaluation by third party - to entrust some service to private organization or to abolish institutions - to manage as a private corporation - to organize minimum size - to set mid-term objectives and to receive periodical evaluation by third party - to entrust some service to private organization or to abolish institutions - to manage as a private corporation - to organize minimum size

36 ② To perform independently -to concentrate authority to president -to exclude the excess involvement of minister -to have an appropriate personnel including executives from private corporation -to concentrate authority to president -to exclude the excess involvement of minister -to have an appropriate personnel including executives from private corporation ③ To perform openly - to open all performances and achievements to the public

37 Evaluation Committee Specialists e.g. Professor, or Head of other IAI Intellectuals e.g. Head of enterprise, or Journalist 【 outside of AIST 】 【 inside of AIST 】 Principal Reviewers etc.

38 Evaluation Interval in the 2nd Research Phase Evaluation Interval in the 2nd Research Phase 研究ラボ ・研究センターの創出 ・時限的 Research Initiative ・ to be Research Center ・ time limit A new Research Center Middle Evaluation Advise for Roadmap 研究部門 ・技術ポテンシャルの発展 ・継続的 Research Institution ・ bottom up ・ no time limit Advise for Roadmap Terminal Evaluation Middle Evaluation 研究センター ・産総研の看板 ・時限的 Research Center ・ top down ・ time limit Advise for Roadmap Terminal Evaluation Middle Evaluation Start up

39 ～ Organizational shift in accordance with society needs at the times ～ Merging research fields and Administrative needs Industry Type Research Areas Concept for Reorganization Catching up advanced nations Aiming at Front Runner Aiming at COE Recovery from World War II Machinery Microorganic Industry Fiber & Polymer Electrical Electronics Production Science 2 Regional Institutes Mining Resources Resources & Pollution Weighing & Testing National Research Laboratory of Metrology 7 Regional Institutes Nat’l Inst.for Advanced Interdisciplinary Research Mechanical Engineering Laboratory Geological Survey of Japan Fermentatio n Alcohol Textile Industrial Arts Geological Survey Chemical Technology Tokyo National Research Laboratory National Institute of Materials and Chemical Research Electrotechnical Laboratory National Institute for Resources and Environment 7 Regional National Industrial Research Institutes Weights and Measures Training Institute 1948 Establishment of Former AIST National Institute of Bioscience and Human-Technology National Institute of Advanced Industrial Science and Technology History of AIST