Strategic Evaluation of Research –Viewpoint of Outcome in the Evaluation of Research Units in AIST– M. Daté, H. Tokunaga, S. Nakatsu, S. Ito, H. Matsuhata, H. Ogi, A. Omori, S. Suto, M. Tajima, T. Tanaka, K. Urabe, O. Nakamura*, O. Nakamura, S. Kosaka, M. Takagi Sawada and N. Kobayashi Natl. Inst. Adv. Ind. Sci. Technol. (AIST), JAPAN *Min. Econ. Trade Ind. (METI), JAPAN 1/15

Outline of AIST New AIST (National Institute of Advanced Industrial Science and Technology) was established in 2001 as an Independent Administrative Institute. Researchers Administrative Staffs Total Employees 2, ,209 As of April /15

Mission of AIST Basic Principle is “Realization of a Sustainable Society”. 3/15

Research Fields in AIST 4/15

Organization of AIST As of August /15 Collaboration Dept Auditor President Research Centers Research Institutes Research Initiatives Senior Vice-President Vice-President Emeritus Advisor AIST Advisory Board Research Coordinator Emeritus Councilor AIST Fellow Research Units administrative units Intellectual Property Dept Technology Information Dept International Affairs Dept General Administration Dept Human Resources Dept Financial Affairs Dept Research Facilities Dept Intl Patent Organism Depositary AIST Innov Center for Start-ups Geo-information Center Metrology Management Center Tsukuba Adv Comp Center (TACC) Planning HQs General Administration HQs Evaluation Dept Safety & Environ Protection Dept Public Relations Dept Legal Office Info Disclosure & Personal Info Protection Promotion Office Gender Equality Office Audit Office

Research Units of AIST Research Centers (29) Priority in investing research resources Operated by top-down management Pioneering and strategic projects within a certain period (<7 years) Research Institutes (21) Proposal of research subjects in a bottom-up manner Mid- and long-term researches Research Initiatives (7) Response to sudden administrative needs Aiming for new research centers or fields within a short period (<3 years) As of August /15

Evaluation of Research Units Purposes of evaluation 2) Reference to the top management of AIST 3) Accountability to the Japanese Nation 1) Activation and efficient management of Research Units 1st Mid-Term (FY ) 2nd Mid-Term (FY ) Evaluation based on Outputs and global level of research Evaluation from the viewpoint of Outcome Viewpoint of evaluation 7/15

Definition of Terms in AIST Outputs: Direct results in R&D (e.g., papers, patents, etc.) ・ Budget ・ Human resource ・ Equipment ・ Academic paper ・ Patent ・ Geologic Map ・ Standard ・ Software ・ Process Develop. ・ Database ・ Popularization of products of products ・ New research field ・ Service ・ Saving energy ・ Risk management ・ Disaster prevention prevention INPUTIMPACTOUTPUTOUTCOME ・ GDP growth ・ New employment ・ International competition ability competition ability ・ High QoL ・ Environmental protection protection ・ Security & safety society society Outcomes: Social and economical effects caused by outputs (e.g., products, standards, novel research fields, etc.) 8/15

Evaluation System in AIST Outputs in the past Realized Outcomes Expected Outcomes in the future 1) Roadmaps 2) Outputs 3) Management PASTPRESENTFUTURE Subjects of evaluation 2) Evaluation of Outputs toward Outcomes 3) Evaluation of Management toward Outcomes 1) Evaluation of Roadmaps toward Outcomes 9/15

Evaluation Indexes Evaluation of Roadmaps Outcomes, Milestones, Technical Elements, Benchmarks Evaluation of Outputs Papers, Patents, Standards, Academic Activities, Press Releases, Prizes, Prototypical Products, etc. Evaluation of Management Research Strategy Other Management 1) Full Research, 2) Innovations, 3) Human Resource Development 1) Overall Management, 2) Budget Management, 3) Risk Management 10/15

Reviewers in Evaluation External Reviewers (5-7 for each Research Unit) e.g., academia, industry, journalism, etc. Specialists and intellectuals outside of AIST Internal Reviewers (3 for each Research Unit) Principal Reviewers of AIST, etc. Main Role: Evaluation of Roadmaps and Outputs Main Role: Evaluation of Management (Roadmaps & Outputs) : Management = 7 : 3 Allocation of grades 11/15

Evaluation Procedures Resource, Reorganization Evaluation Committee 1) Roadmaps 2) Outputs 3) Management Evaluation Department Evaluation Department Activity Report Presentation Activity Report Presentation Comments on Evaluation Claim of Reconsideration Comments on Evaluation Claim of Reconsideration Grades Comments Grades Comments Correction External Reviewers Internal Reviewers External Reviewers Internal Reviewers Research Unit 12/15 Evaluation Committee of Ind. Admin. Inst. Evaluation Committee of Ind. Admin. Inst. President Japanese Nation Research Strategy of AIST Overall Evaluation

Survey of the Unit Directors (1) Was the evaluation of this year useful? Was the evaluation worth all the efforts? FY2005 FY 13/15

Survey of the Unit Directors (2) Was the viewpoint of outcomes useful? Was the evaluation useful in improving management? FY2005 FY 14/15

Comments of External Reviewers CATEGOLYPERCENTAGE Affirmative comments on the evaluation system 20 Comments on the evaluation of roadmaps 14 Misunderstanding, insufficient explanation 14 Insufficient time or activity reports 12 Proposal to improve the evaluation system 10 Request for various evaluation indexes 9 Request for a longer interval of evaluation 8 Request to reduce the load of evaluation 6 Other comments 9 FY /15