1. 1 Naoto Kobayashi Vice President Director, Evaluation Department Outcome-based Strategic Evaluation of Research Units in AIST National Institute of Advanced Industrial Science and Technology (AIST) @Joanneum Research in Vienna on Feb. 12 h, 2007

2. 2 CONTENTS 1.Strategic Research Evaluation 2.Missions and Activities of AIST 3.Strategy and Evaluation 4.Evaluation System from the Outcome Point of View 5.PDCA cycle in the Research Institute

3. 3 Strategic Research Evaluation ◆ Strategic Planning and Effective Promotion of R&D Strategic Research Evaluation ； Research Evaluation Procedure and Implementation Suitable for Strategic Promotion of R&D Strategic Research Evaluation ； Research Evaluation Procedure and Implementation Suitable for Strategic Promotion of R&D ◆ Strategic Reflection of Evaluation Results to R&D ◆ Strategic Design and Implementation of Evaluation

4. 4 Evaluation Planning Management Research Plan Action Do Check Research Organization Evaluation System A A P P D D C C Research Units Research Institute （ AIST ） P P Evaluation Committee for AIST CSTP;Council for S&T Policy MIC;Ministry of Internal Affairs & Communications METI;Ministry of Economy, Trade & Industry IAI;Independent Administrative Institution DIET, CSTP P P Evaluation Committees for IAI’S A A C C C C P P C C A A R&D Administration (MIC) A A R&D Administration (METI)

5. 5 CONTENTS 1.Strategic Research Evaluation 2.Missions and Activities of AIST 3.Strategy and Evaluation 4.Evaluation System from the Outcome Point of View 5.PDCA cycle in the Research Institute

6. 6 Interdisciplinary Researches to enable to plan long-range governmental policies attributing to the current and future needs of society Basic Researches to develop and maintain high standards of scientific and engineering research Advanced Researches to explore broad spectra of research fields and to integrate multidisciplinary subjects Standard of measuring & weighing Type-II Basic Research Full Research The mission of AIST is to generate new technologies and new industries and to contribute to the economic development of our country and, thus, to improve the lives of people. With this mission in mind, AIST has been performing researches in the following areas. Missions of AIST: for all mankind and global consciousness to our future Missions of AIST Environment Energy Resources Geology Life science Information Material Machine Nanotechnology Manufacturing Communication Electronics Oceanography

7. 7 ◆ Top-down management by president of AIST ◆ Flexible reorganization of research units ◆ Complete research support by administrative units ◆ Top-down management by president of AIST ◆ Flexible reorganization of research units ◆ Complete research support by administrative units Technology Information Dept International Affairs Dept General Administration Dept Human Resources Dept Financial Affairs Dept Collaboration Dept Intellectual Property Dept Auditor President Safety & Environmental Protection Dept Tsukuba Advanced Computing Center (TACC) International Patent Organism Depositary Planning HQs Evaluation Dept Research Centers Research Institutes Research Initiatives Audit Office Geo-information Center General Administration HQs Research Facilities Dept Senior Vice-President Vice-President Emeritus Advisor AIST Advisory Board Research Coordinator Emeritus Councilor AIST Fellow Public Relations Dept AIST Innovation Center for Start-ups （ as of July 1, 2006 ） Metrology Management Center Organization of AIST Gender Equality Office Legal Office Information Disclosure & Personal Information Protection Promotion Office Research Units (21) (29) (7) (57) administrative units

8. 8 AIST Chugoku AIST Shikoku AIST Kansai AIST Chubu AIST Tohoku AIST Hokkaido AIST Kyushu AIST Tsukuba 産総研 Tokyo Headquarters AIST Tokyo Waterfront old AIST 15 Research Institutes 8 Institutes in Tsukuba 7 Institutes in other regions new AIST Merged into 1 Institute 9 Research Basis Research Bases of AIST In 2001

9. 9 Information Technology, Electronics 16% Nanotechnology, Materials & Manufacturing 23% Environment & Energy Geological Survey & Applied Geoscience 11% Life Science & Technology 18% Metrology & Measurement Technology 15% (as of April 1, 2006 ） Total number of employees3,209 Researchers2,505 Administrative staffs 704 Executives etc 11 Total number of employees3,209 Researchers2,505 Administrative staffs 704 Executives etc 11 17% Statistics Number of Visiting Researchers at AIST (in 2005 FY) ・ Postdoctoral researchers 650 ・ From private companies 800 ・ From universities2,000 ・ From corporation etc 950 ・ From overseas 850 Number of Visiting Researchers at AIST (in 2005 FY) ・ Postdoctoral researchers 650 ・ From private companies 800 ・ From universities2,000 ・ From corporation etc 950 ・ From overseas 850 AIST: Staff AIST: Staff

10. 10 AIST defines Type-II Basic Research, which integrates multiple disciplines and creates methods for the use of integrated knowledge; traditional basic research is defined as Type-I Basic Research, which is the pursuit and discovery of novel laws, principles and theories that govern natural phenomena. AIST places its highest priority on the pursuit of complete research, “Full Research”, ranging from Type-I Basic Research to the development of “products” by conducting intensive Type-II Basic Research. Full Research

11. 11 （ as of July 1, 2006 ） Characterization of Research Units 1. Research Center (29) -conducts pioneering & strategic projects within a certain period (<7 years) -priority in investing research resources is given -operated by top-down management 2. Research Institute (21) -conducts mid & long term research -research themes are proposed in a bottom-up manner 3. Research Initiative (7) -promotes formation of new research fields and/or research center -contributes to sudden administrative needs

12. 12 CONTENTS 1.Strategic Research Evaluation 2.Missions and Activities of AIST 3.Strategy and Evaluation 4.Evaluation System from the Outcome Point of View 5.PDCA cycle in the Research Institute

13. 13 AIST Goals and Related Research Fields Future Visions and Research Field

14. 14 AIST Research Strategy - Directional Vision toward Innovation - Overall Strategy – –Basic Principle: To realize sustainable society – –Four missions – –Overall Goals (5 outcomes and 2 practices) – –Policies to be specified Strategy in each research field – –Strategic goal – –Strategic research subject – –Focal research subject Strategy with integrating different research fields Innovation Hub Strategy → Definite description as our long-term guideline of research Concrete research goals and plans ・・・ ”2nd Mid-term Plan” （ ”Contract” between AIST and Minister in Charge) Research Strategy

15. 15 Purpose:  Activation & increasing in efficiency for research unit  Reference for top-management of AIST  Accountability to people for activities of AIST Point of View: Promotion of Promotion of 1) evaluation from the qualitative viewpoint of research 1) evaluation from the qualitative viewpoint of research 2) evaluationfrom the viewpoint of “full research” and 2) evaluation from the viewpoint of “full research” and outcomes related to AIST strategy outcomes related to AIST strategy Purpose and Viewpoint Purpose and Viewpoint of Research Unit Evaluation in AIST

16. 16 Evaluation Dept Report Administrative Unit Evaluation （ Every other year ） Research Unit Evaluation METI (AIST) Evaluation Output Evaluation （ Every other year ） Start-up Evaluation （ The Ist year ） Research Units President Administrative Units Evaluation Five-year target Five-year Plan Annual Report Evaluation of Activity Evaluation Committee for IAI in METI Evaluation of Units Resource allocation, Reorganization Report

17. 17 A new Research Center Final Review Start-up eval. Evaluation Interval Advise for Roadmap Research Center -top down -time limit Research Institute -bottom up -no time limit Research Initiative -to be Research Center -time limit Review & reorganization of research unit Output eval. 2nd term’s start-up eval. Intermediate Review Output eval. Intermediate Review

18. 18 CONTENTS 1.Strategic Research Evaluation 2.Missions and Activities of AIST 3.Strategy and Evaluation 4.Evaluation System from the Outcome Point of View 5.PDCA cycle in the Research Institute

19. 19 InputInputOutputOutput OutcomeOutcomeImpactImpact ・ Budget ・ Human resource ・ Equipment ・ Academic paper ・ Patent ・ Geologic Map ・ Standard ・ Software ・ Process Develop. ・ Database ・ Popularization of products of products ・ New research field ・ Service ・ Saving energy ・ Risk management ・ Disaster prevention prevention ・ GDP growth ・ New employment ・ International competition ability competition ability ・ High QoL ・ Environmental protection protection ・ Security & safety society society Definition of Output, Outcome in AIST Output: Direct results in R&D (papers, patents, etc) Outcome: Social and economical effects caused by output (products, standards, novel research fields, etc)

20. 20 ① Evaluation of Realized Outcomes ② Evaluation of Scenario/Roadmap toward Realization of Outcomes ③ Evaluation of Outputs to contribute to Outcomes ④ Evaluation of Management to contribute to Outcomes ① Evaluation of Realized Outcomes ② Evaluation of Scenario/Roadmap toward Realization of Outcomes ③ Evaluation of Outputs to contribute to Outcomes ④ Evaluation of Management to contribute to Outcomes Outputs in the past Expected Outcomes in the Future Present Research Past Future ① Realized Outcomes ② Roadmaps ③ Outputs ④ Management Performance Evaluation: ② Roadmap, ③ Output, and ④ Management Startup Evaluation: ② Roadmap and ④ Management （ by Research Evaluation Committee in AIST ） Evaluation from the Outcome Point of View

21. 21 Example of Roadmap for Outcomes Example of Roadmap for Outcomes: Ultra-Low-Loss power devices Subjects (Targets) （ Power Electronics Research Center ） 2002 2004 2006 2008 2010 2012 2020 Substrate / Wafer Epitaxy Processing Packaging FET Devices Modules Applications InstrumentsInstruments  Few 10μm/h 3 inch SiC Wafer Lamp Ballasts Factory Motor drive Factory Motor drive Electric Power Distribution System Dispersed Power supply Dispersed Power supply Power Transmission System Power Transmission System General Inverter （ＦＥＴ） （ＩＧＢ Ｔ） Technology transfer 600V10A 600V100A 2.5kV100A Low-EMI Bipolar Devices The 1st Term The 2nd Term R&D in AIST low-Ron 4 inch Wafer 6 inch Wafer Air-Con. Multi-Wafer-Epi. Self-Alignment Bipolar-Tech. High-reliable SiO2 Low-Temp Process T j >250 ℃ Wire-less Heatproof Capacitor-less

22. 22 Report External Reviewers: Roadmap, Output in each main research theme External Reviewers: Roadmap, Output in each main research theme ＊ Evaluation of Roadmaps ＊ Evaluation of Outputs ＊ Evaluation of Management ＊ Evaluation of Roadmaps ＊ Evaluation of Outputs ＊ Evaluation of Management Report, Presentation Report, Presentation ＊ Roadmap ＊ Roadmap ＊ Output ＊ Management ＊ Management Report, Presentation Report, Presentation ＊ Roadmap ＊ Roadmap ＊ Output ＊ Management ＊ Management Evaluation : Rating Evaluation : Rating with Comments with Comments Evaluation : Rating Evaluation : Rating with Comments with Comments AIST Strategy President of AIST Research Unit Internal Reviewers: Management, Roadmap of main research themes main research themes Internal Reviewers: Management, Roadmap of main research themes main research themes People Evaluation Dept OUTPUT EVALUATION Output Evaluation Open Comments Reflection on Resource Allocation, Reorganization

23. 23 Evaluation : validity of research plan from the qualitative viewpoint in research target Object ： Roadmap of whole research theme and each main theme ＊ Roadmap of whole research theme and each main theme ＊ Item and its point ： outcome, milestone toward outcome, core technique of research unit, technical elements, benchmark outcome, milestone toward outcome, core technique of research unit, technical elements, benchmark Assignment ： External reviewers ： Roadmap of each main theme （ Rating with comments ） External reviewers ： Roadmap of each main theme （ Rating with comments ） Internal reviewers ： Roadmap of main themes （ Rating with comments ） Internal reviewers ： Roadmap of main themes （ Rating with comments ） Evaluation : validity of research plan from the qualitative viewpoint in research target Object ： Roadmap of whole research theme and each main theme ＊ Roadmap of whole research theme and each main theme ＊ Item and its point ： outcome, milestone toward outcome, core technique of research unit, technical elements, benchmark outcome, milestone toward outcome, core technique of research unit, technical elements, benchmark Assignment ： External reviewers ： Roadmap of each main theme （ Rating with comments ） External reviewers ： Roadmap of each main theme （ Rating with comments ） Internal reviewers ： Roadmap of main themes （ Rating with comments ） Internal reviewers ： Roadmap of main themes （ Rating with comments ） ＊ Main Theme ： Research theme in 5-years plan ＊ Challenging Theme ： Research theme that is not in 5-years plan but is recognized to be important by research unit ＊ Main Theme ： Research theme in 5-years plan ＊ Challenging Theme ： Research theme that is not in 5-years plan but is recognized to be important by research unit Evaluation of Roadmap

24. 24 Evaluation : whether outputs in the evaluation period are contributed to the outcome or not Object ： Academic paper, patent, start-up, prototypal product, intellectual right, prize, commendation Academic paper, patent, start-up, prototypal product, intellectual right, prize, commendation Criterion ： Target recorded in milestone, World top level, Target recorded in milestone, World top level, Government preparation plan Government preparation plan Assignment ： External reviewers ： External reviewers ： Rating with comments to each main research theme Rating with comments to each main research theme Rating with comments to research themes Rating with comments to research themes Internal reviewers ： Internal reviewers ： Comments to each research theme Comments to each research theme Rating with comments to main research themes Rating with comments to main research themes Evaluation : whether outputs in the evaluation period are contributed to the outcome or not Object ： Academic paper, patent, start-up, prototypal product, intellectual right, prize, commendation Academic paper, patent, start-up, prototypal product, intellectual right, prize, commendation Criterion ： Target recorded in milestone, World top level, Target recorded in milestone, World top level, Government preparation plan Government preparation plan Assignment ： External reviewers ： External reviewers ： Rating with comments to each main research theme Rating with comments to each main research theme Rating with comments to research themes Rating with comments to research themes Internal reviewers ： Internal reviewers ： Comments to each research theme Comments to each research theme Rating with comments to main research themes Rating with comments to main research themes Evaluation of Output

25. 25 Evaluation : “full research” and its management from the viewpoint of realizing outcome Object: Policy statement and actual actions for bridging toward innovation Policy statement and actual actions for bridging toward innovation Items ： 1. “Full research” 1. “Full research” 2. Innovation ： Technology transfer of intellectual right, Standard, R&D consortium, 2. Innovation ： Technology transfer of intellectual right, Standard, R&D consortium, Research collaboration, Government project, etc. Research collaboration, Government project, etc. 3. Training of human resources: Training staffs, 3. Training of human resources: Training staffs, Training of young researchers,etc Training of young researchers,etc 4. Research unit management 4. Research unit management ・ Total management ： Leadership of the unit leader, Common mission, etc ・ Total management ： Leadership of the unit leader, Common mission, etc ・ Budget management ： Budget management, Budget management from the ・ Budget management ： Budget management, Budget management from the viewpoint of cost vs effect, Taking of external funds, etc viewpoint of cost vs effect, Taking of external funds, etc ・ Risk management: Risk management (Risk about disaster, accident 、 ethics) ・ Risk management: Risk management (Risk about disaster, accident 、 ethics) Assignment ： Assignment ： External reviewers ： Comments External reviewers ： Comments Internal reviewers ： Rating with comments Internal reviewers ： Rating with comments Evaluation : “full research” and its management from the viewpoint of realizing outcome Object: Policy statement and actual actions for bridging toward innovation Policy statement and actual actions for bridging toward innovation Items ： 1. “Full research” 1. “Full research” 2. Innovation ： Technology transfer of intellectual right, Standard, R&D consortium, 2. Innovation ： Technology transfer of intellectual right, Standard, R&D consortium, Research collaboration, Government project, etc. Research collaboration, Government project, etc. 3. Training of human resources: Training staffs, 3. Training of human resources: Training staffs, Training of young researchers,etc Training of young researchers,etc 4. Research unit management 4. Research unit management ・ Total management ： Leadership of the unit leader, Common mission, etc ・ Total management ： Leadership of the unit leader, Common mission, etc ・ Budget management ： Budget management, Budget management from the ・ Budget management ： Budget management, Budget management from the viewpoint of cost vs effect, Taking of external funds, etc viewpoint of cost vs effect, Taking of external funds, etc ・ Risk management: Risk management (Risk about disaster, accident 、 ethics) ・ Risk management: Risk management (Risk about disaster, accident 、 ethics) Assignment ： Assignment ： External reviewers ： Comments External reviewers ： Comments Internal reviewers ： Rating with comments Internal reviewers ： Rating with comments Evaluation of Management

26. 26 CONTENTS 1.Strategic Research Evaluation 2.Missions and Activities of AIST 3.Strategy and Evaluation 4.Evaluation System from the Outcome Point of View 5.PDCA cycle in the Research Institute

27. 27 Strategy of AIST Research Units external reviewers 1) Roadmap Evaluation 1) Roadmap Evaluation 2) Output Evaluation 2) Output Evaluation 3) Management Evaluation 3) Management Evaluation 1) Roadmap Evaluation 1) Roadmap Evaluation 2) Output Evaluation 2) Output Evaluation 3) Management Evaluation 3) Management Evaluation EvaluationDepartmentEvaluationDepartment internal reviewers Peopl e Evaluation Committee for IAI in METI Evaluation Committee for IAI in METI Report Resources Allocation Reorganization Evaluation of Each Unit’s Results Plan Do President Evaluation Committees ＊ Roadmap for outcomes ＊ Output indexes ＊ Management Evaluation: Rating with comments Results of Evaluation PDCA Cycle Principal reviewers’ report Action Check

28. 28 1.Importance of strategy 2.Significance of ex-ante evaluation (Appraisal) 3.Evaluation from the outcome point of view 4.Reflection of evaluation and the design of new strategy 5.Strategic evaluation linkage 1.Importance of strategy 2.Significance of ex-ante evaluation (Appraisal) 3.Evaluation from the outcome point of view 4.Reflection of evaluation and the design of new strategy 5.Strategic evaluation linkage Novel System of Strategic Research Evaluation Significance of Strategic Research Evaluation Strategic Research Evaluation

29. 29 Strategic Research Evaluation Appendix

30. 30 Budget of AIST Subsidy 66,437 Commissioned research fund 22,486 Miscellaneous 3,851 Direct research cost 44,626 Facilities management cost 5,800 Indirect research cost 13,146 Employment cost 35,002 Expenditure 98,574 Revenue 98,574 Facilities management grants 5,800 Statistics ( unit: million yen ) (as of April 1, 2006 ）

31. 31 Transfer Startups New Industry New Market Market ShareSample Novel Idea Output Research in AIST Pattern of Outcome Creation (Product Type) Acceptance in Market & Society Market Outcome Solar Panel Liquid Crystal Display Collaboration/ Technology Transfer (Prototype ・ Improvement ）

32. 32 Rating 2.Calculation method ・ Arithmetic mean of roadmap & output of each main research theme. ・ The score is calculated by applying weight of each main research theme. 2.Calculation method ・ Arithmetic mean of roadmap & output of each main research theme. ・ The score is calculated by applying weight of each main research theme. 3.Weighting for Final Score Main theme:6 Whole theme:1 Management:3 3.Weighting for Final Score Main theme:6 Whole theme:1 Management:3 1.Performance rating PerformanceRating AA ： Extremely Effective5 Ａ： Highly Effective 4 Ｂ： Effective 3 Ｃ： Needs Improvement 2 Ｄ： Unsatisfactory1 1.Performance rating PerformanceRating AA ： Extremely Effective5 Ａ： Highly Effective 4 Ｂ： Effective 3 Ｃ： Needs Improvement 2 Ｄ： Unsatisfactory1 Roadmap eval. Output eval. Management eval. For each theme For whole theme 6 1 3

33. 33 History of Research Units of AIST History of Research Units of AIST (as of 31 Mar., 2006 ） Research Centers Research Institutes and Initiatives 20022001 2005 20042003 20022001 2005 20042003

34. 34 ・ CSTP has been promoting R&D evaluation as an integral part of S&T policy. ・ Ministries and R&D Organizations carry out the major part of evaluation activities. METI :Ministry of Economy, Trade and Industry MEXT ： Ministry of Education, Culture, Sports, Science and Technology JST Guideline MEXT Guideline AIST Guideline NEDO Guideline METI Guideline Other Ministries National Guideline for R&D Evaluation Decided by the Prime Minister S&T Basic Plan Cabinet decision National Level (CSTP) Ministry Level Funding agency Level Structure of R&D Evaluation in Japan R&D Institute Level RIKEN Guideline R&D Evaluation in Japan

35. 35 Minister of Education, Culture, Sports, Science & Technology Minister of Finance Minister of Public Management, Home Affairs, Post & Telecommunication 15 Regular members of the council Minister Temporary members ＜ Academic S&T ＞ ＜ Industrial S&T ＞ ＜ Finance ＞＜ Policy Evaluation ＞ Minister of Economy, Trade & Industry Council for Science and Technology Policy (Chaired by Prime Minister) Chief Cabinet Secretary Minister of State for S&T policy Chairman of the Science Council 8 members from academia & business New Science and Technology Administration System in Japan The Council for Science and Technology Policy (CSTP) was established in January 2002, as a headquarter for national S&T policy. Universities Research Institutes Industries

36. 36 R&D Budget of Japanese Ministries (including expenses for labors and facilities, 2005) MEXT 65% METI 17% JDA 4% MHLW 4% MAFF 3% MLIT 2% others 3% MIC 2% 36 Billions US$