Overview of Post-Project Evaluation in NEDO T. Isshiki, Y. Sato Evaluation Department New Energy and Industrial Technology Development Organization (NEDO) Hello, everyone. My name is Isshiki. I am in charge of Follow-up Evaluation in NEDO.

Outline NEDO’s R&D Project Evaluation System Motivation Methodology of Post-Project and Follow-up Evaluation Results -Correlation Analysis between the results of Post-project Evaluation and those of Follow-up Evaluation- 5. Summary Appendix : Our new paper on impact assessment Here is the outline of my presentation. My presentation has six parts. First, I will introduce the NEDO’s R&D Project Evaluation System. Then, I will show my motivation. After that, I will show the methodology of Post-project and Follow-up Evaluation in detail. And then, I will report the results and I will conclude with a summary. And lastly, I will introduce our new paper on impact assessment.

Outline NEDO’s R&D Project Evaluation System Motivation Methodology of Post-Project and Follow-up Evaluation Results -Correlation Analysis between the results of Post-project Evaluation and those of Follow-up Evaluation- 5. Summary Appendix : Our new paper on impact assessment First, I will introduce the NEDO’s R&D Project Evaluation System.

NEDO’s Technology Development Fields NEDO is one of the largest public R&D management organizations in Japan. NEDO covers a wide range of technology development fields from energy and environmental technologies to industrial technologies. And carries out various activities from R&D to dissemination of results. Since NEDO was established in 1980, total expenditure for R&D projects is about 30 billion dollars. FY2015 Budget : $131.9MM Total expenditure : About $30B for R&D projects since 1980

NEDO’s R&D Project Evaluation System Project Planning Pre-Project Evaluation Project Implementation R&D theme selection and brush-up Feedback Feedback Intermediate and Post-Project Evaluation In general, NEDO project is implemented for about 5 years. We have four types of Evaluations, Pre-Project Evaluation, Intermediate Evaluation, Post-Project Evaluation and follow-up monitoring and Evaluation for individual projects. Moreover, since 2009, we started to survey the impact assessment on NEDO projects. We reflects those evaluation results to Pre-Project planning and implementation. Up to now, NEDO have evaluated 343 project since 2001. Follow-up monitoring and Evaluation; Impact Assessment NEDO has evaluated 343 projects since 2001.

Past presentations in RTD TIG R&D Evaluation System Results of Post-Project Evaluation Results of Follow-up Monitoring Success Factor Analysis Impact Assessment Evaluation2004 2 1 Evaluation2005 Evaluation2006 Evaluation2007 1 (Case Study) Evaluation2008 Evaluation2009 Evaluation2010 Evaluation2011 1 (NEDO inside) Evaluation2012 Evaluation2013 Evaluation2014 3 This table shows our past presentations in RTD TIG. They are categorized into 5 categories. From 2011, we especially focused on the “Success Factor analysis” and “Impact Assessment” , what we call, “NEDO inside products”. In my presentation, I’ll report our recent results of post-project and follow-up Evaluation. any correlation ?

Outline NEDO’s R&D Project Evaluation System Motivation Methodology of Post-Project and Follow-up Evaluation Results -Correlation Analysis between the results of Post-project Evaluation and those of Follow-up Evaluation- 5. Summary Appendix : Our new paper on impact assessment

Motivation Is there any significant correlation between the results of Post-project Evaluation and those of Follow-up Evaluation ? Is there any case where the R&D activities are restarted after stopped ? Here is my motivation. First motivation is “Is there any significant correlation between the results of Post-project Evaluations and those of Follow-up Evaluations ?” In our previous studies, we tried to reveal their correlation statistically, but it didn’t work out. In my study, I have succeeded in revealing the significant correlation between both Evaluations. Second motivation is “Is there any case where the R&D activities are restarted after stopped ?” In this presentation, I focus on the first one.

Outline NEDO’s R&D Project Evaluation System Motivation Methodology of Post-Project and Follow-up Evaluation Results -Correlation Analysis between the results of Post-project Evaluation and those of Follow-up Evaluation- 5. Summary Appendix : Our new paper on impact assessment I will show the methodology of Post-project and Follow-up Evaluation in detail.

Methodology of Post-Project Evaluation - An ad-hoc subcommittee of 4-9 external evaluators for each project - Open panel discussion involving project participants - Evaluated from the following four viewpoints 1. Purpose and strategy; 2. Project management; 3. R&D achievements; 4. Prospect for practical application, commercialization and other impact*. *In the case of fundamental technology development, we remove “prospect for commercialization ” from viewpoint . Rating Method 1. Either A, B, C or D is given for each viewpoint by each evaluator. A: Excellent, B: Good, C: Acceptable, D: Not acceptable 2. An average of evaluator ratings is used as an indicator. A=3, B=2, C=1, D=0 The post-project evaluations is generally implemented immediately after project completion. An ad-hoc subcommittee of 4-9 external evaluators is set up for each project. The subcommittee holds open panel discussion involving project participants. The project is evaluated from the following four viewpoints: First one is “Purpose and strategy”; Second one is “Project management”; Third one is “R&D achievements”; Forth one is “Prospect for practical application, commercialization and other impacts”. I use this word underlined on forth viewpoint. In the evaluation work, a report is prepared by the subcommittee to include comments and proposals together with four-level ratings for each viewpoint, A: Excellent, B: Good, C: Acceptable, D: Not acceptable, from each viewpoint. After that, we convert A, B, C, D into marks 3, 2, 1, 0. Finally an average of evaluator ratings is used as an indicator for each viewpoint.

Table: An average mark of post-project evaluations (N=167) Methodology of Post-Project Evaluation Evaluation Criteria Ave. Rating by evaluators 1. Purpose and strategy 2.6 A B 2. Project management 2.1 C 3. R&D achievements 4. Prospect for practical application Example; Table: An average mark of post-project evaluations (N=167) Investigation object : 167 projects - without the case of fundamental technology development Mean value* standard deviation 1. Purpose and strategy 2.73 0.26 2. Project management 2.18 0.38 3. R&D achievements 2.34 0.36 4. Prospect for practical application 1.84 0.40 This is example. This table shows the average mark of post-project evaluations. As you see, the score of “Prospect for practical application” is lower than those of others. *Full marks =3.0

Methodology of Follow-up Evaluation - Implemented for 5 years after NEDO project ended - Questionnaire to all participants every other year - In particular, we define the post-project R&D activities as five stages as follows; Terminated Research Technology Development Practical application Commercialization 26% 17% 32% 11% 14% R&D Status after five years (N=1,252, NEDO projects completed from FY2001 to FY2008.) R&D continued On the other hand, Follow-up monitoring is implemented for 5 years after project ended. We define the post-project R&D activities as 5 stages, Terminated, Research, Technology Development, Practical application, Commercialization. In my presentation, I grouped 5 stages into three categories, terminated, R&D continued and practical application. Based on our monitoring, about 25% participants got to practical application group. Practical application

Outline NEDO’s R&D Project Evaluation System Motivation Methodology of Post-Project and Follow-up Evaluation Results -Correlation Analysis between the results of Post-project Evaluation and those of Follow-up Evaluation- 5. Summary Appendix : Our new paper on impact assessment Next, I show the results, -Correlation between the results of Post-project Evaluations and those of Follow-up Evaluations-

(1) Correlation analysis Results (1) Correlation analysis Table: Correlation table between the results of post-project evaluation and those of follow-up evaluation Investigation object : 99 projects - without the case of fundamental technology development - completion of follow-up monitoring Results of Post-Project Evaluations (immediately after project) Purpose and strategy Project manage-ment R&D achieve-ments Prospect for practical application Results of Follow-up Evaluations (5 years after project ended) Number of Participants 0.04 0.03 0.06 Percentage of practical application 0.01 0.15 0.28** R&D continued -0.14 0.00 -0.05 terminated -0.22* -0.24* -0.33** In this study, investigation object is 99 projects in which the follow-up Evaluation is completed. I prepared the mark of post project evaluation and the results of follow-up evaluation for each project. And created the correlation table shown here. As the result, positive correlation appeared just at the intersection point between the mark of not the R&D achievements but the prospect for practical application in post-project evaluation and the percentage of practical application stages. This means that the higher the mark of prospect for practical application in post-project evaluation, the higher the practical application ratio rise. Moreover, all the mark of in post-project evaluation have negative correlation with the percentage of terminated. These results indicate the validity of post-project evaluation. ＜Significance Probability＞ **P＜0.01　 *P＜0.05

(2) Comment analysis (Prospect for practical application) Results (2) Comment analysis (Prospect for practical application) Each viewpoint consists of several details. 1. Purpose and strategy (1) Validity as a NEDO project (2) Validity of objectives 3. R&D achievements (1) Success at achieving objectives (2) Significance of R&D results (3) Patent acquisition (4) Dissemination 2. Project management (1) Validity of R&D goals (2) Validity of R&D plan (3) Validity of project team formation (4) Response to change in situation 4. Prospect for practical application (1) Feasibility of practical application 　(clarification of the problem and solution) (2) Scenario for commercialization 　(Scenario, cost-effectiveness, competitiveness) (3) Ripple effect Moreover, to extract the key factor improving practical application ratio, I payed attention to the comments of evaluators. As shown here, each viewpoint in post-project evaluation consists of several details. I focused on the comments of Prospect for practical application which is related to practical application ratio, And I compared the comments of higher mark group and lower mark group.

(2) Comment analysis (Prospect for practical application) Results (2) Comment analysis (Prospect for practical application) Item （1) : Feasibility of practical application (clarification of the problem and solution) Item (2) : Scenario for commercialization (cost-effectiveness, competitiveness) Item (3) : Ripple effect (spillover for other fields, development of human resources etc.) character Item (1) Item (2) Item (3) Higher mark group (N=18) (more than 2.3) Positive 100% 67% Negative 0% 61% Lower mark group (N=15) (less than 1.4) 27% 33% 20% 87% 7% Here is the result. Higher mark group is the project whose mark of Prospect for practical application is more than 2.3, on the other hand, lower mark group, lower than 1.4. I read all the comments and categorized them into positive or negative, and then count the number of them. This table shows the percentage of positive or negative comments for each item and for each group. There are big differences on Item 1 and Item 2. “clarification of the problem and solution”, “cost-effectiveness” and “competitiveness” are the key factors for enhancing practical application ratio within 5 years after NEDO project completed.

Summary Is there any significant correlation between the results of Post-project Evaluations and those of Follow-up Evaluations ? 2. Is there any case where the R&D activities are restarted after stopped ? Yes - Positive correlation regarding the practical application between the results of post-project evaluation and those of follow-up evaluation Yes From 2014 follow-up monitoring, we started to send the questionnaire to all the companies which had stopped R&D activities after NEDO project had been completed. It became clear that about 8% of them restarted their R&D activities (N=249) and several companies got to practical application stage. Main reasons of restart are as follows; - apply for other use case (ripple effects) - drastic change of investment environment (hydrogen utilization technologies) Here is the summary. For the second motivation, I showed the brief result here. From 2014 follow-up monitoring, we started to send the questionnaire to all the companies which had stopped R&D activities after NEDO project had been completed. It became clear that about 8% of them restarted their R&D activities (N=249) and several companies got to practical application stage. Main reasons of restart are as follows; - apply for other use case (ripple effects) - drastic change of investment environment (hydrogen utilization technologies) We continued to analyze the case of restart. In progress

Outline NEDO’s R&D Project Evaluation System Motivation Methodology of Post-Project and Follow-up Evaluation Result -Correlation between the results of Post-project Evaluations and those of Follow-up Evaluations- 5. Summary Appendix : Our new paper on impact assessment And lastly, I will introduce our new paper on impact assessment.

Past presentations in RTD TIG R&D Evaluation System Results of Post-Project Evaluation Results of Follow-up Monitoring Success Factor Analysis Impact Assessment Evaluation2004 2 1 Evaluation2005 Evaluation2006 Evaluation2007 1 (Case Study) Evaluation2008 Evaluation2009 Evaluation2010 Evaluation2011 1 (NEDO inside) Evaluation2012 Evaluation2013 Evaluation2014 3

“Research on social benefits resulting from NEDO projects Appendix : Our new paper on impact assessment “Research on social benefits resulting from NEDO projects - Study of the top 70 NEDO Inside Products- ” published in Synthesiology, Vol.8. No.2, p.70-88 (2015) Impact of 70 NEDO Inside Products - Economic Impact (Cumulative and Forecasted Sales) - CO2 reduction effect etc Estimation of induced economic effects of NEDO inside products on the upstream and the downstream of its supply chain with using the inter-industry relations table in Japan. 3. Social Benefits resulting from Recycling Technology Projects As I distributed, recently our new paper was published in Synthesiology Vol.8. No.2, p.70-88 (2015). Title is “Research on social benefits resulting from NEDO projects - Study of the top 70 NEDO Inside Products- ” This paper includes mainly three topics, Impact of 70 NEDO Inside Products Economic Impact (Cumulative and Forecasted Sales) CO2 reduction effect etc Estimation of induced economic effects of NEDO inside products on the upstream and the downstream of its supply chain with using the inter-industry relations table in Japan. Social Benefits resulting from Recycling Technology Projects https://www.aist.go.jp/aist_e/research_results/publications/synthesiology_e/

Appendix : Latest results of “NEDO inside products” In FY2014, 109 items were selected as NEDO Inside Products. NEDO Budget*1 Cumulative Sales*2 After Release (up to 2011) Forecasted Sales *2 (Total for 2013～2022) 7.6 billion US$ 275 billion US$ 697 billion US$ *1 Spent on projects for which follow-up surveys were conducted. This amount is part of the 3 trillion yen that has been spent on technology development projects since the establishment of NEDO in 1980. *2 Assuming that the rate of contribution of the developed results against sales was 100%. Sales are calculated based on the questionnaires and interviews conducted with companies or the public data. So far, we selected 109 NEDO inside products with the highest sales revenues or greatest potential for social benefit. As the result of calculation, We find out our 109 products create 697 billion USD sales though we had spent 7.6 billion USD. In this study, we assume that the rate of contribution of the developed results against sales was 100%. We are facing “How should we evaluate the social benefits with accuracy and how should we consider other than economic impact, like welfare or ripple effect?” And “How should we evaluate the failures” We have a good deal of data. If you have any idea, any comment, any corroboration, please contact to us. Perpendicular Magnetic Recording (HDD) X-ray CT Equipment PV Fuel Cell

Thank you all very much for your attention isshikitsy@nedo.go.jp

+ A report prepared by Subcommittee on Post-Project Evaluation (general remarks) 1. General opinion 2. Proposals Representative opinions from evaluators ( about 10 lines in Japanese) (viewpoints) 1.Purpose and strategy 2.Project Management 3.R&D Achievements 4.Prospect for practical Application <positive comments> ･comment by each evaluator < problems and comments for improvement> ･ comment by each evaluator <other comments> (technologies) 1. --- 2. --- + Average mark (shown in p11)