1. Purposes of Follow-Up Monitoring & Evaluation Evaluation not for measurement of project’s goals, for accountability and improvement of organization What’s Follow-up Monitoring & Evaluation? We would like to find the contribution/impacts of projects to society for ACCOUNTABILITY, and seek out points that should be improved in our PROJECT MANAGEMENT, and also seek out needs of follow-up/new projects for POLICY PLANNING! What’s the subject of it? All R&D projects are subjects, except for international cooperation /dissemination of new energy projects!

2. American Evaluation Association Conference October 28, 2005 Hiroshi Sano, sanohrs@nedo.go.jp Shuji Yumitori, yumitorisji@nedo.go.jp Kazuaki Kohmoto, Kohmotokza@nedo.go.jp New Energy and Industrial Technology Development Organization (NEDO) http://www.nedo.go.jp/english/index.html Follow-Up Monitoring & Evaluation Systems for National R&D at NEDO Part II ～ Feedback for Management ～

3. ☞ Purpose ☞ A Need for the New Tool (Follow-up Chart) ☞ Case Studies ☞ Feedback for Management ☞ Conclusions Contents

4. Objective Derive concrete implications to improve R&D management methods. Follow major events throughout the project and understand key points on management or the reason why certain events happened through interviews.

5. Statistical data didn’t help to derive concrete implications. For example, it would be too bold to conclude from the following graph that we should exclude companies whose objective for participation were improvement of their technical level. A Need for New Tool Understanding when certain events happened or the magnitude of each event in detail solely by interviews was particularly difficult. Matched a purpose of the organization Challenge a new technical target Invitation of other organization Matched a purpose of the organization Challenge a new technical target Invitation of other organization Improvement of technology Create network Personnel training Other Personnel training Other Practical application & commercialization (n=78) No post-project activity (n=246) ※ Multiple answers allowed Objective for participation

6. Follow-up Chart Methodologies ☞ Conduct a interview to a company about major events on the project and influences of management. ☞ Plot the above information in a conceptual chart. ・ The horizontal axis is the time from the beginning to present. ・ The vertical axis is possibilities or expectation for practical use of technology. (The vertical axis is interviewee’s personal opinion because to measure the possibilities was difficult.) ☞ Determine the form of the chart through the interview. ☞ Understand the positive/negative factors which would affect the possibility of practical use of the R&D results through the interview.

7. Time (-) Large Possibilities/expectation for practical use Follow-up Chart (Image) ～ Status from the beginning ～ Plan- ning Present 1st year Mid. term Evaluation Mid. Small 2st year 3rd year 4th year 5th year End term Evaluation Follow the events at the turning points and influ- ence/effect of management for practical use. : Turning point ◆ It is difficult to find a fixed quantity of possibilities. The vertical axis is based on interviewee’s opinion.

8. Case Studies Large Plan- ning Present 1 st year 2 nd year 3 rd year Final year ④ Project goals were high. A list of targets was created to demonstrate what needed to be accomplished. Although there was hope to collect necessary materials, odds for success seemed low due to legal issues. ☆ Factors that raised the possibilities of practical use. Red ： by NEDO Italic ： by except for NEDO ① Development of bold approach. (Relation to ⑤ ) ② Large scale demonstration test. ③ Influential figure. ④ High level targets. ⑤ Old approach was examined thoroughly (Relation to ① ) ⑥ Government spurred industry. ⑦ Key person who was integrate the R&D/business unit. ① A bold technical approach was developed. (Relation to ⑤ ) ③There was an influential figure in NEDO. This particular individual persuaded the technical committee and the government to adapt to the new technical approach. ②The trial productions were tested by real users, which led to the grater possibilities for commercialization. This is due to a large scale demonstration test that earned user's confidence. ⑥ Under the leadership of METI, the demand promotion committee was established, which helped spur the industry. ⑦ The R&D went smoothly because there was a liaison personnel who was able to integrate the R&D and business unit. ● It became clear that the method did not go well technically. In addition, the law did not allow the use of materials that they were counting on. ⑤The old technical approach was examined thoroughly for six months. This helped achieve the development of the prominent new approach.. （ Relation to ①） Possibilities/expectation for practical use Mid. Small Time Case Study A ● Demand has reached its critical level. Measures under government leadership are desirable.

9. Time Planning Present 1 st year2 nd year3 rd year4 th yearFinal year End term evaluation Various trial & error testing methods were available. Efficiency and performance increased threefold. Appropriate evaluation based on the results. Budget allocation based on mid-term evaluation. Collaboration scheme went well in the basic R&D phase. Collaboration between basic science, electronic board and device group was good. Setting goals emphasized on “ The World ’ s first ” and “ The World ’ s best ” to attempt to be the World ’ s #1. Solid relationship with universities. Universities acquired the same equipments as companies which fostered data analysis and evaluation. Modification of company organization to identify business opportunities. Company executives attended conferences to present project results, which helped them to create a stronger network with other attendees. This helped win support for the project within the company. Med. Term Evaluation Attainment of “ World ’ s best ” data. Collaboration scheme is not always beneficial. Well organized collaboration system in the basic R&D phase became an obstacle in the practical R&D phase. Emergence of technical problem for practical use. Technology for stable/mass production, cost reduction, patent problems etc. From research phase to practical phase. Well planed collaboration scheme (Producer and Consumer, etc.) Important information such as spec or permission range of quality was obtained. Enhanced business opportunity by human network. Capitalization on human network acquired from the project. Positive factor Negative factor Color Indication Human network Environment was favorable for a company to enter into a new tech. field. Collaboration with universities and/or user companies was useful. Possibilities/expectation for practical use Large Mid. Small Case Study B

10. Consideration ☞ Positive/negative effects and the magni- tude of events. ☞ Change of personal opinion concerning possibilities for practical use. ☞ Status of R&D, business and social environment. Follow-up chart shows on a simple peace of paper: ◆ (Attention) - Selection of the items or magnitude of the events that were recorded on the chart is influenced by interviewee’s personal opinions. - Personal opinions determine the shape of the chart.

11. Feedback for Management (Image) Budget Request Check List Concrete Implications ・ Does the project earn user’s confidence? (from case study A) ・ Do you have backup schemes for situations where a company ente- ring a new business/tech. field joins the project? (from case study B) Goal Setting Check List Mid. term Evaluation Member Setting Check List Check List G u i d e l i n e

12. Conclusions Follow-up chart is a useful tool for improving management. Follow-up chart is based on personal opinion. (Further consideration will be needed.) ☞ Follow major events throughout the project and understand key points on management or the reason why certain events happened by hearing. ☞ Derive concrete implications on project management. OK

13. Finally I have introduced how to feedback of follow-up evaluation for management. Today, I would like to know about your activity about feedback for management. I would be happy if I could acquire some suggestions about the vertical axis. Thank you for your attention.