1. New Direction of National R&D Evaluation System in Korea 2011. 11. 3 Changwhan Ma Director of Performance Policy Division EVALUATION 2011 @ Anaheim, CA, U.S.A.

2. Contents 1.Introduction 2.Background 3.Initiative Focus 4.Conclusion Contents 13 slides 1 3 5 13

3. Contents 1.Introduction 2.Background 3.Initiative Focus 4.Conclusion 1 3 5 13

4. R&D Program Management Process R&D Budget Survey/Analysis Evaluation Implementation (Programs/Projects) In-depth Recommendation Feedback Evaluation strategy & Data collection Self → Meta Evaluation process Utilization process Feasibility Study (Ex-ante) Program (Planning) Policy (Planning) 1

5. National R&D Evaluation System in Korea 2 (Guidelines) Self Eval. Results (Grades, Recommendations) Meta Eval Program Evaluation GRI Evaluation Ex-post In-depth Eval Ex-post/post Follow-up Survey Follow-up Eval Management & Research Self Eval.Meta Eval MinistriesNSTC/(KISTEP) Feedback Report to NSTC Efficiency/Effectiveness Of R&D Investment ↑ Preliminary Study Feasibility Study Feasibility Study Results (Budget Size, Recommendation) Ex-ante Program Planning Ex-post NSTC/MOSF/(KISTEP) NSTC *NSTC: National Science & Technology Commission

6. Contents 1.Introduction 2.Background 3.Initiative Focus 4.Conclusion 1 3 5 13

7. Background of New Direction 3  Continuous increase of R&D investment  3 rd on R&D investment / GDP (3.74%, 2010), 14.89 billion USD in 2011  Quantitative Efficiency of R&D Performance has been continuously increased  (Publication and Patent) Near top 10 in the world  Qualitative Excellence is stalled around 30 th in the world  S&T and Socioeconomic Competitiveness depend on the qualitative excellence of researchers and their performance  Therefore, evaluation system should meet the direction of S&T advancement  And new NSTC has recently been launched to respond to the recent change.

8. New NSTC 4 8 ce 삭제 Ministries, GRIs etc. Planning NSTC MoSF Suggestion about direction of Budget allocation Allocation · Coordination · Formation Self evaluation Meta & Specific evaluation After Before Constructing overall R&D System MoSF Budget Formation Self evaluation Ministries MoSF NSTC NSTC Meta & Specific evaluation Suggestion about direction of budget allocation & Budget allocation / coordination of main National R&D program NSTC (Multi-departmental R&D program, etc.)

9. Contents 1.Introduction 2.Background 3.Initiative Focus 4.Conclusion Contents 1 3 5 13

10. 1. Open Evaluation System 10 ① More Openness in Performance Information & Expert Participation To ensure variety of evaluation perspectives by increasing Openness of Performance Information and Various Opinions to the information To promote expert review activity by establishing comprehensive expert community pool To introduce on specific evaluation first and escalated to other evaluation types To ensure variety of evaluation perspectives by increasing Openness of Performance Information and Various Opinions to the information To promote expert review activity by establishing comprehensive expert community pool To introduce on specific evaluation first and escalated to other evaluation types ② Establishing Online Open Evaluation System To share and communicate more by establishing user-friendly online open evaluation system (OOES) To develop model and manual for OOES To share and communicate more by establishing user-friendly online open evaluation system (OOES) To develop model and manual for OOES 5

11. 2. Mission-oriented GRI Evaluation System 11 ① Introducing Absolute Rating System To introduce absolute rating system considering characteristics of each GRI (government-supported research institutes) To expand the autonomy of GRI during evaluation process To focus on accomplishment of GRI mission To encourage to set challengeable objective to reach world class technology level To introduce absolute rating system considering characteristics of each GRI (government-supported research institutes) To expand the autonomy of GRI during evaluation process To focus on accomplishment of GRI mission To encourage to set challengeable objective to reach world class technology level ② Alleviating Burden on Evaluation To reduce the portion of management evaluation gradually and integrated into performance evaluation (depends on revision of act on R&D performance evaluation) To reduce the portion of management evaluation gradually and integrated into performance evaluation (depends on revision of act on R&D performance evaluation) 6

12. 3. Rationality and Autonomy of Self Evaluation 12 ① Stringent Pre-check of Performance Plan To ensure the quality of self evaluation by pre-check the appropriateness of performance objective and indicator To operate official committee on reviewing performance indicator To ensure the quality of self evaluation by pre-check the appropriateness of performance objective and indicator To operate official committee on reviewing performance indicator ② More Autonomy on Self Evaluation To set up the evaluation schedule (or cycle) according to milestones of each programs by budget-spending ministry To carry out meta evaluation in general and re-evaluation on specific programs To set up the evaluation schedule (or cycle) according to milestones of each programs by budget-spending ministry To carry out meta evaluation in general and re-evaluation on specific programs 7

13. 4. Evaluation of Qualitative Excellence 13 ① Qualitative Evaluation considering Impact To carry out comprehensive evaluation considering scientific/technological and socioeconomic impact and sustainability To encourage to set challengeable performance objective by using qualitative performance indicator To carry out comprehensive evaluation considering scientific/technological and socioeconomic impact and sustainability To encourage to set challengeable performance objective by using qualitative performance indicator ② Guideline on R&D Performance Evaluation To develop and disseminate R&D performance evaluation guideline for better understanding of qualitative evaluation and for strengthening evaluation capacity of ministries 8

14. 5. R&D Policy Evaluation and Cross-cutting Review 14 ① Evaluation of R&D Policy and Issue To introduce R&D policy evaluation for better coordination of R&D programs and R&D environment ② Cross-cutting Review To expand the scope of specific evaluation from individual program to fields of technology and similar program groups To ensure optimize and coordinate the delivery system the similar, overlapping programs To expand the scope of specific evaluation from individual program to fields of technology and similar program groups To ensure optimize and coordinate the delivery system the similar, overlapping programs 9

15. 6. Strategic Performance Budgeting & Program Improvement 15 ① Strategic Performance Budgeting To set priority and allocate budget of the similar programs according to evaluation program selection and results To carry out integrated performance budgeting by reviewing mid/long-term direction of R&D investment To keep maintaining performance budgeting with the evaluation results To set priority and allocate budget of the similar programs according to evaluation program selection and results To carry out integrated performance budgeting by reviewing mid/long-term direction of R&D investment To keep maintaining performance budgeting with the evaluation results ② Monitoring System of Performance Information To establish the DB for evaluation history of the program including evaluation results, recommendation and management action plan, etc. To ensure the continuous monitoring evaluation feedback efficiently To establish the DB for evaluation history of the program including evaluation results, recommendation and management action plan, etc. To ensure the continuous monitoring evaluation feedback efficiently 10

16. 7. Customized Evaluation 16 ① Evaluation according to Program Type To apply differentiated evaluation perspective according to types, size of the program To develop evaluation model and checklists with common or specialized indicators To apply differentiated evaluation perspective according to types, size of the program To develop evaluation model and checklists with common or specialized indicators ② Milestone-based Performance Evaluation To select the program after considering the cycle or time for producing key performance (creative period; 3 yr, 5 yr, exempted, etc.) To encourage to carry out objective and comprehensive performance analysis for self monitoring To select the program after considering the cycle or time for producing key performance (creative period; 3 yr, 5 yr, exempted, etc.) To encourage to carry out objective and comprehensive performance analysis for self monitoring 11

17. 8. Infrastructure and Network 17 ① Infrastructure of R&D Performance Evaluation To establish DB for major performance and budget information as a program management platform To serve as an active channel for the exchange of information on evaluation trend and various opinions To carry out R&D performance evaluation capacity building To establish DB for major performance and budget information as a program management platform To serve as an active channel for the exchange of information on evaluation trend and various opinions To carry out R&D performance evaluation capacity building ② Network of R&D Evaluation To hold the evaluation forum regularly (global R&D evaluation network) To carry out collaborative research on evaluation To participate the conference on evaluation regularly To hold the evaluation forum regularly (global R&D evaluation network) To carry out collaborative research on evaluation To participate the conference on evaluation regularly 12

18. Contents 1.Introduction 2.Background 3.Initiative Focus 4.Conclusion 1 3 5 13

19. Future Direction 13 Professiona l Evaluation Feasible Policy Optimal Investment National R&D Program Qualitatively Excellent Researchers with Performance Strong (S&T → National) Competitiveness Open system Customized Mission …

20. Thank you!