Meeting an Evaluation Challenge: Identifying and Overcoming Methodological Problems Joint CEA/AEA Evaluation 2005 “Crossing Borders, Crossing Boundaries” RTD TIG, Think Tank Session Toronto, Canada October 26, 2005 Rosalie T. Ruegg Managing Director TIA Consulting, Inc. Connie K.N. Chang Supervisory Economist Advanced Technology Program, NIST U.S. Department of Commerce

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 2 The 3 rd in a series of Think Tanks on barriers to evaluation 2003: Identification of 6 Categories of Barriers to Evaluation 2004: Focus on Institution and Cultural Barriers—Feedback Loops 2005: Focus on Methodological Barriers

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 3 Review of ‘03 Think Tank Main Finding Striking commonality of evaluation barriers among programs and across countries

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 4 Review of ‘03 Think Tank, cont’d Six categories of barriers identified 1. Institutional/cultural 2. Methodological 3. Resources 4. Communications 5. Measurement/data 6. Conflicting stakeholder agendas

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 5 Review of ‘03 Think Tank, cont’d Barriers were said to impede:  Demand for evaluation  Planning and conducting evaluation  Understanding of evaluation studies  Acceptance and interpretation of findings  Use of results to inform  program management  budgetary decisions  public policy

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang Think Tank Six Categories of Barriers Identified 1. Institutional/cultural 2. Methodological – today’s focus 3. Resources 4. Communications 5. Measurement/data 6. Conflicting stakeholder agendas

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 7 Methodological issues identified in 2003  Problems in measuring the value of knowledge creation  Lack of standardization leading to comparability problems  Inability to replicate studies  Difficulties in apportioning benefits to complex contributing factors

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 8 Methodological issues identified in 2003, cont’d  Difficulty in defining success against multiple program goals  Selecting the appropriate method for a given purpose  Reliability and acceptance of new methods  Adherence of studies to best practices

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 9 Additional methodological issues? Potential new issues to consider:  Problems in measuring commercialization and its economic value  What are best practices in methodology?  What constitutes market failure or market imperfection?  Question of additionality effect of public funding  What is the state-of-the-art of evaluating R&D portfolios and collections of R&D portfolios?

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 10 Methodological issues identified – today’s focus  Problems in measuring the value of knowledge creation  Lack of standardization leading to comparability problems  Inability to replicate studies  Apportioning benefits to complex contributing factors  Defining success against multiple program goals  Selecting the appropriate methods for a given use  Reliability and acceptance of new methods  Adherence of studies to best practices

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 11 Measuring value of knowledge creation How do we define value?  By quality or significance of knowledge created?  By economic value of the knowledge created?

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 12 Measuring value of knowledge creation, cont’d Methodology depends on the definition of value: For value in terms of quality, use bibliometrics (counts—with adjustment for quality variation among journals, citation analysis, content analysis); research value mapping; network analysis; third-party recognition through awards for scientific merit For value in terms of economics, follow multiple paths of knowledge flow to downstream users, using indicator metrics, historical tracing supported by citation analysis, then, benefit-cost analysis with “snow-ball” technique

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 13 Lack of standardization leading to comparability problems Standardization to whose standards?  By program practitioner? By research field? By country? Standardization of what methodological terms?  Performance indicators? Rate of return on investment? Net present values? Social vs. public rate of return? Underlying Assumptions? Other? How do we define comparability?  Across studies? Within a program? Across programs? Across national borders?

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 14 Lack of standardization leading to comparability problems, cont’d Some ways to promote standardization for improved comparability: Providing a standard format and guidelines for similar studies (e.g., NAS/NRC matrix and guidance for evaluating DOE EE & FE R&D projects) Commissioning clusters of studies to be performed within a common framework (e.g., ATP’s cluster studies of component-based software projects, tissue engineering projects, and composite manufacturing technologies) Other?

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 15 Ability to replicate studies What do we mean by ability to replicate?  Simple meaning: Like scientists, allow others to run the experiment using the same data to see if the same results are obtained  Another meaning: Re-do studies, testing for robustness by varying key data and assumptions in sensitivity analysis, and possibly refining the approach  Yet another meaning: Confirm prospective benefit estimates with later retrospective cost-benefit studies

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 16 Ability to replicate studies, cont’d Key is to avoid “black box” Transparency and clarity of approach are essential

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 17 Summary Six Categories of Barriers Identified Institutional/cultural Methodological Resources 4. Communications 5. Measurement/data 6. Conflicting stakeholder agendas

2005 CES/AEA Evaluation Conference Think Tank Ruegg and Chang 18 Contact information Rosalie T. Ruegg Managing Director TIA Consulting, Inc. Connie K.N. Chang Supervisory Economist Advanced Technology Program