1. TIA Consulting, Inc. Starting an Evaluation Program: Challenges and Lessons Learned Technology Program Evaluation: Methodologies from the Advanced Technology Program Collaborative Expedition Workshop #71 NSF March 18, 2008 Rosalie T. Ruegg Managing Director, TIA Consulting, Inc. ruegg@ec.rr.com

2. TIA Consulting, Inc. 2 Outline Starting an evaluation program now Starting ATP’s evaluation program 18 years ago Challenges and lessons learned

3. TIA Consulting, Inc. 3 Starting an evaluation program now Several points: Public programs are starting evaluation programs ground-up today Two examples from abroad: - Austrian genomic grant program—design of an evaluation plan - Chilean National Innovation System for Competitiveness—design of a multi-level evaluation system.

4. TIA Consulting, Inc. 4 Steps in designing an evaluation program 1. Identify purposes to be served by evaluation, e.g., To guide program management & strategy - how is the program performing? - are there performance problems? - are there problems with operational efficiency? - are adjustments needed? For accountability and public Policy - is the program doing what it was intended to do? - is it worth continued support? at the same level; at a reduced level? - to provide information for stakeholders, including public opinion - as input to broader public policy

5. TIA Consulting, Inc. 5 Steps in designing an evaluation program 2. Develop an understanding and model of the program/system to be evaluated, e.g., Review program documents—legislative, operational guidance, descriptive, analytical Interview stakeholders—legislators, program managers, clients, others Conduct studies of underlying program structure Construct descriptive program model(s) and test them with stakeholders

6. TIA Consulting, Inc. 6 Steps in designing an evaluation program 3. Develop a Program Logic Model

7. TIA Consulting, Inc. 7 Steps in designing an evaluation program 4. Formulate in detail questions that need to be answered, e.g., for different evaluation purposes, ranging from efficiency questions to ultimate success questions for internal management and for accountability for other diverse audiences, at different time periods, subject to different levels of scrutiny

8. TIA Consulting, Inc. 8 Steps in designing an evaluation program 5.Develop a plan for answering the questions, e.g., what, how, who, and when: What: what kinds of questions (e.g., descriptive, normative, impact) How: what evaluation methods are needed and what information is needed in support Who: who best to implement—internal versus external, identification of experts When: when will each question need to be addressed and when will planning need to start.

9. TIA Consulting, Inc. 9 Steps in designing an evaluation program 6. Develop a data collection plan, e.g., What data will be collected Who will collect it How will it be collected How will it be made available to evaluators How will issues of confidentiality (where they exist) be handled Etc…

10. TIA Consulting, Inc. 10 Steps in designing an evaluation program 7.Identify methods of evaluation (and practitioners) to address different kinds of questions Examples of commonly used methods: Peer review/expert judgment Monitoring, data compilation, use of indicators Bibliometrics Network analysis Historical tracing Case study method Survey method Benchmarking method Benefit-cost analysis Econometric methods

11. TIA Consulting, Inc. 11 Why multiple evaluation methods? to answer different stakeholders questions to provide alternative perspectives to provide multiple lines of evidence to meet both program management & accountability requirements

12. TIA Consulting, Inc. 12 Implementation Target audiences “on-board” Internal evaluation component in place Program monitoring and data collection approach in place External evaluation component in place Studies commissioned Coordination Communication of methods and results Feedback loops and adjustments

13. TIA Consulting, Inc. 13 Development of ATP’s Evaluation Program Conforms to steps just presented, BUT - more evolutionary - finding our way as budget, staff, and insight allowed Why evolutionary? - 18 years ago - predated GPRA and PART - no good comprehensive models--agency evaluation largely piece-meal at the time

14. TIA Consulting, Inc. 14 Development of ATP’s Evaluation Program Why did ATP’s Evaluation Program emerge as a leader in absence of GPRA and PART? Strong support by ATP’s management for internal use Existing NIST experience with evaluation Legislative requirement for ATP interim evaluation (4 yrs out) ATP’s administrative budget that allowed funding for evaluation (albeit quite small initially) Experimental nature of ATP which meant no status quo and created a climate open to evaluation Perceived need for ever more robust evaluation in face of growing political opposition to ATP Ability to attract leading evaluators to advise and participate in ATP’s evaluation program – interest in public-private partnership programs

15. TIA Consulting, Inc. 15 Some lessons learned Evaluation is necessary but not sufficient for program survival. Evaluation driven by internal management tends to be more comprehensive than that driven by accountability requirements. There are legitimate roles for both internal and external components of program evaluation. Evaluation is a field that emerged considerably over the past 18 years, but there is still much room for advancement. Other…

16. TIA Consulting, Inc. 16 Some challenges faced by evaluation Gaining better understanding of evaluation by stakeholders- -its uses and limitations Extending evaluation methods to provide more useful and reliable measures—e.g., for program portfolios Better bridging between the needs and use of evaluation for program management and for accountability, including informing public science policy Other…

17. TIA Consulting, Inc. 17 Questions/Discussion Questions/Comments/Discussions