1. R&D Evaluation at NIAID: Making the Most of Expert Opinion Kevin Wright AEA Annual Conference Nov 4, 2011

2. Overview NIAID at a Glance – Highlights – Organization R&D Evaluation at NIAID – Intent – Implementation – Improvements Looking Ahead – Opportunities – Threats

3. NIAID at a Glance 3

4. NIAID R&D Evaluations Level 3 Level 2 Level 1 NIAID ODDivision 1Program ASolicitedUnsolicitedProgram BDivision 2Program CProgram D

5. Initial idea Program DefinitionReviewResearch RFA/PA written, grants submitted Peer review by study section, Council Research conducted Scientific results Division retreat/ program planning Concept review by RIC, Council NIAID Program Cycle

6. Primary Conclusion of the Interagency Working Group: “Expert judgment” remains the best available decision support tool for science policy makers, but a nascent community of practice is emerging in the science policy arena that holds enormous potential to provide rigorous and quantitative decision support tools in the near future. – The Science of Science Policy: A Federal Research Roadmap, Nov 2008.

7. Mission Core Values To enhance the quality, utility, transparency and application of evaluation in support of decision making throughout NIAID. Quality – The Evaluation Section will strive to meet and exceed expectations Utility – All services and products will be developed and applied where appropriate Collaborative – The Evaluation Section will listen to and partner with its customers Innovative – The Evaluation Section will continually seek to enhance products and services NIAID R&D Evaluations

8. Evaluation Planning Template Program Type Program Objectives Indicator(s)Data SourceCollection Schedule Basic Research Fund research to fill gap Projects and Pubs IMPAC II and Pubmed Quarterly Applied Research Identify promising candidates Press Release, Pubs, Pats Web, Pubmed, USPTO Annually Clinical Research Evaluate candidates Protocols, Licenses, NDA ClinicalTrials.g ov, Web, FDA Annually Physical Capacity Reduce barriers Indirect costsIMPAC IIAnnually Intellectual Capacity Maintain pipeline of investigators NI/ESIIMPAC IIAnnually

9. 9 Analytic Tools

10. Opportunities Improving Tools and Resources Community of Knowledge Alignment of Research Priorities Increased Scrutiny of Results

11. Threats Few Success Stories Inconsistency – Rigor – Use by Decision Makers Culture – Management v. Research – Results v. Process

12. Contact Info Kevin D. Wright, MPA Deputy Chief Strategic Planning and Evaluation Branch OSPFM, OSMO, NIAID, NIH, DHHS (p) 301-402-3574 wrightk@mail.nih.gov