1. From Outputs to Impacts: Emerging Approaches to Track Scientific Research Impacts American Evaluation Association Saturday October 27, 2012

2. Agenda Greatest "HITS": A New Tool for Tracking Impacts at the National Institute of Environmental Health Sciences –Christie Drew & Kristi Pettibone, Program Analysis Branch NIEHS The Becker Model: A Framework for Quantifying Research Impact –Kristi Holmes, Cathy Sarli, Washington University, St. Louis [Innovation: Stefano Bertuzzi, NIMH] Discussion

3. Greatest "HITS": A New Tool for Tracking Impacts at the National Institute of Environmental Health Sciences Christie Drew, Ph.D. Program Analysis Branch, NIEHS

4. Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Screen Shots

5. Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Features and Screen Shots

6. National Institute of Environmental Health Sciences NIEHS Mission: understand how the environment influences the development and progression of human disease The Division of Extramural Research and Training (DERT) funds a broad portfolio of research across the country and abroad The Program Analysis Branch (PAB): –Evaluation; Tools; Communication NIEHS Website: http://www.niehs.nih.govhttp://www.niehs.nih.gov

7. Evaluation Context at NIEHS Constantly bombarded with questions about our portfolios –Topics, Methods, Approaches, Results Logic modeling to go beyond bibliometrics –Publication content – not just count! Growing emphasis on impacts… –Evaluation Metrics Manual –CareerTrac

8. Logic Models Logic Model – organized, project specific, informs metrics Inputs – resources available Activities – actions that use available resources Outputs – direct products of activities Impacts – benefits or changes resulting from activities, outputs

9. Evaluation Metrics Manual Partnerships Logic Model

10. Neurodegeneration Program Logic Model

11. Key challenges Data structures not designed for tracking science impacts Not all data are readily available to evaluators –Annual progress report content not searchable –Program officer notes inaccessible Opportunities

12. What is HITS? High Impact Tracking System Progress reports and program notes accessible and searchable Robust free-form and structured coding (“tags”) –Portfolio characteristics and impacts Dynamic query and reporting Retains information over time Imports data from IMPAC II, SPIRES Complements existing tools: eSPA, QVR, SPIRES, CareerTrac

13. Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Features and Screen Shots

14. Progress Reports and PO Notes To receive ‘non-competitive’ funding, each NIH grantee submits a progress report (structured PDF) Program officers run through a checklist and enter “signoff notes” to authorize the continued funding We pull both of these artifacts, as searchable text, into HITS

15. Structured Tagging Portfolio Coding –Science Type –Solicitation Type –Landmark Program –Experimental Analyses –Social Science Research –Lifestage –Biological Sample –Exposure Agents –Strategic Plan Goals

16. Dissemination Coding Congressional Justification Congressional Testimony Papers of the Month Website Highlights Director’s Presentations ARRA Story Ad hoc calls for program highlights

17. Ontology How to structure our discussion of outputs and impacts? Informed by research at the Washington University in St Louis on research impact Drawn to it because it matches the NIEHS Extramural Program Logic Model Assessing the Impact of Research, http://becker.wustl.edu/impact/assessment/ http://becker.wustl.edu/impact/assessment/

18. Outputs Scientific Findings Publications Patents Collaborations Animal Models Biomarkers Curricula and Guidelines Databases and Software Measurement Instruments and Sensors

19. Impacts Improved Health/ Disease reduction Exposure reduction Policies and Regulations Community Benefit Economic Benefits

20. Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Features and Screen Shots

21. Searching Three main areas to the search screen –Grant info –Document search Progress Reports Program Notes –Tag Searching Use the search for Identifying tags that exist, OR to Identify Grants that you want to tag.

22. Search Screen

23. Search Results

24. Grant Details Grant Details Page: -Basic Grant information -Existing Tags -Progress report links -PO Notes -Publications -Abstract

25. Tagging Tag the grant. Grant Details Page: -Basic Grant information -Existing Tags -Progress report links -PO Notes -Publications -Abstract

26. Tagging Grants

27. Tag Summary Frequency count of tag types by PO

28. Next steps Alpha/Beta: Still working on code – portfolio tags Coding on paper still so that we can add it to the system Focus on process to minimize burden on Program staff –Special tag in program notes to signify an important finding –Analysts can access PO assessment of emerging outputs and impacts, and their relative importance –No additional requests for information

29. How will we use HITS? Respond to frequent requests from NIEHS and NIH OD Congressional Justifications NIH Director Data Calls Presentations and Publications Media Requests ARRA Reports and Highlights

30. Take home messages Accountability = the new reality Data structures lacking –Need better access to existing information for evaluation purposes –No “right way” but we need to leverage efforts; work together Lack of accepted ontology Need to track products before we can measure impacts –Do we need to be tracking large volume datasets that are produced?

31. Questions? Contact: Christie Drew, Ph.D. Chief, Program Analysis Branch National Institute of Environmental Health Sciences Research Triangle Park, NC 27709 drewc@niehs.nih.govdrewc@niehs.nih.gov / 919-541-3319 NIEHS Program Analysis Branch Martha I. Barnes Helena L. Davis Kristianna G. Pettibone Jerry Phelps Elizabeth Ruben SRA International Michael Thornton Tyler Wright Mary Jo Sminkey Henry Rogers