From Outputs to Impacts: Emerging Approaches to Track Scientific Research Impacts American Evaluation Association Saturday October 27, 2012

Agenda Greatest "HITS": A New Tool for Tracking Impacts at the National Institute of Environmental Health Sciences –Christie Drew & Kristi Pettibone, Program Analysis Branch NIEHS The Becker Model: A Framework for Quantifying Research Impact –Kristi Holmes, Cathy Sarli, Washington University, St. Louis [Innovation: Stefano Bertuzzi, NIMH] Discussion

Greatest "HITS": A New Tool for Tracking Impacts at the National Institute of Environmental Health Sciences Christie Drew, Ph.D. Program Analysis Branch, NIEHS

Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Screen Shots

Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Features and Screen Shots

National Institute of Environmental Health Sciences NIEHS Mission: understand how the environment influences the development and progression of human disease The Division of Extramural Research and Training (DERT) funds a broad portfolio of research across the country and abroad The Program Analysis Branch (PAB): –Evaluation; Tools; Communication NIEHS Website:

Evaluation Context at NIEHS Constantly bombarded with questions about our portfolios –Topics, Methods, Approaches, Results Logic modeling to go beyond bibliometrics –Publication content – not just count! Growing emphasis on impacts… –Evaluation Metrics Manual –CareerTrac

Logic Models Logic Model – organized, project specific, informs metrics Inputs – resources available Activities – actions that use available resources Outputs – direct products of activities Impacts – benefits or changes resulting from activities, outputs

Evaluation Metrics Manual Partnerships Logic Model

Neurodegeneration Program Logic Model

Key challenges Data structures not designed for tracking science impacts Not all data are readily available to evaluators –Annual progress report content not searchable –Program officer notes inaccessible Opportunities

What is HITS? High Impact Tracking System Progress reports and program notes accessible and searchable Robust free-form and structured coding (“tags”) –Portfolio characteristics and impacts Dynamic query and reporting Retains information over time Imports data from IMPAC II, SPIRES Complements existing tools: eSPA, QVR, SPIRES, CareerTrac

Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Features and Screen Shots

Progress Reports and PO Notes To receive ‘non-competitive’ funding, each NIH grantee submits a progress report (structured PDF) Program officers run through a checklist and enter “signoff notes” to authorize the continued funding We pull both of these artifacts, as searchable text, into HITS

Structured Tagging Portfolio Coding –Science Type –Solicitation Type –Landmark Program –Experimental Analyses –Social Science Research –Lifestage –Biological Sample –Exposure Agents –Strategic Plan Goals

Dissemination Coding Congressional Justification Congressional Testimony Papers of the Month Website Highlights Director’s Presentations ARRA Story Ad hoc calls for program highlights

Ontology How to structure our discussion of outputs and impacts? Informed by research at the Washington University in St Louis on research impact Drawn to it because it matches the NIEHS Extramural Program Logic Model Assessing the Impact of Research,

Outputs Scientific Findings Publications Patents Collaborations Animal Models Biomarkers Curricula and Guidelines Databases and Software Measurement Instruments and Sensors

Impacts Improved Health/ Disease reduction Exposure reduction Policies and Regulations Community Benefit Economic Benefits

Objectives Overview What is HITS? Why do we need it? Innovation Progress Reports and Notes Structured Tagging Output and Impact Ontology Demo Key Features and Screen Shots

Searching Three main areas to the search screen –Grant info –Document search Progress Reports Program Notes –Tag Searching Use the search for Identifying tags that exist, OR to Identify Grants that you want to tag.

Search Screen

Search Results

Grant Details Grant Details Page: -Basic Grant information -Existing Tags -Progress report links -PO Notes -Publications -Abstract

Tagging Tag the grant. Grant Details Page: -Basic Grant information -Existing Tags -Progress report links -PO Notes -Publications -Abstract

Tagging Grants

Tag Summary Frequency count of tag types by PO

Next steps Alpha/Beta: Still working on code – portfolio tags Coding on paper still so that we can add it to the system Focus on process to minimize burden on Program staff –Special tag in program notes to signify an important finding –Analysts can access PO assessment of emerging outputs and impacts, and their relative importance –No additional requests for information

How will we use HITS? Respond to frequent requests from NIEHS and NIH OD Congressional Justifications NIH Director Data Calls Presentations and Publications Media Requests ARRA Reports and Highlights

Take home messages Accountability = the new reality Data structures lacking –Need better access to existing information for evaluation purposes –No “right way” but we need to leverage efforts; work together Lack of accepted ontology Need to track products before we can measure impacts –Do we need to be tracking large volume datasets that are produced?

Questions? Contact: Christie Drew, Ph.D. Chief, Program Analysis Branch National Institute of Environmental Health Sciences Research Triangle Park, NC / NIEHS Program Analysis Branch Martha I. Barnes Helena L. Davis Kristianna G. Pettibone Jerry Phelps Elizabeth Ruben SRA International Michael Thornton Tyler Wright Mary Jo Sminkey Henry Rogers