1. DIVISION OF LIBRARY SERVICES | OFFICE OF RESEARCH SERVICES | NATIONAL INSTITUTES OF HEALTH Assessing Our Value: This is Our Story Alicia Livinski, Diane Cooper, Bradley Otterson, Nancy Terry, Terrie Wheeler, Anne White-Olson National Institutes of Health (NIH) Library

2.  Federal library supporting a biomedical research institution  NIH employees, students, fellows, visiting researchers etc.  Also serve various US Department of Health & Human Services (HHS) staff and operating divisions  HHS users include: policy analysts, budget analysts, healthcare providers, mental health providers, administrative staff, program officers etc.

3. HHS users include:  Administration for Children & Families (ACF)  Administration for Community Living (ACL)  Agency for Healthcare Research & Quality (AHRQ)  Assistant Secretary for Financial Resources (ASFR)  Assistant Secretary for Planning & Evaluation (ASPE)  Assistant Sec for Preparedness & Response (ASPR)  Health Resources & Services Administration (HRSA)  Indian Health Service (IHS)  Office of the Asst Secretary for Health (OASH) (+ 14 sub-offices)  Office of General Counsel (OGC)  Office of Global Affairs (OGA)  Office of the Inspector General (OIG)  Office of the National Coordinator for Health IT (ONC)  Office of Civil Rights (OCR)  Program Support Center, Federal Occupational Health  Substance Abuse and Mental Health Services Administration (SAMHSA)

4. Purpose of the study  Specifically, this assessment will seek to address whether the services and information provided by National Institutes of Health (NIH) librarians, as perceived by our HHS users, impacted our user’s decisions and/or actions related to patient care, policy, administrative, research or educational work.

5. Study design  Setting: federal biomedical research library  Assumptions: Individual survey responses will be confidential, and participation is entirely voluntary.  Population: Health policy, public health professionals, clinicians and other healthcare professionals from HHS. Our population (N) are the HHS individuals served by the NIH librarians. The survey will be administered to a convenience sample (n) of the people served beginning in April 2012 to when 20% of all total incidents in 2011 (162) is reached. In 2011, the total incidents for HHS customers completed by the NIH librarians were 811.

6. Study design  Questions: were primarily based on the survey questions developed and used for the Federal Libraries Value Project in 2010 & 2011;  Patient care questions from previous studies of federal medical libraries;  Policy questions were developed with input from HHS users & review of public health policy and knowledge management literature;

7. Study design  Methods: critical incident technique was used  SurveyMonkey™  Study questions were piloted prior to launching the survey  Began in April 2012 and will end when at least 162 surveys are completed  Librarians email a link to the survey after completing the following: document delivery by the librarian; literature searches; reference questions; editing/formatting of manuscripts; instruction on searching  Each librarian enters the # of distributed surveys into a Excel spreadsheet

8. Survey results to date  As of April 2013, 73 surveys completed (out of 162 needed); Q: Job titles of respondents:  Physician (10%)  Program analyst (7%)  Fellows (6%)  Nurse (6%)  Public health analyst (6%)  Policy analyst (6%)  Executive staff (4%)  Other (31%)

9. Question asked of all users QuestionYesNoPartially Did the information service you received from the NIH Librarian answer your question? (n=70) 89%0%11% Did the information provide you with new or updated data or knowledge that you can use? (n=68) 96%4%0%

10. Purpose of information request (n=69)

11. Asked of all users Q: Thinking about the information you received, how much influence did it have on your work?

12. Asked of all users Q: If you had not asked the NIH librarian, how would you have sought the information?  Searched Google=29%  I would not have searched=18%  Asked a colleague=10%  Other=43%  I would have been unable to meet my goals;  Asked subordinates/colleagues/contract staff;  Search databases, PubMed, Google, journals myself;  Bought the book for a lot of money;  I would have tried my university alumni library

13. PATIENT CARE

14. Patient care (n=5) 100% of respondents indicated that the information provided by the NIH Librarian was useful for direct patient care.

15. Patient care (n=5) Q: After reading the information provided, how did it influence you?  Provided me with standard of care info =60%  Advice made to patient or family= 40%  Reinforced my plan of treatment=40%  Changed the length of stay=20%  Improved patient management=20%

16. Patient care (n=5) Q: Were any of the following events avoided because of the information provided?  I don’t know=40%  Additional test or procedures=20%  No events were avoided=20%  Other:=20%  program activities or plans based on outdated info can be avoided because of this information

17. Patient care (n=5) Q: Did the information provided have a potential financial impact for either patient or the medical center.  I don’t know=40%  Avoided unnecessary costs= 20%  Other: =40%  Potential for cost savings based on prevention of adverse health outcomes;  May increase pharmacy expenses while decreasing morbidity & mortality;

18. RESEARCH

19. Research (n=36) Q: What research functions did the information you received assist you with doing?  Publication in peer-reviewed literature=36%  Technical/scientific report=33%  Decision making=33%  Practice guidelines=31%  Protocol development=31%  Systematic review=25%  Treatment improvement guidelines=19%  HHS publication/report=14%  Consensus development=8%  Conference presentation=8%  Other:=14%

20. Research (n=36) Q: Was the research part of obtaining research funding or IRB/IACUC approval?

21. Research (n=4 who answered yes to previous question) Q: Please estimate funding to be sought or received.  1 million – 1.9 million=25%  100,000 – 249,000=25%  Not applicable=50% Q: Select statement that best represents the financial impact of the information provided on your research.  Avoided unnecessary costs=50%  Support for a research proposal=25%  Other: =25%  Avoidance of repeating studies due to technical errors

22. POLICY OR ADMINISTRATIVE

23. Policy or administrative (n=18) Q: Which best describes the outcome(s) of the information requested?  Review agency policies/procedures=39%  Conduct program evaluation/development=33%  Conduct policy review/implementation=33%  Identify best practices=28%  Perform legal research=17%  Identify collaborators/business opportunities=11%  Conduct research impact analysis=11%  Other=17%  Review of social phenomenon to guide policy research; online news monitoring

24. Policy or administrative (n=18) Q: How would you describe the value of the information received from the NIH Librarian?  Helped me to make a more informed decision = 67% or recommendation  Confirmed my decision or recommendation = 39%  Increased my confidence in making a decision =22% or recommendation  Refreshed my memory of details or facts =11%  Caused me to consider another approach = 6%  Other = 22%

25. EDUCATION

26. Education (n=10) Q: Did the information you requested support one or more of the following?  Your own professional development=50%  A class you are taking=20%  Background information for speech/paper=20%  Other=50%  Health systems redesign;  Improve nutrition counseling we provide;  Grant writing efforts;  Information for tribal health partners

27. Challenges  Remembering to include the survey when responding to information requests (and sending follow up)  Question development was a challenge –  a majority of our users are not healthcare providers or bench scientists (although the Indian Health Service does provide most of the clinical care) &  how do we measure our impact on policy, analysis or other public health focused work

28. Conclusion  We are not done yet. We had anticipated completing our data collection and analysis by May 2013; however, we are still collecting our data.  However, these preliminary results are positive and are beginning to show us how our HHS users perceive the impact of our services on their work.  We ask that anyone working with similar user populations and are interested in question development to please contact us.