1. AHRQ Annual Conference September 2011

2. Thank-you to:  Survey respondents  Focus group participants  Key informants  AMIA and ACCP  AHRQ  Mary Nix  Judi Consalvo  Gov Delivery Staff  Marjorie Shofer  HIT Portfolio  AMA and AHIP  ECRI Institute  NGC Evaluation PET  Florence Chang  Belinda Ireland  Richard Shiffman  Katrin Uhlig  Cally Vinz Evaluation Project Team Members AFYA, Inc. Michelle Tregear Jenice James Debra Dekker Craig Dearfield Ajay Bhardwaj Robin Pugh-Yi The Lewin Group Carol Simon Jaclyn Marshall Jacob Epstein

3.  Gain a better understanding of how NGC:  Is used by its stakeholders (including AWARENESS among key stakeholders)  Supports dissemination of evidence-based clinical practice guidelines and related documents  Has influenced efforts in guideline development, implementation, and use  Can be improved

4.  Quantitative Data  Web-based Survey ▪ Skip logic and Branching ▪ Respondents solicited by e- mail lists for AHRQ, AMA, and AHIP  Qualitative Data  Focus groups (4) ▪ Stakeholder-specific  Key informant interviews (26) ▪ Mix of stakeholders Mixed-Methods Approach Key Project Milestones CIPP evaluation framework – Logic model to develop key questions Developed instruments which were Informed by the PET Received OMB clearance Feb 2011 Conducted survey, focus groups, and interviews (Mar – Jul 2011)

5. Referral Source SourceCount Total 9,389 AHRQ 9,298 (99%) AHIP 42 AMA 49 Respondent Demographics Survey Reach  Majority from the U.S. (87%) Occupation Mix  56% - Providers, clinicians, nurses  13% - Researchers, librarians, or similar  12% - Consultants, managers, administrators  19% - Other Survey Sample Majority familiar with AHRQ ( 99% )

6. Section / ModuleCount Total9,389 (100%) NGC Unaware2,075 (22.1%) NGC Aware7,314 (77.9%) Non NGC User1,395 (19.3%) NGC User5,828 (80.7%) Guideline Developer1,076 (18.5%) Health Provider3,271 (56.1%) Medical Librarian204 (3.5%) Informaticians292 (5.0%) Researcher1,219 (20.9%) Policymaker1,219 (20.9%) Measure Developer351 (6.0%) Stakeholder-specific questions NGC questions NGC awareness and use, demographics, other guideline source questions

8. Key Notes Majority of respondents derived from AHRQ e-mail invitation Awareness of NGC greatest among AHRQ and AHIP respondents AHRQ Opportunity to increase physician awareness of NGC Awareness of NGC in 2011 substantially higher than in 2001 evaluation N=9,289N=42 N=49

9.  Survey Findings  Most use multiple sources to find CPGs (3-5 most common) ▪ PubMed, government sources, general search engines, medical/ professional societies  Qualitative Findings:  NGC often cited as “first go-to source” Key Finding Most NGC users equally satisfied or more satisfied with NGC compared to other guideline sources

10. Key Finding NGC is doing well on these needs “indicate the degree to which use of the NGC Web site fulfills your needs for:”

11. Finding NGC does better or about equal compared to other sources This is supported by qualitative findings

13.  Survey Findings  75% -very good or good ▪80% - would definitely, very likely, or probably recommend NGC to colleague No differences by key stakeholder group  Qualitative Findings: Differences by key stakeholder group ▪ Guideline developers and informaticians less trusting ▪ Others note that they “trust” the content because “it comes from AHRQ”

14.  Survey Findings  63% - appropriate ▪ By length-of-use: longer users more likely to find the criteria too loose No differences by key stakeholder group  Qualitative Findings: Differences by key stakeholder group ▪ Guideline developers/informatics specialists said they are ‘too loose’ ▪ Policymakers, medical librarians, researchers generally satisfied

15.  Qualitative Findings (Guideline Developers and Informaticians): ▪ “…the quality of the criteria was good when NGC started out but it has gotten more complicated...” ▪ “…If the goal is to be ‘all inclusive,’ then the criteria are fine. [but]….there needs to be some other ways to separate the wheat from the chaff.” ▪ “They could raise the bar.” ▪ “I’d say too loose…. I think there’s a belief that: A) NGC creates these guidelines…and then, B) there’s also a belief that NGC somehow has a very rigorous process for only allowing certain guidelines in, or certain types of very high quality guidelines, or that it’s an endorsement of these guidelines. And it isn’t.” AHRQ Opportunity to revisit NGC’s Inclusion Criteria

16. AHRQ Opportunity to revisit NGC’s Age Criterion  Survey Findings  ~Equal # say 5 years is “appropriate” or “too long” ▪ Reduce to 3 years most common selection  No differences by key stakeholder group  Qualitative Findings:  Common theme: the age criterion is too long

17. NGC Influences

18.  Respondents, by group, indicated NGC greatly influenced:  Guideline developers’ ability to identify guidelines, and develop and use quality measures  Health providers’ ongoing learning activities, clinical decision- making processes, and ability to identify guidelines  Medical librarians’ ability to meet their client’s needs  Medical librarians’ and researchers’ ability to identify current and high quality guidelines  Measure developers’ measure development activities and approach to identifying guidelines  Policymakers’ and purchasers’ ability to identify guidelines and convert clinical information

19.  Respondents, by group, indicated NGC greatly influenced:  Guideline developers’ ability to identify guidelines, and develop and use quality measures  Health providers’ ongoing learning activities, clinical decision- making processes, and ability to identify guidelines  Medical librarians’ ability to meet their client’s needs  Medical librarians’ and researchers’ ability to identify current and high quality guidelines  Measure developers’ measure development activities and approach to identifying guidelines  Policymakers’ and purchasers’ ability to identify guidelines and convert clinical information

20.  Survey Findings (n=199)  NGC guideline submitters reported greater NGC influence for guideline updating frequency, and how organizations document or report their guidelines  Qualitative Findings (n=24)  NGC serves primarily as a source for locating guidelines  NGC’s age criterion applies some “pressure” to stay current  NGC has had little influence on how guideline developers do their work– e.g., methodology, reporting AHRQ Opportunity to increase knowledge among guideline developers about how to create and report trustworthy guidelines

21.  Noteworthy finding  65% said excellent or good  21% were neutral  14% said fair or poor Question: How would you rate NGC's dissemination of your organization’s guidelines? AHRQ Opportunity to identify additional efforts to enhance the dissemination of guidelines

23.  Enhancements  Two thirds said they would “definitely,” “very likely,” or “probably” use the following NGC enhancements if available: Ratings of guideline quality Subject-specific e-mail alerts  Enhancements for providers  72% - having NGC content at point-of-care would be useful  66% - would take CME if available  Enhancements for informaticians  52% - NGC Summaries as XML file according to the Guideline Elements Model (GEM) or other similar Examples of Potential Enhancements Ratings of guideline quality Subject-specific e-mail alerts Access to archived guidelines Additional data download options and xml formats AHRQ Opportunity to invest in major enhancements that will increase the value of NGC

24. Additional Common Themes  Guideline Developers  Commentary/responses to guidelines from users  Guideline developer conferences / methodology workshops  Informatics Specialist  Assessment of the executability of guidelines  Medical Librarians  Integration with other Web sites (PubMed)  Quicker access to “new” guidelines  Researchers  Assessment of attributes in IOM standards for guidelines  More information about treatment in multi-morbidity or comorbid conditions

26.  AHRQ Opportunities  AHRQ Opportunities include: Build on NGC’s user base, including with health providers Revisit the NGC inclusion criteria Revisit the guideline age criterion Increase knowledge among guideline developers about how to create and report trustworthy guidelines Enhance guideline dissemination efforts Invest in major enhancements to the NGC Website that will provide significant added value