1. Evidence and Organizational Decision Making: Never the Twain Shall Meet? Thomas Rundall, PhD Academy Health, Annual Research Meeting June 27, 2006

2. Research Team John Hsu, MD, MBA, MSCE Estee Neuwirth, PhD Julie Schmittdiel, PhD Laura Arroyo Ilana Graetz Kaiser Permanente Division of Research and Institute for Health Policy Kaiser Permanente Care Management Institute Thomas Rundall, PhD Peter Martelli, PhD Candidate Rodney McCurdy, MHA, PhD Candidate U. C. Berkeley, Center for Health Research Mark Gibson Pam Curtis, MS Oregon Health Sciences University, Center for Evidence Based Policy Financial Support: Agency for Healthcare Research and Quality

3. Introduction There is a substantial gap between health services decision makers and the research community There is a substantial gap between health services decision makers and the research community Studies of health organization decision makers in Canada, the United Kingdom, and the United States report that decision makers rarely use research evidence Studies of health organization decision makers in Canada, the United Kingdom, and the United States report that decision makers rarely use research evidence The research community needs to understand the reasons for this “gap” and work with managers and policy makers to close it The research community needs to understand the reasons for this “gap” and work with managers and policy makers to close it Source: Kovner and Rundall, Evidence-based Management Reconsidered. Frontiers of Health Services Management, Spring 2006.

4. Rationale for the Study Using research evidence when making decisions about the organization, financing and delivery of health care has great appeal Using research evidence when making decisions about the organization, financing and delivery of health care has great appeal Decision makers, however, do not appear to use this information Decision makers, however, do not appear to use this information We queried health organization decision makers about their definition and use of evidence when making strategic decisions We queried health organization decision makers about their definition and use of evidence when making strategic decisions The findings of this study will help inform the construction of an evidence-based management toolbox The findings of this study will help inform the construction of an evidence-based management toolbox

5. Study Design Qualitative Qualitative Four peer-to-peer discussion groups (full day) Four peer-to-peer discussion groups (full day) Questions designed to elicit perceptions of Questions designed to elicit perceptions of  High-priority decisions  Current levels of research evidence use  Characteristics and definitions of useful evidence  Barriers and facilitators to using evidence  Recommended methods for communicating evidence

6. Methods Focus group discussions were held in San Francisco, Oakland, Chicago, and New York Focus group discussions were held in San Francisco, Oakland, Chicago, and New York Purposive sampling of health care senior managers Purposive sampling of health care senior managers 26 organizations of various types 26 organizations of various types  Private provider groups  Health insurance plans and insurers  Public health systems  Employers and employer groups  Regulatory agencies  Organizations with limited resources and wealthier ones 5-10 managers in each focus group (31 total) 5-10 managers in each focus group (31 total)

7. Methods Moderator posed questions and recorded major themes Moderator posed questions and recorded major themes Two researchers recorded all comments and compared notes with meeting audiotapes Two researchers recorded all comments and compared notes with meeting audiotapes Using content analysis techniques, two researchers independently coded the text Using content analysis techniques, two researchers independently coded the text Initial 72.1% agreement on codes; discrepant codes resolved Initial 72.1% agreement on codes; discrepant codes resolved Investigators categorized codes into themes Investigators categorized codes into themes Coded themes compared with moderators’ themes Coded themes compared with moderators’ themes

8. Results Broad list of high-priority decisions Broad list of high-priority decisions  Purchasing to improve quality and reduce costs  Organization of patient care  Organizational sustainability  Benefit design and coverage  Workforce recruitment development, retention Limited use of research evidence in decision making Limited use of research evidence in decision making Difficulty distinguishing between evidence-based medicine and evidence-based management Difficulty distinguishing between evidence-based medicine and evidence-based management

9. Results Broad definition of acceptable evidence Broad definition of acceptable evidence  Internal data/measurements  Colloquial evidence  Externally dictated standards  Scientific evidence Useful evidence must be Useful evidence must be  Accurate  Applicable  Actionable  Accessible Research (scientific) evidence often not useful Research (scientific) evidence often not useful Disconnect between definitions and evaluation of evidence Disconnect between definitions and evaluation of evidence  Lack of experience, resources, and incentives for assessing evidence  Desire for tools and mechanisms for validating evidence

10. Results Tools need to be concise and focused Tools need to be concise and focused Need for processes to help decision makers appraise evidence Need for processes to help decision makers appraise evidence Desire for evidence that is “vetted” Desire for evidence that is “vetted”

11. Conclusions There is a substantial gap between health services decision makers and the research community There is a substantial gap between health services decision makers and the research community Decision makers rarely report using research evidence and instead rely on other types of information such as peers with experience. Decision makers rarely report using research evidence and instead rely on other types of information such as peers with experience. Decision makers perceive that existing research does not meet their needs, both in content and accessibility. Decision makers perceive that existing research does not meet their needs, both in content and accessibility.

12. Next Steps? Build an evidence-based decision making tool box for managers Build an evidence-based decision making tool box for managers Continue to foster collaborative research between health services researchers and managers (e.g. The work of the Center for Health Management Research) Continue to foster collaborative research between health services researchers and managers (e.g. The work of the Center for Health Management Research) Encourage national organizations to work with appropriate information technology companies and experts to improve access to relevant research Encourage national organizations to work with appropriate information technology companies and experts to improve access to relevant research Training and re-training of health services researchers to improve their ability to conduct and communicate research that will be used by health services managers and policymakers Training and re-training of health services researchers to improve their ability to conduct and communicate research that will be used by health services managers and policymakers

13. Next Steps? Work with health organizations to build cultures, processes, and structures that support the use of research evidence in decision making Work with health organizations to build cultures, processes, and structures that support the use of research evidence in decision making Recognize the prevalence of sub-optimal management decisions Recognize the prevalence of sub-optimal management decisions Conceptualize, measure, and assess managerial decision making Conceptualize, measure, and assess managerial decision making How about a new private/public partnership program: TRIM (Translating Research into Management) How about a new private/public partnership program: TRIM (Translating Research into Management)