1. Evaluating Efforts to Support Collaborative Research: Lessons Learned from the AHRQ MCC Research Network Jessie Gerteis, MPH Abt Associates, Inc. 27 th Annual Conference of the American Evaluation Association Washington, DC October 18, 2013

2. Abt Associates | pg 2 Disclosure  The information upon which this presentation is based was performed under contract #HHSA290-2010-00004i, Task Order #2, “Technical Assistance to ARRA Complex Patient Grantees” funded by the Agency for Healthcare Research and Quality (AHRQ), Department of Health and Human Services.  The content of this presentation does not necessarily reflect the view or policies of the Department of Health and Human Services, nor does the mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. The presenter assumes full responsibility for the accuracy and completeness of the ideas presented.

3. Abt Associates | pg 3 What is the AHRQ Multiple Chronic Conditions Research Network?  Research/Learning Network of 27 individually funded grants –14 exploratory grants (comparative effectiveness research on multiple chronic conditions (MCC)) –13 infrastructure development grants (developing new MCC related datasets)  Technical Assistance Center (TAC)

4. Abt Associates | pg 4 Why Did AHRQ Fund a Research Network on MCC?  To facilitate collaboration and innovative thinking across the grants to help advance scientific progress within the emerging and complex field of MCC research

5. Abt Associates | pg 5 Anticipated Challenges to Facilitating Collaborative Research  The AHRQ MCC Research Network: –Individual grants without supplemental funding for collaborative work –Short, two year grant periods (without option for extensions) –Network developed after grants awarded  General research culture: –Typically restricted to disciplinary and institutional silos –Other researchers seen as competition –Collaborative research more time consuming and costly

6. Abt Associates | pg 6 Efforts to Facilitate Collaboration  Held 4 in-person meetings  Coordinated collaborative activities  Regular communications

7. Abt Associates | pg 7 Challenges to Evaluate Efforts to Encourage Collaboration  Not a typical Research Network or a Learning Collaborative  Due to funding structure, funding period, and project goals, also need to measure collaborative activities that occur prior to joint publications or datasets.

8. Abt Associates | pg 8 Model of Collaboration Seven Stage Model of Collaboration* * Frey BB, Lohmeier JH, Lee SW, Tollefson N. Measuring Collaboration Among Grant Partners. American Journal of Evaluation. 2006 Sept; 27(3): 383-392. Coexistence Communication/ Networking CooperationCoordinationCoalitionCollaborationCoadunation - No interaction - Awareness of other groups - Little communication - Decisions made independently - Provide each other information - Formal communication - Decisions made independently - Share information and resources - Frequent communication - Some shared decision making - Share ideas and resources - Frequent and prioritized communications - All members have a vote in decision making - Members belong to one system - Frequent communication and mutual trust - Consensus decision making - Single unified organization

9. Abt Associates | pg 9 Evaluation Design  Did network participation encourage collaboration among investigators?  Data Collection Methods Two annual online surveys of all 27 investigators - 67% and 70% response rate respectively Telephone interviews with a subset of 15 investigators Interviews with AHRQ Project Officers & project team

10. Abt Associates | pg 10 Results: Annual Survey Has your research been enhanced or facilitated by collaboration (e.g., shared research methods, codes, variables, or datasets, etc.) with other investigators in the AHRQ MCC Research Network?

11. Abt Associates | pg 11 Results: Annual Survey Have you created any new measures, developed new coding schemes, or otherwise created something new with other investigators in the AHRQ MCC Research Network?

12. Abt Associates | pg 12 Results: Investigator Interviews  In-person meetings most successful method for encouraging collaboration  Investigators appreciated opportunity to engage with colleagues with different points of view on a topic of interest.  Most investigators reported engaging in some type of collaborative activity with other network investigators in network

13. Abt Associates | pg 13 Results: Investigator Interviews  Spontaneous collaborative activities mainly included information sharing and network building: “I learned another investigator’s process for linking data, and I changed my research strategies based on this.” “We’re looking into the possibility of collaborating with one of the other investigators on a paper or something, but we haven’t done it yet.”  Often, investigators did not see these activities as “collaboration” “I’ve talked to people, exchanged ideas, and exchanged emails… but I didn’t end up really working with other grantees.”

14. Abt Associates | pg 14 Results: AHRQ Interview  Overall, AHRQ felt that the level of collaboration achieved by investigators was a success. “The fact that they have come together as much as they have within this time period has been a success.” “This project demonstrated that there was something of value to be gained by the investigators for participating in a research network.”

15. Abt Associates | pg 15 Results  So, did network participation encourage collaboration among investigators? –Second survey indicates maybe not –Interviews indicate that investigators are engaging in collaborative activities, but not as anticipated –AHRQ views level of collaboration achieved as a success given barriers and grant constraints

16. Abt Associates | pg 16 Discussion Seven Stage Model of Collaboration* * Frey BB, Lohmeier JH, Lee SW, Tollefson N. Measuring Collaboration Among Grant Partners. American Journal of Evaluation. 2006 Sept; 27(3): 383-392. Coexistence Communication/ Networking CooperationCoordinationCoalitionCollaborationCoadunation - No interaction - Awareness of other groups - Little communication - Decisions made independently - Provide each other information - Formal communication - Decisions made independently - Share information and resources - Frequent communication - Some shared decision making - Share ideas and resources - Frequent and prioritized communications - All members have a vote in decision making - Members belong to one system - Frequent communication and mutual trust - Consensus decision making - Single unified organization 0123456

17. Abt Associates | pg 17 Discussion: Lessons Learned  Include explicit time frame for each question in survey  Interviews key for understanding that collaborative activities occurred below level described in survey  Collaboration takes time (and funding support) –Research networks usually funded for minimum of 5 years –Network participation may lead to future collaborations that are not measured –Supplemental funding needed to support collaborative activities  More nuanced definition of collaboration (and clearer target level for collaboration) would have aided evaluation

18. Abt Associates | pg 18 Discussion: So What?  Collaborative research approach is being applied to different types of project models Take Away for the Field:  Establish clear definitions and realistic goals for collaboration at outset  Adapt existing models of collaboration to fit project  Include interviews to capture subtle and nuanced information about collaborative activities

19. Abt Associates | pg 19 Contact Information and Coauthors  Jessie Gerteis, MPH Jessie_gerteis@abtassoc.com  Coauthors: Lisa LeRoy, MBA, PhD, Abt Associates, Inc. Therese Miller, DrPH, Agency for Healthcare Research and Quality